Think like a test writer

One great way to start smashing test questions is to start thinking like a test writer, instead of a test taker.  You’d better believe that the test writers spend tons of time studying how you think,* so don’t you think you should return the favor?

If you were assigned to write questions for your choice of test, what would your objective be?  Well, this is a standardized test, and the standardization means that the people who take the test this year have to have basically the same experience as last year and the year before, only the questions have to be completely different.  Not only does it have to feel the same, but the same person (who hasn’t done any test smashing) should get the same score every time she takes the test, within some narrow band.  And in order for that to happen, you have to know as precisely as possible, how many people — and of what “sort” — will get your question correct or incorrect.  Of those who get the question incorrect, you should have a pretty good idea of the distribution of which wrong answer they will choose.  The hopeless test takers will go for one wrong answer, and the almost-there test takers will, perhaps, fall for a different trap.

Every time you take an official test, you are contributing your lab rat skills on those experimental sections.  (Does that make you feel good?  You are contributing to science!  For free!)  That is where they test and calibrate their questions.  But you’d better believe that they aren’t writing random questions that they then learn about on the experimental section: they already have a pretty good idea what is going to happen by the time the question shows up on your test.

So, if you are the manager of a bunch of question writers, what are you going to do?  Well, there have to be templates that writers share, so that Sarah’s questions look the same as Karim’s questions.  And, dear TestSmasher, if you can figure out the templates, it is like having a master key to the test — or a sledge hammer!

This could work for any question type, but I think arguments are particularly well suited for this exercise, which is to sit down and write (at least) 3 questions in the style of your test:

  1. Pick one stem (maybe a question type you hate?) and use it for all three.
  2. Find a couple of official questions of the same type.
  3. Analyze the best you can the argument structure: number of sentences, the role they serve, etc.
  4. Choose 3 totally unrelated topics
  5. Craft an argument on each topic that conforms to the structure
  6. Write at least three answer choices (five might get tedious)

Most of the art of question writing lies in writing the answer choices, so let’s drill down into that one.

The correct answer has to be:

  1. Indisputably correct.  (Think about aggrieved test takers and their lawyers.)
  2. Not obvious.

How do you do both at once?  The way I would go about it is write an obvious and correct answer choice and then revise by picking synonyms or other phrases that mean the same thing but aren’t the way you would think to characterize the correct answer if you were a normal person instead of a test writer.

The wrong answers have to be:

  1. Indisputably incorrect.  (Think lawyers again!)
  2. Attractive.

And how do you do that?  Well, you say things that are correct but not answering the question, of course!  Or things that are in response to the question but are slightly, subtly, but fatally wrong.   Write at least one of each.

Once you have done this exercise, first, post your fake questions in the comments — I’d love to see what you come up with.  Then, go do a set of real questions with the same stem.  Do you approach the questions differently?  Identify the wrong (or right) answers more consistently?  Do you catch yourself before you fall into a trap that used to trip you up?

*There is an entire scientific field, psychometrics, that studies how you think and respond to questions designed to measure something about your cognition, psychology, or character.

Advertisements

My current project

I’ve been neglecting the website lately, as I have been spending most of my TestSmasher time working on my new book, Leaving Low Scores Behind: LSAT Edition.  It is a three week self-study guide to walk you through how to get past a low score, re-establish a much better relationship with the test, and develop study skills and a strategy to launch you into a successful period of study and test taking where you actually get the score you need to go to the law school of your dreams.  The book is based on insights from neuroscience, social psychology, and my years of teaching struggling students.  I’ve now finished a draft and am starting to think about the best way to publish, market, and distribute the book.

If this sounds intriguing to you, please consider signing up to be my beta tester!  I would love to work with a few more students who are looking to improve their scores significantly so that I can understand how well it speaks to different people and figure out ways to help you and it kick ass even harder.  I’m still offering my “Low Score Recovery” package, which is a steeply discounted opportunity to work with me as you go through Leaving Low Scores Behind.   If you are interested in the book but not the tutoring, please shoot me an email, and we can work something out.

Finally, if you want to walk proudly away from an old low score and on to a much higher score on the GMAT or the GRE, please also contact me!  My plan is to adapt this to each of the graduate-level tests, and I’d love to help you too.

A note on question types

In my last post, I was a bit snarky about the tendency of test prep companies to develop complicated taxonomies of, say, argument question types.  I’d like to be a bit more nuanced about that.  I like simple taxonomies, ergo my DESTROY, SUPPORT, DESCRIBE for arguments, and I do think that all of the arguments fit into one of those three broad categories.

HOWEVER.  In the course of your studying, if you start to notice that it helps you to think about “inference” questions differently than “conclusion” questions, that is fantastic.  If it identifies for you a difference that is meaningful to you in what approach you take/traps to look out for, then you know you are studying and learning well.  My only quarrel is when you spend a ton of time (and money) learning someone else’s taxonomy, with nuances that don’t mean a whole lot to you.  Then you stress out a bunch over your inability to see any difference between an inference or a conclusion question, when at the end of the day, there isn’t a difference, and it shouldn’t cause you any stress, because if you weren’t so fucking stressed out, you’d get the right answer.

This applies to other question types as well (RC, LSAT games, math questions on the various math-y tests).

What are you studying for?

I want to post some more worked examples for arguments, but I’ve been chewing over this question for quite a while now, and want to provide some thoughts on what the bulk of test-prep study time should be focused on.

There is a little bit of substantive stuff that is worth making sure you know for each of the tests.  Some math maybe, or logic rules for the LSAT.  This has to be done.  It is also possible to create some things to study for, such as finely grained categories of argument questions.  This is usually a waste of time.  Or you could just do practice questions until the cows come home, hoping for improved results.  The tests are pretty resistant to magical thinking, though.

So what are you studying for?

I like to think of it as taking the decisions out of the test.  You want to absolutely minimize (i.e. get to zero) the moments on the test where you are looking at a question and don’t have a plan for solving it.  This is different from knowing what the right answer is off the bat.  Instead, it is having a process that you know and trust and FOLLOW EVERY SINGLE TIME.  My suspicion for the students who take a test prep class, feel like they are improving, and then get the same fucking score as they have gotten at the start, is that they get into the exam and wing it, rather than relying on a set process.

When you are nervous, you make DUMB decisions.  Big time DUMB.  You are not dumb, but when you are thinking about “ohmygodIhopeIdowellbecause IreallyreallywannagetintoHarvard” you are not at your tip top in thinking through the ins and outs of your approach.  It’s just the way the brain works.  So the way to cope is not to start thinking “ohmygodIbetternotfreakoutbecausethen I’mgoingtomakebaddecisionsandnotgetintoHarvard” but instead to not make decisions when it isn’t safe.  Sort of like not driving drunk.

When you turn the page/click to the next screen, and see a reading comprehension passage, you want to have a tried-and-true process that you don’t question and don’t change on test day.  You are going to read the first question stem, skim for an answer, prove every fucking word of your answer choice is addressed in the passage, rinse and repeat.  You are not suddenly going to decide that maybe reading the passage first would be a good idea.  You aren’t going to go all magical thinking and pick (C) because it just “sounds” like the right answer.  You aren’t going to panic about the time and pick the first plausible answer because “whatifI’mtooslowanddon’tfinishontime bettergofasterorImightnotgetintoHarvard.”

You want to walk into the test with a very clear, realistic idea of what your score is going to be and how you are going to earn it.  Of course there will be some variation and some moments that may call for a bit of creativity or insight, but that is to be kept to a minimum.  If you don’t already know what your score will be, you shouldn’t be taking the test.  If you think you know what your score will be but hope it will be lots better, you DEFINITELY aren’t ready to take the test.  Instead, you want to work systematically through the question types on your test, and figure out your approach, practice it consistently until you would NEVER do it differently, then move on to the next question type.  This is the heart of Test Smashing.  The test should be little smithereens before you walk into the test center.

A note on arguments on the GMAT

In recent years, the GMAT has been trying hard to become more “relevant” to business school.  So, a decade ago, its arguments were just slightly dumbed down versions of the LSAT arguments.  Now, some of their arguments (but by no means all, see the first worked example for a classic one) have a more “business decision” feel to them.  These questions fit less cleanly into the basic structures of the question type, though at root, the rules are the same.  However, to smash these questions, you need to have a stronger intuition about how these questions work so that you can apply the rules more flexibly.

As a consequence, I actually recommend to GMAT studiers that you go buy some LSAT tests — they sell individual tests, which might be enough, or books of 10 tests, which, if this stuff is hard for you right now, might be a good investment.  These questions are good for building your foundation for two reasons: they have very transparent, predictable structures, and the types of logic they use can, on harder questions, be more sophisticated than what you will see on the GMAT.  This builds extra capacity and good intuition.  Then you can go work on the GMAT questions with a much stronger foundation.

Arguments: Working an example II

This argument is Q26 in the Critical Reasoning Practice Questions in the 13th Ed. of the GMAT Official Guide.  Let’s work through it in the steps I discuss in this post.

1.  The stem reads: “Which of the following conclusions can most properly be drawn from the information above?” This may sound like it doesn’t fit neatly into my DESTROY, SUPPORT, DESCRIBE framework, but it actually does.  When you are asked to draw a conclusion from an argument, you may not make any assumptions.  This means that the correct answer is an amalgamation of what is in the argument already, and therefore is a DESCRIBE question.

2.  The argument reads:

When a polygraph test is judged inconclusive, this is no reflection on the examinee.  Rather, such a judgment means that the test has failed to show whether the examinee was truthful or untruthful.  Nevertheless, employers will sometimes refuse to hire a job applicant because of an inconclusive polygraph test result.

This is actually all premise (you get to come to a conclusion), though the facts have some discrepancies.

3.  I do not worry about how I happen to know that polygraph testing in general is highly unreliable with lots of false positives.  This is not relevant.

4.  Okay, I don’t need to look for assumptions, I just need to describe.  But it is still useful to simplify the argument to build my description:

(1) Inconclusive = meaningless re applicant.  (2) Inconclusive = bad test.  (3) Inconclusive –> employers turn down applicants.

Which answer choice combines these 3 statements accurately?  Let’s do this by process of elimination.

  • (A) Most examinees with inconclusive polygraph tests are in fact untruthful.  This says that Inconclusive = usually untruthful re applicant.  Contradicts (1).
  • (B) Polygraph tests should not be used by employers in the consideration of job applicants.  This says that Inconclusive + Conclusive results = useless.  We know nothing about Conclusive results.
  • (C) An inconclusive polygraph test result is sometimes unfairly held against the examinee.  We know that Inconclusive = meaningless re applicant (1), and Inconclusive –> bad outcome for applicant.  Since suffering a bad outcome for something that you did not control can be described as unfair, this answer choice is not totally sucky.
  • (D) A polygraph test indicating that an examinee is untruthful can sometimes be mistaken.  The argument tells us nothing about conclusive results. Note Step 3!!!
  • (E) Some employers have refused to consider the results of polygraph tests when evaluating job applicant.  The argument only tells us about how employers use polygraph results, nothing about how they don’t use results.

You should be hyper wary of (C) because of its use of a value-laden word “unfairly.”  Almost always, these answer choices are traps.  But here is an exception worth noting, and you can be confident that it is an exception because the other choices are so badly wrong.

Arguments: Working an example I

This argument is Q2 in the Critical Reasoning Practice Questions in the 13th Ed. of the GMAT Official Guide.  Let’s work through it in the steps I discuss in this post.

1.  The question stem is: “The argument is flawed primarily because the author”.  So, I write down DESTROY on my paper.

2.  The argument is:

Homeowners aged 40 to 50 are more likely to purchase ice cream and are more likely to purchase it in larger amounts than are members of any other demographic group.  The popular belief that teenagers eat more ice cream than adults must, therefore, be false.

The first statement fits in the premise slot, the second in the conclusion slot.  How do I know?  The first sentence is stated as a fact.  The second sentence uses the telltale “therefore” to indicate that it is the conclusion.

3.  I do not indulge in thoughts about how 30-something single females are inclined to eat large quantities of ice cream after romantic missteps!

4.  Since I didn’t write down DESCRIBE, my work is not done.

5, 6.  A shortened, focused summary of the argument:

H buy most X.  Therefore, T do not eat most X.

What is the important shift between premise and conclusion?  buy –> eat.  So I need to find an answer choice that phrases that in a way that DESTROYS the argument.

7.  Looking at the answer choices, A pops out:

fails to distinguish between purchasing and consuming

Since purchasing = buying and consuming = eating, we have a perfect match.

8.  I read again the stem, yes, I am supposed to DESTROY.  A is phrased in a way, “fails to distinguish,” that DESTROYS.

Smash!!!