How I reverse-engineered the SHSAT

Every year, tens of thousands of New York City eighth-graders sit for the Specialized High School Admissions Test (SHSAT), and the families behind them spend a great deal of time, money, and worry trying to prepare.

However, most of that preparation is tapping in the dark: relying on tutors, prep books, or the experience of current high-schoolers who have taken the test before is all that parents can do. The NY Department of Education (DOE) never provides any information or guidance about how the test is really designed, so most parents and educators try to make guesses about the content of the upcoming test.

I am one of those parents. My daughter is a middle school student and she will be taking this test too. What I wanted for her is something better than this.

Unglamorous work

Fortunately (for her), I am an educator with (mildly autistic) love for meticulous practices of breaking things down, so I decided to break down the SHSAT into its component pieces. I really wanted to figure out what is this test really made off, how did it evolve over the years, and what kinds of skills it really tests for.

As I was on my summer break from teaching (I’m a university professor), I had some time on my hands to dig deep.

And deep did I dig.

In this article, I’ll describe you how I did it, and I’ll report on all the things I found out. If you have a child gearing up for the test, or you are a tutor prepping students, you really need to read this.

The data and the method

First, in order to make any meaningful conclusions, I needed all the data. So, I did the unglamorous (and pain-in-the-butt) work of collecting every single test the DOE published on their website, extracting the questions, answer choices, answers, reading passages — the whole shebang — and converting them into a single dataset.

There are plenty of different practice tests out there, but since the real test is administered by the DOE, I consciously limited my analysis only to this verified and valid source.

As you can imagine, the amount of data here is fairly large. We’re talking about eight years of released test samples (two per each year), which amounts to more than a thousand pages of test material; close to a hundred reading passages, roughly 1,300 questions spanning both ELA and math, and more than five thousand answer choices (both correct and wrong).

The dataset I ended up with gave me a ground truth for answering a bunch of interesting questions I had about the test. For example:

• What do the SHSAT questions really ask?
• How are the reading passages chosen and/or constructed?
• Are correct answer positions truly random?
• How are the wrong answer choices engineered?
• What are the most common traps?

Finding answers for these was really fun.

The main method I used was statistical and linguistic (NLP) analysis. I understand you may not be an expert at these things, so I won’t bother you with details, but I will explain some things because I want you to understand.

Statistics gives us great tools to analyze large amounts of data in general, and NLP (Natural Language Processing) allows us to analyze language in a precise mathematical way.

Reading passages

As you may already know, SHSAT is organized around separate ELA and math sections. Each section consists of 50 questions (in the latest iteration of the test, used to be 57). ELA itself is divided into revising/editing, and reading comprehension questions.

Both of these sub-sections of ELA rely heavily on reading passages. Students are given short chunks of text and they are tasked to either revise/edit them or answer a series of comprehension-based questions.

The first (and most basic) thing I was able to find out was the length of passages over the years. In the early years (2018 onwards), the passages would average around 600 words, but in the last few years, the DOE pushed the length all the way to around 800.

But, as all good statisticians know, averages hide as much as they reveal. If we look at the length within each year, you’ll see some interesting variance:

As you can see, passages sometimes go down to an 200 words, and sometimes shoot up all the way to 1,600 or more. Of course, revising/editing subsection tends to run the shorter ones, while reading comprehension the longer ones, but this is a very important finding: a reading passage in the next test could be anywhere in this range. Which means: students need reading stamina in case they get a long one.

Vocabulary

The second big aspect of the ELA is, of course, the vocabulary. SHSAT reading passages are known to be demanding vocabulary-wise: students must be familiar with a lot of complex words.

Here’s how vocabulary richness evolved over time:

Again, the averages mask the outliers. Ever few years a passage arrives with an unusually dense vocabulary, the kind that can confuse even a strong reader. Check out the year-to-year distribution:

Complexity

The single most important reading passage metric is, probably, reading complexity. It is measured using something called Flesch-Kincaid Grade Level scale, which maps reading complexity to school grades (it’s a staple method in computational linguistics and NLP).

My analysis discovered that the SHSAT passages, on average, sit just under 11th grade reading level. Which means the students taking SHSAT, who are all eight-graders, must read three grades above their level.

That’s demanding, and if you’ve ever read a dense SHSAT passage, you’ll immediately recognize this.

But, there is some good news. That complexity has trended gently downward over the past several years. Check the data:

As always, the year-by-year spread reveals more than the average. Within any single test, some passages may run well above 11th grade (genuinely college-level text), even as others are lower.

There is also a small, but consistent difference between the two sections. The revising/editing passages are slightly harder than reading comprehension ones (which makes sense given what is asked from students to do):

Questions

Once I got a sense of the reading passages, I wanted to find out more about the questions. How are the ELA questions constructed, what do they aim for?

The first (and most obvious) question a struggling student might ask is this: “if I don’t know the answer, what choice should I select”? I’ve heard many people say that no pattern in the position of correct answers exists, so I wanted to verify that.

That’s largely true. Here is the distribution of the correct answer positions:

What do the reading comprehension questions on SHSAT actually ask? To me this was the most important finding of this entire exercise because it tells the students where to focus their prep efforts. I classified every reading comprehension question by the kind of task it requires. Here’s the result:

As you can see, the most important skill is interpretation. More than a third of the questions ask the reader to draw an inference and find the evidence for it, and nearly a third ask about why the author made a specific rhetorical move or a narrative choice.

Anatomy of a wrong answer

All questions in SHSAT come with four answer choices, one of which is correct, and three are so-called distractors. These distractors are not random, however. They are cleverly designed to catch particular mistakes students make.

If you read a passage and choose one of these questions yourself (which I recommend doing), you’ll see exactly what am I talking about. Very often, there will be questions in which all answer choices will sound plausible to a reader, and it will take careful reading and analysis to find the correct one.

This is not an exaggeration; I even verified it computationally. For my fellow nerds out there, here’s how I did it: I embedded all answer choices (converted them to vectors) and computed the cosine similarity (distance in vector space) between them. They all clustered close to one another.

For my fellow non-nerds, here’s a visual explanation of how this works:

Every passage is about something, right? So, every passage comes with a bound set of meanings (if a passage is a story about Native Americans bringing a long-lost tribe member, then statements about NYC’s flood-prevention programs are outside of that bound).

Imagine that ‘universe of meaning’ in a passage is represented by a rectangle, denoted by Ω (Greek letter omega) here. Imagine we have a reading comprehension question, which are always about meaning, and two answer choices, A and B, where A is the correct one and B is the incorrect one. They might be similar to one another because they may overlap in meaning to some degree (they will both cover the ground truth), but only one of them — A in this case — covers the entire ground truth, while the other one only part of it. Unlike A, B covers just a small part of what is true about the passage, but also brings in irrelevant detail, causing students to misidentify it.

These distractors work in different ways. Sometimes, a distractor will not bring irrelevant detail but it will not be broad enough, like this:

Or it will try to confuse students by covering something about the true meaning of the passage but it will also bring things never mentioned in the passage at all (usually something students might already agree with), like this:

It’s challenging, I know, and I feel the frustration many students experience when they grapple with this.

But, the good news is that my analysis discovered the most common distractors. I analyzed all of the wrong answer choices in ELA reading comprehension section, and these are the most common ones:

A misidentified detail answer points to something genuinely in the passage, but not the thing the question asks about: it is within Ω but outside circle A (the difference is often very nuanced). A wrong concept answer sounds reasonable otherwise but rests on a misunderstanding of what the passage actually argues. The partly-true-but-wrong trap (the third most common) is the cruelest: an answer that is accurate in part, which is exactly what makes a hurried student grab it. And the inference leap trap offers a conclusion that goes one step further than the text supports (the circle falls outside Ω).

Revising/editing distractors are somewhat different, which makes sense given the task these questions demand. Here are the most common ones:

In most cases, they will try to trap the students into believing that the real mistake is somewhere else in the sentence (or passage) or will offer plausible-sounding answer choices that subtly change the meaning of the sentence.

Math

In many ways, the math section of the SHSAT is easier than the ELA. This is mainly because math is fairly straightforward — the interpretation is much easier if one knows the definitions — but this doesn’t mean it is a breeze.

The math section is organized around several sub-topics, with ratios/proportions and algebra sitting on top as the most frequent skill needed to answer math questions correctly.

Here’s the entire distribution of math question skills:

It is not surprising then that the most common distractors try to trap the student by offering answer choices based on a wrong setup of the problem. My analysis showed that more than a half of all math distractors play this strategy. See it here:

The other half is a mix of wrong choices that pray on small and predictable mistakes students make in arithmetic, as well as habits of jumping to the answer prematurely, or omitting a sign before the parentheses.

Not all distractors concentrate on every math area equally. Here’s the distribution of the main five:

How to prepare for the test?

Everything I’ve discovered in my reverse-engineering of the SHSAT points out to what good tutors probably already know: students should, as early as possible, do some practice tests and determine which areas are they struggling with the most. More importantly, they should identify which traps catch them the most, and then work on avoiding them.

This is hard, if not impossible, to do in large classes where students get a general (averaged) feedback focused on the most common problems for most students. Every individual is unique; we all have different learning paths and preferences.

What each student needs is a precise feedback on their own skill and struggle configuration and preparedness.

That’s exactly why I did not stop with this analysis alone: I also developed a battery of tests that accomplish exactly that. I took everything I learned about the inner workings of SHSAT and developed Baseline and Benchmark tests that reflect, as exactly as mathematically possible, the real distribution of complexity, skills, and distractors used by the DOE. Tests on this site are as close to what the actual tests are as they can possibly be.

It took me a whole summer to build it, and I did it initially for my daughter, but it seemed strange to keep it private.

What I also developed is a scientific and precision-based feedback system, with a precise error analysis, visualization of the most common mistakes, and pointers to what the student should focus on the most.

The best part of it is that the full Baseline test, with accompanying error analysis, is completely free. I decided to err on the side of caution and kept the original 114 questions, so the students don’t get a false sense of their preparedness for the test.

You can access the test here

Good luck.