Ibrahim AbuAlhaol
A new hire tells you, "this part is hard." You explain it again, more slowly, with a better example. A week later they are still stuck.
It rarely works, because they are not stuck on the page in front of them. They are missing one earlier step, and they cannot see which one. Neither can you: the step is so familiar to you that you stopped noticing it years ago.
AI changes the cost of finding that step. A tutor that asks a few well-chosen questions before it explains anything can locate it in minutes, the way a good mentor does in an hour. Left on its defaults, the same assistant does the opposite: it hands over the answer, and the clue disappears.
Stuck usually means one step back
In 1968 the psychologist David Ausubel put the whole idea in two sentences: "The most important single factor influencing learning is what the learner already knows. Ascertain this and teach him accordingly."1
Take a backend engineer three weeks into an AI team. Their new model scores slightly higher than the old one, and on a second test set the order flips. The team's guide says to "report a confidence interval." They read the page twice and still cannot say whether the gain is real. The guide is not wrong. It assumes they already know how much a score wobbles from one test set to the next, and that is the step they lack.
Knowledge Space Theory, a mathematical account of learning proposed in 1985, gives this a precise shape.2 A subject is a set of topics, and some topics need others first. What a person knows is the exact set of topics they can do, not a score. That is why a skills matrix misleads: "statistics: 3 of 5" cannot say which topic is missing. In the arithmetic map of ALEKS, a tutoring system built on the theory, 1,668 different combinations each contain exactly 75 topics.3 Same score, different gaps.
Nobody can see the gap from inside
The missing step does not feel missing. The engineer can call a library function that prints a confidence interval, and that feels like understanding. Psychologists call this the illusion of explanatory depth: people rate their own understanding of how something works much lower once they try to explain it.4
The expert who wrote the guide cannot see the gap either. The step is so automatic for them that it never made it onto the page. Nor is this a beginner's problem. When 120 psychology researchers and 442 students judged six false statements about confidence intervals, both groups accepted more than three on average.5
I have been that expert, as a teaching assistant for graduate statistics at the University of Mississippi. When a student said "this is hard," my instinct was to explain the same idea again, better. It was the wrong instinct.
Explaining the same topic again cannot fix a gap that sits underneath it. Do not ask a stuck person whether they understand. Ask a question with one right answer.
Three questions find the missing step
Because topics depend on each other, one answer tells you about more than one topic. Get a question on confidence intervals right and you almost certainly hold the steps beneath it. Miss it, and the gap is at that step or below.
So the search is short. Start in the middle of the chain beneath the goal, move down after a miss and up after a right answer, and stop at the first step missed whose own earlier steps are answered right. For our engineer, three questions do it. They miss confidence intervals, miss the step below, and answer the step below that correctly. The missing step sits in between: how far a score typically sits from the model's true accuracy.
Answers are evidence, not proof. Anyone can slip on a topic they know or guess right on one they do not, so one more question on the suspected step is cheap insurance. ALEKS runs a sturdier version of the same search; in one case its developers describe, 24 questions were enough to place a student among 57,147 possible combinations.3
For the technical reader: the numbers and a tutor prompt
The engineer's numbers: 91.2% against 90.8% accuracy on the same 1,000 examples, a lead of four examples. Only examples the two models disagree on carry evidence, and even if model B won all four disagreements, an exact paired test (McNemar's) gives p of about 0.13. The habit I learned measuring wireless error rates applies: count the events, not the percentages.
The chain beneath the goal runs: comparing two models, paired differences, confidence intervals, the standard error, sampling variability, with train-test splits on a second branch. The standard error of 91% on 1,000 examples is about 0.9 points, so a 95% interval runs from about 89% to 93%. In the theory's terms, the topics a learner is ready to learn form the outer fringe, and the search above halves one chain; ALEKS instead weighs every possible knowledge state and asks the question a learner has about an even chance of getting right.3
The prompt below sets up a tutor for one guide page. Swap in your chain. The answer keys come from your own worked examples, never from the model.
You tutor one page of our team's onboarding guide, on practice
questions only. Before you explain anything, find the step I am
missing.
Chain for this page, goal first:
1. Comparing two models
Ask: Model A scores 90.8% and model B 91.2% on the same
1,000 test examples. Is B better?
Key: Not shown. B leads by four examples, and only the
examples the models disagree on count. Even if B won
all four, an exact paired test gives p of about 0.13.
2. Paired differences
Ask: Two models score the same 1,000 examples. Which
examples decide whether one beats the other?
Key: Only those they disagree on: how many, and how they
split.
3. Confidence intervals
Ask: 91% accuracy on 1,000 test examples. What is a 95%
confidence interval for the model's true accuracy?
Key: About 89% to 93%: plus or minus two standard errors.
4. Standard error
Ask: About how far is that 91% typically from the model's
true accuracy?
Key: About 0.9 percentage points: the square root of
0.91 x 0.09 / 1,000.
5. Sampling variability
Ask: Why would the same model score differently on a new
random test set?
Key: A new random sample of examples gives a new score.
Rules:
* Ask one question at a time, starting with the middle topic
of the chain. Give no hints or answers yet.
* Skip any topic my earlier answers already show I can do.
* After each answer, ask about the middle of the part of the
chain that the answer leaves open.
* Stop at the first topic I miss whose topics below I get
right. Confirm it with one more question.
* Then teach only that topic, with hints and the worked
examples supplied.
* Mark answers only against the keys supplied.
* End with one line for the log, with no names:
"Missing step: [topic]".Make the AI ask before it explains
A default AI assistant does the opposite of a good mentor. Paste in the two scores and it will declare a winner, or write a test the engineer runs without reading. The stall disappears and the gap stays. A stall tells you where the missing step is, and an answer handed over erases the clue.
One-to-one tutoring has always found missing steps, at a price. In 1984 Benjamin Bloom reported tutored students scoring about two standard deviations above a conventional class, a gain "too costly for most societies to bear on a large scale."6 AI makes the conversation cheap. It does not make it diagnostic unless you set it up that way.
The early trials show how. In a randomized trial with nearly 1,000 high school students in Turkey, a GPT-4 tutor that held the solutions and was told to give hints, not answers, largely avoided the harm unrestricted access did to exam scores, though it did not raise them.7 In a Harvard physics course, a tutor built from expert-written instructions more than doubled median learning gains over an active-learning class, on a test taken right after the lesson.8
Neither tutor was a chatbot left on its defaults. Each was given worked solutions and told how to teach, and that part is the leader's job. The evidence is still young: both trials were short, each at one institution, and an independent review found ALEKS about as effective as classroom teaching overall and most useful as a supplement.9
Asking the right three questions takes a map of what depends on what, and most teams have never drawn one; The Map Nobody Drew shows how. Unrestricted help can also hide what people learned at all, the subject of The Lucky Guess Problem.
What leaders should do
- Pick the one onboarding page where most new hires stall, list the steps beneath it, and write one question with one right answer for each. The team lead owns it, and it fits in an afternoon.
- Replace "let me explain again" in one-on-ones: when a hire says something is hard, ask one question per earlier step, then spend the hour on the first step they miss.
- Set up an AI tutor for that page, separate from the coding assistant, using the prompt above. Log each missing step by topic, never by person, and add any step three hires miss to the guide.
- In a course, pilot the hint-only tutor in one section, keep its questions ungraded, and compare an unaided quiz against another section. The course coordinator owns the pilot and its report.
