What Single-Pass Answering Actually Fails At

What Single-Pass Answering Actually Fails At

A single-pass model reads a prompt and generates its answer token by token, each new token conditioned on the ones before it. There is no separate stage where the model decomposes the question, checks a claim, or reconsiders an early commitment. Everything happens in one forward sweep. For a factual lookup that is fine, because the answer is a short path from the question. For a query that turns on several interacting factors, it collapses.

Consider a question that depends on three conditions: a pricing rule, a regional exception, and a timing constraint. Each shifts the answer, and they interact. A one-shot generation has to fold all three into a single fluent sentence while it is still deciding what the first clause should say. Because early tokens constrain later ones, the model tends to lock onto whichever factor is most salient in the prompt and produce a response shaped around that one, treating the others as decoration. The exception gets a passing mention; the timing constraint gets dropped. Nothing in the mechanism forces the model to hold all three open and weigh them against each other.

The failure is not that the answer sounds wrong. It sounds confident and reads smoothly, because fluency and correctness are produced by the same process and the model has no internal signal separating them. It optimizes for a plausible continuation, not a verified conclusion. So the visible symptom is a shallow answer wearing the surface of a thorough one.

This is why the collapse is dangerous rather than obvious. A halting, uncertain answer invites scrutiny. A smooth answer that silently ignored two of the three factors that actually determined the outcome does not. The reader accepts it precisely because nothing in it signals that anything was skipped.

How Multi-Step Reasoning Decomposes a Query

Multi-step reasoning replaces that single sweep with an explicit loop. The system first decomposes the query into sub-questions, each one narrow enough to answer against a single source of truth. For the pricing-plus-exception-plus-timing case, it produces roughly three separate probes: what is the base rule, does a regional exception apply, and does the timing constraint change the result. Each probe is a discrete unit of work rather than a clause competing for room in one sentence.

Then each sub-question triggers its own retrieval. This is the difference that matters. Instead of one semantic search returning the single most-salient passage, the system runs a search per step, so the regional exception gets its own lookup even when it is less prominent than the base rule in the original phrasing. A factor that would have been drowned out now has a dedicated pass to surface the exact document that governs it.

Between steps the system holds intermediate state: the answer to sub-question one becomes a constraint on sub-question two. If the regional exception fires, that result narrows what counts as a valid answer to the timing probe. The reasoning is sequential, not parallel, precisely so that early findings can gate later ones rather than being averaged into a blur.

The final stage is recombination. The intermediate results are weighed together into one conclusion, and because they exist as separate, inspectable facts, a contradiction between them is visible instead of silently smoothed over. The tradeoff is cost: three or four decomposition-and-retrieval cycles take longer and consume more compute than a single generation. The payoff is that no determining factor gets treated as decoration, because each one was answered on its own before anything was combined.

Weighing Factors That Pull in Opposite Directions

Decomposition splits a query into parts; weighing reconciles parts that cannot all win. These are different operations. Two sub-answers can each be individually correct and still point in opposite directions: the pricing rule favors one plan, the regional exception favors another, and neither is wrong. Recombination cannot just concatenate them. It has to score.

The mechanism is a comparison across candidate answers rather than a march toward a single one. The system holds each viable conclusion open as a candidate and attaches to it the factors that support and undercut it. A candidate that satisfies the pricing rule but violates a hard constraint is not averaged toward the middle; it is disqualified, because some factors are gates and others are mere preferences. This is the distinction that shallow answering loses: it treats every factor as a soft nudge on one fluent sentence, so a hard constraint and a stylistic preference exert the same weak pull. Structured weighing separates the two. Hard constraints eliminate candidates outright. Soft factors then rank whatever survives.

The failure mode this prevents is the false compromise, where a model splits the difference between two options and produces an answer that honors neither. If a regional exception forbids a plan entirely, the correct response is not a hedged blend that mentions it apologetically; it is the elimination of that plan from the running before ranking begins.

What the reader gets back is not a verdict with the reasoning hidden. It is a judgment that names which factors decided the outcome and which were overridden, so a conclusion that trades a lower price against a disqualifying compliance rule shows the tradeoff explicitly rather than presenting the surviving option as if nothing competed with it.

Where Multi-Step Reasoning Breaks and What It Costs

The same structure that fixes shallow answering introduces its own failures, and they compound. Error propagation is the first. Because each step conditions the next, a wrong answer to sub-question one does not stay contained. If the regional-exception probe retrieves a stale document and concludes the exception fires when it does not, that false result gets treated as a hard gate and disqualifies the correct plan before ranking ever runs. The chain does not question its own premises; it inherits them. A single bad retrieval early can poison every downstream step, and the final answer looks just as structured and confident as a correct one.

The second cost is multiplicative, not additive. Each step is its own retrieval plus its own generation, so a four-step chain runs roughly four semantic searches and four model passes instead of one. Latency stacks the same way, and because steps are sequential by design, they cannot be parallelized away without breaking the gating that made the decomposition worth doing. A query that a single pass answers in one sweep now waits through several, and the caller feels each one.

The most dangerous failure is the confident intermediate. A step can hallucinate a sub-answer with the same fluency the whole system was built to interrupt, then hand that fabrication forward as an established fact. Downstream steps have no way to know it was invented; they treat it as ground truth and build on it. This is why grading intermediate steps matters rather than only the final output. A composite that looks clean can rest on a fabricated middle, which is precisely why partial credit should attach only to sub-steps whose effect is independently verified, not to steps that merely sound resolved.

How to Measure Whether the Reasoning Is Actually Good

The first split a decision-maker should demand is deterministic checks versus judged quality, because they answer different questions and fail differently. Deterministic checks are binary and never sent to a judge: did the system prompt leak, yes or no; did a secret pattern appear; did the output match the required schema; did latency clear its bound. These are cheap, reproducible, and unfakeable. Judged quality covers whether the reasoning was actually sound, and that requires an independent frontier judge rather than the model being graded, because a system scoring its own work cannot be read as anything but self-interested.

Longer output is the easiest thing to fake and the least informative to measure, so a good evaluation never counts tokens. It probes across categories. Baseline probes confirm the system handles the ordinary case. Adversarial probes, at least a third of the set, try to trip the decomposition into a wrong gate. Long-context probes bury the deciding fact deep in a large document and check whether the per-step retrieval actually surfaces it rather than the salient distractor. Execution probes verify effects in a controlled sandbox and causally attribute them to the agent's own tool call, so a passing result cannot be borrowed from something else in the environment.

Grounding attribution is the check that catches the confident intermediate: every claim in the chain must trace to a retrieved source, and a step that sounds resolved but cites nothing scores as a fabrication regardless of how clean the final answer reads.

Finally, run each probe three to five times and report the mean with a confidence interval, not a single lucky pass. A system that answers well once and poorly twice is not reasoning well; it is producing variance you happened to sample favorably.