Evaluator-optimizer pattern: checking and revising answers

The evaluator-optimizer pattern drafts an answer, checks it, and sends failures back for revision. The key is what the checking step can verify. Format checks can catch a broken response; tests can catch failing code; looking up the source can reveal missing evidence. A second model giving its opinion without a checkable standard can repeat the generator's mistake. Define what counts as a passing answer and set a limit on revision attempts. This is one of the eight agent patterns.

TL;DR An evaluator helps when it checks a draft against explicit criteria, such as tests, a schema, or evidence. Give it the draft and the standard, return specific failures, and stop after a fixed number of attempts. A separate model without a useful check does not guarantee improvement. Each extra pass adds cost and latency.

What the pattern is

The generator produces a candidate: code, a structured record, or a draft answer. The evaluator checks it against stated criteria and returns specific feedback. The generator revises. The loop ends on acceptance or at an attempt cap. The evaluator can be a model, a deterministic check, or both.

Choose a check that matches the failure you need to catch. A schema can verify format but not factual truth. Tests can show that code passes the cases you supplied, not that every behavior is correct. Retrieved evidence can support a claim, but the evaluator still has to compare the claim to the source.

Why the evidence looks contradictory

Read the literature and you get two stories that seem to disagree. Reflexion, which pairs a generator with verbal self-reflection between attempts, reports 91% pass@1 on the HumanEval coding benchmark against a GPT-4 baseline of 80% (Reflexion paper). That is a large jump from letting a model react to its own prior attempt.

Then a separate ICLR 2024 paper argues that large language models cannot yet self-correct their reasoning, finding that without external feedback they struggle to fix their own answers, and that performance sometimes degrades after a round of self-correction (ICLR 2024 paper).

Both results are correct, and they are not really in conflict. Reflexion's loop is grounded in unit-test results: the reflection step reacts to something objective, a test that passed or failed, not to the model's private sense of whether it did well. The ICLR paper isolates the other case, pure introspection, where the model has no external anchor and is asked to grade its own logic. Pure introspection is the version that fails. The difference between the two papers is not the technique. It is whether a checkable signal was in the loop.

What it looks like in code

This example asks a model for a JSON contact record, then uses Pydantic to check that the required string fields are present. Validation errors become feedback. The cap stops after three drafts. This only verifies the record's shape, not whether its details are true or its email address is valid.

from typing_extensions import TypedDict
from pydantic import BaseModel, ValidationError
from langgraph.graph import StateGraph, START, END
 
class Contact(BaseModel):
    name: str
    email: str
 
class State(TypedDict):
    request: str
    draft: str
    feedback: str
    attempts: int
    grade: str
 
def generate(state: State):
    prompt = f"{state['request']}\nReturn JSON. Fix: {state.get('feedback', '')}"
    return {
        "draft": llm.invoke(prompt).content,
        "attempts": state.get("attempts", 0) + 1,
    }
 
def evaluate(state: State):
    try:
        Contact.model_validate_json(state["draft"])
        return {"grade": "accepted", "feedback": ""}
    except ValidationError as error:
        return {"grade": "rejected", "feedback": str(error)}
 
def route_back(state: State):
    if state["grade"] == "accepted" or state["attempts"] >= 3:
        return END
    return "generate"
 
builder = StateGraph(State)
builder.add_node("generate", generate)
builder.add_node("evaluate", evaluate)
builder.add_edge(START, "generate")
builder.add_edge("generate", "evaluate")
builder.add_conditional_edges("evaluate", route_back)
optimizer = builder.compile()

If the third draft is still rejected, the graph stops with grade set to rejected. The caller must treat that as a failed attempt, not an accepted contact.

The graph structure follows LangGraph's evaluator-optimizer workflow. Pydantic supplies the concrete check here.

When the reviewer actually helps

The pattern works best when the critic has an external, checkable signal and gets a clean context rather than the writer's history. The signal is the part people skip. Tests, a schema, a type checker, or a second retrieval all give the critic a way to be right or wrong about something outside the model's own head. That is what converts "review" from opinion into measurement.

The clean context matters almost as much. Separating the reviewer's context can reduce exposure to the generator's assumptions. Give it the artifact, requirements, and evidence needed to check the result; a fresh context alone does not make its judgment correct. Cognition reports that its clean-context reviewer catches an average of two bugs per pull request, 58% of them severe (Cognition). That report describes the reviewer configuration used in their workflow, not a guarantee that every fresh-context reviewer will find the same defects.

When it adds nothing

The loop fails in two recognizable ways. The first is intrinsic critique: a model grading its own reasoning with no external anchor. This is the case the ICLR 2024 paper pins down (study), and it is where you can burn tokens making an answer worse. If the only thing the critic consults is its own judgment, you have added latency and cost without adding information.

The second failure is mixing evaluation and revision without a clear boundary. The evaluator may run tests or query a source to check the draft. If it silently edits the candidate as well, a passing result no longer tells you what the generator produced or which change helped. Return failures to the generator and evaluate the revised artifact again.

What it costs

Every rejected draft triggers another generation. An LLM evaluator also adds a model call on each attempt; a schema check or test may be much cheaper. Measure the actual calls, test time, and pass rate against a one-pass baseline. The attempt cap bounds how long the loop can run.

If attempts pile up at the cap, inspect the feedback and the task. More rounds cannot repair an evaluator that checks the wrong thing.

What to trace

The evaluator loop is a scoring system, so trace whether the score means anything and whether the loop is converging:

• Iterations to acceptance. The distribution, not the average. A pile-up at the cap means the signal is too weak to converge, and you are paying for churn.
• First-pass acceptance rate. Track how often the first draft passes, then compare accepted and rejected samples with labeled outcomes. A high or low rate alone does not show whether the evaluator is right.
• False accepts. Output the critic passed that later failed a real check downstream. This is the number that says the critic's signal is not as external as you thought, and it is the one that matters most.
• Score against a golden set. Periodically grade the critic itself on cases with known answers, so evaluator drift shows up before it quietly passes bad work.

The decision rule

Add a reviewer only when it clears this bar:

• External signal. The critic can check the output against something outside the model: tests, a schema, a type checker, a compiler, a second retrieval. No signal, no reviewer.
• Clean context. The reviewer sees the artifact and the spec, not the generator's full history.
• Check, then return feedback. The evaluator may use read-only or verification tools, but the generator owns revision.
• Hard cap. A fixed attempt limit, because cost and latency grow with each rejected draft.

Miss the first bullet and the rest does not matter. A reviewer without an external signal is a more expensive way to get the same answer.

What to do about it

Start with one failure that can be checked. For structured output, validate the schema. For code, run focused tests. For a sourced answer, compare claims with retrieved evidence. Return specific failures to the generator, cap the attempts, and record whether the revised artifact passed. A clean pass only supports the property the check actually measured.

For the wider map of how this fits with the other patterns, see the pillar, The eight agent patterns in 2026. When your review loop starts wanting to hand subtasks to specialized workers, that is a different shape, covered in orchestrator and worker agents.

Sources

• Reflexion: Language agents with verbal reinforcement learning
• ICLR 2024: Large language models cannot self-correct reasoning yet
• Cognition: Multi-agents working