LLM as a Judge: Did the Coding Agent Actually Fix It?

You give a coding agent a bug and say, “This is the problem. Fix it.”

The agent works on it, changes a few files, and comes back with “Implemented.”

Okay, good. But did it actually fix the problem?

You can run the application, open your browser, and try the behavior yourself. You can also run automated tests. Another check we can stitch into that workflow is an LLM as a judge.

Let’s reason this out.

We give the LLM an output, a rubric describing what good looks like, and evidence. It assesses where the result meets those criteria, falls short, or still needs verification.

1. Start with the issue

Describe the bug → coding agent → proposed fix.

Say you enter the wrong password, click Log in, and the loading spinner keeps spinning.

You ask the coding agent to fix it.

The agent returns a change. That gives us something to assess.

Define what “fixed” means

Before accepting it, we put together what “fixed” should mean:

• A wrong password shows a clear error.
• The spinner stops when the request finishes.
• You can correct the password and try again.
• A correct password still logs you in.

That is our rubric. Notice the last check: fixing the broken path should preserve the path that already works.

2. Give the judge something to assess

Proposed fix + rubric + evidence → judge’s findings.

Now we stitch together three inputs:

• The proposed fix: what the coding agent changed.
• The rubric: what a successful fix must achieve.
• The evidence: what was actually checked.

The evidence might include code changes, automated test results, or a recording of the login flow.

The judge dissects that material and marks each check pass, fail, or unable to verify, with reasons.

Keep the evidence visible

Suppose a test shows that the spinner stops. Good. But if nothing checks whether the user can retry, that requirement still needs verification.

Reading code and watching the application work provide different kinds of evidence.

A useful assessment makes those limits visible. “Looks good” alone doesn’t tell us enough.

3. Compare the findings

Compare both reviews, then accept the fix or investigate.

Next, you try the login flow yourself and compare your findings with the judge’s, using the same rubric.

Imagine the judge says retrying should work, but you discover that the button stays disabled after a failed attempt.

The spinner is fixed; the user is still stuck.

Now we have a gap to dissect. Did the judge miss something? Was its evidence incomplete? Did the requirement need a clearer check?

Human review can miss things too, so a disagreement is a reason to investigate.

We send the specific failed check and what we observed back to the coding agent, then assess the next fix.

4. Stitch the whole workflow together

The complete workflow, including the feedback loop.

This is the full connection: describe the issue, produce a fix, assess it against the rubric and evidence, compare findings, and decide what still needs work.

One thing to keep in mind

You can use the same LLM to review its own work, but it may repeat the assumptions it made during implementation. A separate review can help; it still needs evidence.

The rubric can be precise and testable. The LLM’s judgment can still be wrong.

For me, the useful part is that pause after “Implemented.” We put together our expectations, dissect the result, and verify the behavior.

“Implemented” gives us something to assess. Evidence helps us decide whether the fix works.