Deterministic verdicts and an LLM judge: how Hover Cloud evaluates a test run

A test run tells you pass or fail. That is not the interesting question. The interesting question is what a failure means: did the UI move and the test need a new selector, did the app actually break, or did someone change a label and the assertion is now stale? A red X can't answer that. A person reading logs can, slowly, one failure at a time.

Hover Cloud answers it in two layers. The first is deterministic and does most of the work. The second is an LLM judge that only gets the cases the first can't decide. Keeping them separate is the whole design.

Layer 1: deterministic assertion evaluation

Most of a verdict comes from facts, not judgment. Four of them:

• Status — did the spec pass, fail, or flake, this run.
• Record — how has this spec behaved across recent runs and branches. A spec already red on mainline is not your PR's fault.
• Coverage — which business lines does the suite actually exercise, read from the .hover/hover-map.md your repo commits.
• Determinism — how often does this spec flip between pass and fail. A flaky spec is a different problem from a real regression.

From these, Hover computes verdicts with plain code and zero AI:

• The Heal Queue classifies each failure by the shape of its error. An action step whose locator no longer resolves is drift (heal the test). A dead app, a 500, a crashed page is a bug (fix the app; the spec is your repro). An assertion that was reached but not satisfied is unclear, because a copy change and a regression look identical from here.
• A PR gets a merge-confidence call per failed spec: pre-existing, flaky, regression, or unknown. That call needs the cross-run, cross-branch history a single CI job never sees, and Cloud has it. Every verdict carries its reason, so you can audit it.

None of this calls a model. It is cheap, it reproduces exactly, and it explains itself. This is the layer that guards the green path, so it must never depend on a model having a good day.

Layer 2: the LLM judge

Determinism runs out at the unclear bucket, and at anything with no single right answer. "Did this screen still do the right thing after the copy changed?" has no assertion that settles it. This is where a judge earns its place.

The rule that keeps it honest: AI only on what already ran. The judge never touches a green build and never authors code. It reads a failure that already happened, against the flow's original recorded intent, and scores it rather than voting pass/fail. Today Hover uses the model to phrase the PR failure summary the way a careful teammate would; the verdict underneath still comes from Layer 1. The judge's next job is the unclear queue: score whether a changed outcome still matches intent, so fewer failures need a human.

An LLM-as-judge is good at reading intent and bad at being a source of truth. So it scores; it doesn't decide the deterministic facts. Put it anywhere else and you pay tokens on every green run and inherit variance you didn't write.

The compounding asset

Every flow you crystallize is a happy path saved as plain Playwright. Do this for a while and the suite stops being a chore and becomes an asset: a regression net that guards the system you already shipped. The more you ship, the more of your product is pinned down by tests that run for free in CI, and the more precisely the two layers above can say whether a new change broke something that used to work.

That is the quiet part of AI-generated code. It arrives faster than anyone can read it, so the only durable defense is a growing net of checks that were cheap to author and cost nothing to run.

The loop: define the business, not the code

Follow the two layers to their conclusion. You stop writing code by hand, and you stop clicking through the app by hand. Instead, when you build a feature, your Claude Code (or any MCP agent) runs a loop:

1. From a clear definition of the business rule, design the regression test first.
2. Generate the code.
3. Push it; CI runs the suite as plain Playwright.
4. Hover's two layers return a verdict: heal the drift, flag the bug, score the unclear.
5. Iterate until the local code and the regression agree.

The human job moves up a level. You don't write the assertion and you don't write the implementation. You define what should be true. The bottleneck becomes the one thing a model can't do for you: saying, clearly, what your business is supposed to do. Hover keeps that definition where it belongs, in the business map and remembered rules committed with your code, so the loop always has a spec of intent to test against.

Where this is today

The deterministic layer is shipped: the drift/bug/unclear classifier, merge-confidence, business-coverage on PRs, and flakiness scoring all run now, on runs your own CI produces. The LLM judge phrases the PR summary today; scoring the unclear queue and closing the full author-first loop is where it's heading. The order is deliberate. Get the deterministic core trustworthy first, then add judgment only where there is genuinely no right answer.