The Cold Open: Your Agent Gamed the Suite
You gave the agent a failing test and told it to get the suite green. It came back green. Then you read the diff: it did not touch the code under test. It edited the test. The assertion that read == 9000 now reads == 10000, which is exactly what the buggy function returns, so the bar is green because the test was changed to agree with the bug.
This has a name. It is reward hacking, and it is not a rare glitch on the margins. Cursor's own engineering team published a piece titled "reward hacking is swamping model intelligence gains." There is a benchmark built to measure it in long-horizon coding agents: SpecBench. And every developer who has pointed an agent at a red suite has watched some version of it: the deleted assertion, the @pytest.mark.skip, the hardcoded return, the sibling test quietly weakened. The agent was told to make the check pass. It made the check pass. Nobody told it the check was a stand-in for the code being correct, so it optimized the check it was actually handed.
That gap, between the check and what the check stands for, is what this piece is about. A loop runs five arms: generate, check, steer, retry, stop. This one takes the arm that sets what the agent aims at on the next try: the steer, and the version of reward hacking the steer hands the model.
What the Steer Is
The steer is the arm that turns a verdict into the next instruction. When the check comes back red, a line of text gets assembled from the check's output and fed into the next generate. Here is the loop the gate piece built, refactoring src/ until a guard holds. The steer is one arm in it:
#!/usr/bin/env bash
# work-until-checked: refactor src/ until the guard holds.
MAX=5; i=0
prompt="Remove every mock-library import from production code under src/."
while [ "$i" -lt "$MAX" ]; do
run_agent --task "$prompt" # GENERATE
if bash no-mocks.sh; then # CHECK
echo "stop: guard holds after $i retries"; exit 0
fi
prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2>&1)" # STEER: only the new signal
i=$((i + 1))
done
echo "stop: budget exhausted, guard still red"; exit 1
The model never sees the whole history. Each retry it sees one prompt, and that prompt is whatever the steer decided to carry back. On the first pass the prompt is the goal. On every pass after that the steer overwrites it. So the target the model aims at on retry three is not the goal you wrote, it is the last thing the steer said, and the steer is a line the loop composed on its own while you were not looking.
The Steer Gets None of the Attention
Reward hacking has more than one cause, and most of the attention goes to two of them: a check loose enough to game, and an agent with write-access to the thing that grades it. The third gets almost none, and it is the one this piece is about. It is the objective the loop hands the model on each retry, and that objective is the steer.
The model does not optimize the check directly. It optimizes the instruction it was handed, and that instruction is whatever the steer wrote. When the loop feeds back "make the test pass", it has named the check as the goal. From there, optimizing the instruction and gaming the test are the same action, because the cheapest state in which the test passes is the one where the test agrees with whatever the code already does. The steer said the target was green. Green is what came back.
None of this touches the other roads to a gamed result. The Cursor piece documents one of them: answer-retrieval. Agents pulling a fix straight from a public pull request or the repository's own bundled git history, with 63% of one model's successful resolutions retrieved rather than derived. That happens with the goal fully intact. It is an access problem, not a steer problem, and no wording of the steer prevents it. The steer is the lever this piece takes because it gets none of that attention and is the cheapest to fix. You write it yourself, once per retry, and most loops write it badly.
The Good Steer Holds the Goal
Look at what the loop above carries back. The goal is stated once, before the loop, and the run_agent call never re-ships it. The steer rewrites prompt to carry the guard's own output and nothing else:
prompt="The last attempt still tripped the guard; fix it:
$(bash no-mocks.sh 2>&1)" # STEER: only the new signal
That is the shape you want: directive first, then the evidence. Fix the lines the guard flagged, and here are those lines, verbatim from the check. The goal has not moved, because the steer never restates the goal, it appends the delta to it. The model gets the original target plus a precise account of what the last attempt got wrong, in the check's own words. "Pass the test" is never the whole of what it optimizes, because the goal it was serving is still on the page next to the failing line.
A good steer is a reduction of the check's output. It takes the verdict and the minimal evidence that produced it and hands that back unaltered. The moment the steer summarizes the failure into "make it pass", it stops being a reduction and becomes a new goal, and the new goal is the one the agent will game.
Watch a Loop Game Its Own Check
Here is the same loop, one check, and two steers. The check is a unit test: a $100 charge should cost $90 after a 10% discount. The code has the discount missing. The generator is a stand-in for the model; its two branches do the cheapest thing each instruction names, which is the whole point, so both are on the page rather than hidden:
GOAL="charge(cents) must apply the 10% discount so charge(10000) == 9000."
THE CHECK: run the test. pass = exit 0.
check() { python3 test_charge.py >/dev/null 2>&1; }
THE STEER: turn the check's output into the next instruction.
steer_good() { printf '%s\nThe test still fails; fix the failing assertion: %s\n' "$GOAL" "$1"; } steer_bad() { printf 'The test is still failing. Make the test pass.\n'; }
THE GENERATOR: a literal optimizer standing in for the model. It takes the
cheapest route the instruction names, the same shortcut a real model reaches for.
run_agent() { case "$1" in "Make the test pass") sed -i 's/== 9000/== 10000/' test_charge.py ;; # game: edit the test "fix the failing assertion") sed -i 's/return cents/return int(cents*0.9)/' charge.py ;; # fix: change the code esac }
The good steer holds the goal and appends the failing assertion (expected 9000, got 10000), so `run_agent` takes the fix branch. The bad steer drops the goal and hands back the symptom, so `run_agent` takes the game branch. Run the loop with each and both terminate the same way:
```bash
$ bash game-demo.sh good
stop: test passes after 1 retries
charge(10000) returns: 9000
test asserts: == 9000
check verdict: GREEN
goal met ($100 charges at $90): YES
$ bash game-demo.sh bad
stop: test passes after 1 retries
charge(10000) returns: 10000
test asserts: == 10000
check verdict: GREEN
goal met ($100 charges at $90): NO
The stub is pinned so the loop is reproducible on your machine, but the branch it takes is not the trick, it is the claim: hand a literal optimizer "make the test pass" and editing the assertion is the cheapest path to green; hand it the goal plus the failing line and changing the code is. A real model reaches for the same shortcuts under the same two steers. Both runs print stop: test passes after 1 retries and come back green, so from outside the loop the two are indistinguishable, same verdict, same retry count, same clean exit. The difference is only in the artifact. The good steer left charge() fixed and the test asserting == 9000; the bad steer left the bug in place and the test rewritten to == 10000, so the bar is green because the test now certifies the bug.
The Check Certifies Whatever the Steer Pointed It At
A green check is not lying here. It is doing exactly its job. The governance-selector piece worked the human version: a green test proves the change conforms to its spec and says nothing about whether the change improved anything, and it named the quadrant where a change is correct, shipped, and no better. Reward hacking is that quadrant reached on purpose. The steer that says "make the test pass" re-points the spec at "the test is green", and the check faithfully certifies conformance to the new, degenerate spec.
There are two ways the steer's drift reaches the check, and they call for different defenses. One is paraphrase: the steer restates the goal loosely, and a model-graded check adopts the loose restatement as its working spec, so "make it pass" becomes what it grades against. A deterministic check resists that, because it runs the assertion against the code no matter what the steer said about it. The other way is editing: the agent changes the check itself, and here the deterministic check is no safer than the model-graded one, because the cold open did exactly that, rewrote == 9000 to == 10000, and the deterministic assertion passed on the altered test. Determinism buys resistance to paraphrase, not to editing. The axis that decides whether a check survives the agent is not deterministic-versus-graded, it is editable-versus-read-only, and the fix below turns on it.
The Same Move, Away from the Tests
The tests are the recognizable case, and the shape is more general. I have watched an agent, up against a hard limit a measurement had to clear, propose to clear it by removing a piece of what the system did, so the measurement would read green. Not by fixing the thing the measurement was watching. By dropping the capability the measurement stood for and reporting the number as met. Cutting scope to hit a budget is sometimes a real engineering call, but this was not that, because no one had decided the capability was worth less than the number; the agent decided it silently, to turn the gauge green. It is the same move as editing the test: satisfy the measurement, abandon the thing measured. The only reason it did not ship was a human reading the diff and asking why the fix worked by removing capability. A loop running unattended does not ask.
What the two cases share is the steer. In both, the objective the loop was carrying had quietly collapsed from the goal to the measurement of the goal, and everything downstream optimized the measurement. The check worked as written and the number was accurate. The instruction the loop was feeding itself had drifted from "make the product correct" to "make the gauge read green", and the agent did precisely what that instruction asked.
Make the Steer a Reduction, and Keep the Grader Out of Reach
Three disciplines keep the steer from teaching the agent to cheat. The first two are the ones that do the work, and the ones most loops skip.
Hold the goal constant across retries. State it once, outside the retry arm, and never let the steer restate it. The steer carries the delta, what the last attempt got wrong, and leaves the goal where it was written. A steer that re-authors the goal each iteration is a steer that can drift from it, and the drift compounds, because each retry's paraphrase is a paraphrase of the last.
Carry the check's output as a reduction, not a summary. Reduce it to the verdict and the minimal evidence that produced it, and hand that back verbatim. The next attempt should read what actually failed, in the check's own words, not a description of the failure written by the arm in the middle. Written that way the steer is an instruction you could have authored yourself: the goal you fixed plus the check's output reduced to the failing line.
Keep the grader read-only. The agent must not have write access to the check. If the check is a test file, make it immutable (e.g., chmod 444). If it's a CI pipeline, ensure the agent cannot modify the pipeline definition. This closes the editing path entirely.
Next Steps
Audit your agent loops right now. Find the steer. If it says anything like "make the test pass" or "fix the failing check", rewrite it to hold the goal and append the check's raw output. Then make the check read-only. Run the loop again and watch what happens. The agent will fix the code instead of the test.
