Research6 min read

Why agents shouldn’t grade their own homework.

After our first voxel-world run failed playable acceptance despite 45 passing tests, we tried stricter single-shot review rules. They failed too. Subsequent fresh-context review and repair rounds produced a build that passed the final programmatic acceptance probes.

DeepSeek V4.1 Flash · High thinking · 52-file Python baseline · Single-shot vs. fresh-context review

Persistent findings and an outcome contract did not prevent acceptance failures in these runs.

Three single-shot runs all declared themselves verified and all failed independent playability checks. Separate fresh-context reviewer sessions that wrote executable probes against the finished workspace reproduced the coordinate, collision, and raycast defects missed by the author’s checks.

Three runs reported verified; all failed independent playability checks.

In our first paired voxel experiment, Carthagent finished in 8 minutes 58 seconds ($0.3183) with 45 passing Python tests, yet the rendered terrain was upside down relative to physics (+15 logical height rendered at −15). We wanted to test whether stricter review rules inside the same coding run would catch that defect.

We ran two matched follow-up experiments on the same 52-file baseline, prompt, model (DeepSeek V4.1 Flash, high thinking), and 30-minute limit: one with persistent review findings, and one requiring an explicit outcome contract linking every component to an end-to-end check.

Single-shot runs on the 52-file voxel baseline (DeepSeek V4.1 Flash)
ConditionWall timeVerified usage costSelf-reported statusIndependent playable acceptance
Original CTG run538.50s (8m 58s)$0.3183VerifiedFailed (inverted vertical axis, camera movement mismatch)
Persistent review findings420.51s (7m 01s)$0.3267VerifiedFailed (smoke ran handlers, but ordinary startup crashed; coordinates disagreed)
Mandatory outcome contract407.71s (6m 48s)$0.5188VerifiedFailed (native window launched, but geometry, strafe sign, and safe placement failed)

These are single attempts per condition, not a statistically powered speed or cost comparison. The original run executed concurrently with another agent; the two reruns executed alone.

The mandatory-outcome-contract run cost $0.5188, 63% more than the original run’s $0.3183. Its review retries included invented finding IDs and attempts to resolve findings that were already closed. It still missed the camera-relative strafe error and player-overlapping block placement. Outcome contracts were made opt-in afterward.

The unit tests did not check agreement between coordinate systems.

Code and tests written in one session can encode the same coordinate assumption. For example, separate tests could accept yaw 0 as +X in movement and +Z in the camera without checking that the modules agree.

Same-context self-check versus fresh-context reviewIn same-context testing, one conversation writes both code and tests using the same flawed coordinate assumption. In fresh-context review, a separate session receives only the original task and the finished workspace, and writes independent boundary probes.Same-context self-check1. Author assumes +Y is down in mesh transform2. Author writes test asserting mesh Y == −15Result: 45/45 tests green · World upside downFresh-context reviewer (zero shared history)1. Reads only original prompt + workspace files2. Probes 60 camera angles + AABB overlapResult: Reproduces strafe, raycast & Y bugs
The reviewer received the original task and workspace files, but not the author session’s conversation history.

The stricter runs produced additional review records, but those records did not prevent the startup, coordinate-agreement, movement, and placement failures.

What happened when separate reviewer sessions probed the workspace.

Instead of asking the author session to review itself again, we opened a separate Carthagent session with no shared conversation history. It received only the original task prompt and the finished workspace on disk, with instructions to test the implementation using executable probes.

Round 1: Render and control mismatches.

The first fresh reviewer probed the live scene graph and input loop rather than relying on the reported unit-test results. It reproduced the render and control defects and returned failure scripts to the repair session. When subsequent testing exposed intermittent native graphics output on the host, the repair pass added a documented software-renderer fallback.

Round 2: AABB overlap and raycast edges.

A second fresh reviewer ran against the repaired workspace and found two additional defects: an axis-aligned bounding-box (AABB) overlap flaw that allowed placing blocks inside the player’s volume, and a raycast boundary error at block faces. The implementation model repaired both and added regression tests.

After those review-and-repair rounds, the final fresh review recorded 75 passing Python tests, confirmed native launch, checked image-to-state correspondence, and tested camera-relative movement across 60 angle combinations alongside safe-placement and chunk-isolation probes.

Additional review rounds, time, and token usage.

The experiment used an optional fresh-context review workflow, documented at the time with the experimental ctg review workspace --request-file ... command. This result has three limits:

  • It is not a single-shot win. Passing the final programmatic acceptance probes took multiple review-and-repair rounds and more than the original 30-minute single-run budget.
  • It costs extra tokens. Starting a fresh reviewer means paying to read the repository and run probes a second time. Fresh-context review in Carthagent remains optional rather than mandatory on every edit.
  • It is still not human playtesting. Even the 75-test, 60-angle acceptance pass exercised programmatic probes without physical OS mouse hardware.

Fresh-context probes found defects missed by the author’s checks.

These experiments did not test whether fresh-context review helps with small changes. In the voxel trials, stricter same-context review rules did not catch the defects found by independent acceptance checks. In the later review rounds, fresh sessions wrote probes that reproduced coordinate, overlap, and raycast defects and guided repairs.

Research record: three matched single-shot trials and multi-round fresh-context review/repair experiments using DeepSeek V4.1 Flash on the 52-file Python voxel baseline. Documented in B-icy/Carthagent (docs/review-experiment-results.md).

All research notes