Two coding agents built a voxel world. Neither was ready to play.
CTG finished sooner, used fewer tokens, and cost less. Then we checked whether the player could actually move through the world.
Two completed builds. Zero playable acceptance passes.
This is a single paired experiment, not a general ranking. Both submissions contain real work. Both also contain bugs serious enough to break the requested experience.
One task, two fresh starts.
We gave CTG and plain Pi identical copies of an existing Python game-reference project and the same prompt: build a small, native Minecraft-style world using Panda3D. No browser. No prerecorded demo.
| World | Approximately 64 × 64 blocks; seeded grass, dirt, stone and trees. |
|---|---|
| Play | Mouse-look, WASD, gravity, jump, collisions, safe spawn, block placement and removal. |
| Efficiency | Chunked meshes; omit hidden faces; rebuild only affected chunks. |
| Evidence | Tests, launch instructions, a screenshot and at least 300 rendered smoke-test frames. |
| Run conditions | Fresh sessions, high thinking, concurrent starts, 30-minute limit, no corrective steering. |
CTG used its isolated delivery workflow. Pi ran without optional extensions or skills. The comparison tests these complete workflows on one task—not the model against itself under perfectly interchangeable conditions.
CTG did less work to reach its finish line.
The measured advantage is clear. The meaning of “finished” is where the experiment gets interesting.
Coding time
25.2% less time
Wall-clock time, including tool execution.
Verified usage cost
31.9% lower cost
USD for these coding runs; excludes evaluation.
Cumulative tokens
83.6% fewer tokens
Includes cached input repeated across requests.
Each chart starts at zero and uses its own scale. Differences in execution conditions and pricing mean these observations do not isolate a causal harness advantage. One attempt per tool is not enough to establish repeatability.
The token and work breakdown
| Measure | CTG | Pi |
|---|---|---|
| Uncached input | 254,304 | 349,691 |
| Cached input | 1,673,088 | 11,592,064 |
| Output, including reasoning | 48,623 | 108,304 |
| Reasoning subset | 15,767 | 64,274 |
| Total tokens | 1,976,015 | 12,050,059 |
| Model responses | 41 | 127 |
| Tool calls | 57 | 136 |
| Flagged tool errors | 9 | 3 |
Reasoning is already included in output. Cached input counts repeated context, not newly generated content. Flagged errors do not capture every failure concealed inside a successful shell command.
Actual output, not a concept render.
Both built deterministic terrain, chunked meshes, and a native renderer. These are the first screenshots from fresh independent runs: seed 42, 600 reported frames each.
The evaluator could not visually inspect images in this session. These captures are included for reader inspection; conclusions below come from reproducible code and behavior checks. Pi also reported intermittent blank captures during development.
The bugs live between the components.
Terrain generation passed. Collision math passed. The missing checks were at the boundaries: world coordinates to rendered coordinates, and camera direction to player movement.
CTG: the world is on the wrong side.
The terrain’s rotation turns logical height +15 into rendered height −15. The camera uses positive height. The visible ground and the ground the player collides with are therefore in different places.
Independent checks also found reversed camera-relative movement at some headings, two wrongly wound face directions, and placement that could put a block inside the player. In that placement fixture, 60 ticks of movement and jumping left the player stuck.
Pi: walking loses a dimension.
Pi computes a horizontal camera vector, zeroes its vertical component, then reads that zero as the player’s other horizontal axis. World Z never changes. At the initial heading, pressing W produces no displacement.
Its rendering conversion and placement guards were otherwise more coherent. It rejected player-overlapping blocks and provided a solid boundary and floor. Those strengths cannot compensate for broken traversal.
Green tests. Red acceptance.
| Check | CTG | Pi |
|---|---|---|
| Python test suite | 45 passed | 46 passed |
| Data/document tests | 15 passed | 15 passed |
| Fresh 600-frame runs | 3 completed | 3 completed |
| Seeded terrain / supported spawn | Passed on 6 seeds | Passed on 6 seeds |
| Hidden-face oracle | Exact match | Exact match |
| Basic gravity, jump, collision | Passed independent fixtures | Passed independent fixtures |
| Movement / camera integration | Failed | Failed |
| Player-safe block placement | Failed | Passed tested fixture |
| Playable acceptance | Failed | Failed |
CTG’s pure physics module was easier to test and had meaningful gameplay unit tests. Pi’s tests concentrated on storage, terrain, and meshing; movement and interaction were under-covered. Neither suite checked the coordinate conversions that connected those parts.
The independent review exercised production calculations without OS mouse input, checked six generation seeds, tested placement overlap and reach limits, and compared emitted faces against a separate oracle. It was not a manual play session. No source repairs were applied before scoring.
Why we did not publish an FPS winner
We ran three sequential 600-frame smoke tests per implementation after the coding runs ended. CTG’s reported p95 intervals ranged from 5.47–5.78 ms, but its first interval included seconds of setup and smoke mode skipped physics. Pi stepped the application and explicitly rendered again, so its reported iterations were not equivalent to CTG’s frames.
The world heights, geometry, and rendering workloads also differed. Both completing a rendering loop is useful evidence. Calling one the faster game from those timings would overstate what was measured.
Friction worth recording.
- Setup was not frictionless. Both had to work around missing Python environment bootstrapping support.
- CTG spent calls negotiating its workflow. Planning guards, tool-argument errors, and a review rejection account for part of its nine flagged failures.
- Pi debugged a real meshing error. A tuple/integer indexing mismatch appeared during development. It also reported intermittent blank screenshots.
- Existing tests were not an oracle. The original ore generator passed its suite but changed output across Python hash seeds. Both retained this pre-existing defect; it is separate from their new terrain generators.
- The final claims needed checking. CTG described 45 tests as existing regression coverage; the total was 24 original plus 21 added. Pi changed a cleanup-test exclusion for its virtual environment. We recorded both rather than taking completion summaries at face value.
Test the handoffs, not just the parts.
The next version of this task should make the missed behavior executable. A small number of integration checks would have been more valuable than another successful screenshot write.
| Add this check | What it catches |
|---|---|
| Transform a known block and compare its rendered height with collision height. | CTG’s inverted vertical axis. |
| Hold W and D at five camera headings; assert both horizontal axes are reachable. | Pi’s lost axis and CTG’s direction mismatch. |
| Cast through the crosshair and verify the hit corresponds to the camera’s forward vector. | Disagreement between targeting and what the player sees. |
| Try placing a block into the player’s bounding box. | Self-trapping placement. |
| Check triangle winding against outward normals. | Wrong-sided faces hidden by count-only tests. |
The useful result is twofold. CTG reached its declared finish with substantially less recorded work in this run. But its declared checks missed critical behavior, just as Pi’s tests did. Harness structure can organize evidence; it cannot make an incomplete acceptance target complete.
A stronger follow-up would repeat paired trials, normalize execution limits, and score input-driven gameplay before comparing cost per successful delivery. In this experiment, there were no successful playable deliveries to put in that denominator.
The result, with its limits intact.
Use CTG’s lower time, cost, and token totals as an observation about this attempt. Use the failures as a concrete guide to better acceptance tests. Neither “the agent finished” nor “the tests passed” answered the original question: can someone play the world?
Research date: October 13, 2026. One concurrent coding run per tool, high thinking, 30-minute limit. Independent rendering repeated three times per submission. Cost figures are verified usage charges, excluding research orchestration and evaluation. Screenshots are original captures; coordinate figures are explanatory diagrams.
← All research notes