Research8 min read

The 400KB wall: when planning overhead starves the code.

Our structured workflow required planning and approval before implementation. Five of six structured runs in the nine-cell v4 matrix hit the 400KB request ceiling, and across two Fastify 5.6.2 pilots none of the eight structured runs reached final handoff.

DeepSeek V4.1 Flash & GLM 5.3 Flash · Voxel, Shop, Ledger & Fastify 5.6.2 (384 files, 72k LOC)

Plain Pi completed more handoffs in the v4 matrix and in each Fastify pilot.

In the frozen v4 matrix (with compaction disabled), Plain Pi and Carthagent+Jev both passed 25/25 external functional categories, but five of six Carthagent runs hit the 400KB request-body ceiling before final handoff. On Fastify, discovery, schema retries, provider failures, and candidate-test problems prevented structured handoffs. Plain Pi completed both tickets in Pilot 1 and vary in Pilot 2; its Pilot 2 appendHeader test hung at the deadline.

Three tasks, nine runs, and a 400KB ceiling.

The v4 matrix contained nine runs using DeepSeek V4.1 Flash with high thinking and compaction disabled. Each run had a 900-second wall limit, a maximum of 16,384 output tokens per response, and a 400KB incoming request-body limit. We tested Plain Pi, Carthagent, and Carthagent with an experimental advisory critic called Jev. Each ran three tasks: a Rust voxel engine, a full-stack web shop, and an exact-money financial ledger.

Frozen v4 nine-cell matrix (DeepSeek V4.1 Flash, compaction disabled)
TaskArmExternal checksTerminal outcomeTime / toolsKnown costCandidate’s own test suite
Voxel (Rust)Plain Pi8/8Completed handoff284.3s / 28$0.0422Pass: 15 unit + 8 HTTP
Voxel (Rust)Carthagent0/8Output-token limit (16k)276.7s / 22$0.0383No Cargo.toml written
Voxel (Rust)Carthagent + Jev8/8400KB input-byte limit332.5s / 36$0.0575Fail: src/http.rs:214 borrow error in test
Shop (Web)Plain Pi8/8900s wall-time limit900.3s / 41$0.0512*Pass: 14 tests
Shop (Web)Carthagent7/8400KB input-byte limit264.5s / 31$0.1145Fail: referenced test files missing
Shop (Web)Carthagent + Jev8/8400KB input-byte limit255.9s / 53$0.1267Pass: 13 tests
Ledger (BigInt)Plain Pi9/9Completed handoff129.1s / 15$0.0153Pass: 22 tests
Ledger (BigInt)Carthagent9/9400KB input-byte limit317.6s / 39$0.052929/30 pass (contradictory test assertion)
Ledger (BigInt)Carthagent + Jev9/9400KB input-byte limit321.0s / 32$0.051536/38 pass (wrong math in test assertion)

*Shop / Plain Pi had an additional $0.15 unresolved hold from an in-flight request canceled at the 900s deadline. Shorter times on the Carthagent rows reflect hitting the 400KB request ceiling early, not faster completion.

Every Carthagent run validated a design, captured an inspection, and completed approval at least once. Schema-formatting repairs often required 2 to 5 plan revisions. Because every turn re-sent the tool schemas, full plan captures, review transcripts, and reasoning metadata, five of the six structured runs crossed 400KB of request payload after writing the product code but before they could run final verification and handoff.

What the external evaluators caught in the artifacts.

Even when a run hit the 400KB ceiling, we scored the code it left on disk against our hidden external evaluators (including a BigInt financial oracle and real headless Firefox via geckodriver). The external checks found a mobile checkout obstruction, while the candidates’ own suites included incorrect ledger assertions:

The CSS banner that blocked mobile checkout.

In the Shop task, the Carthagent-only build passed 7 of 8 external categories: API pricing, filtering, idempotency across restarts, concurrent inventory, and desktop cart persistence all worked. It failed only mobile checkout in headless Firefox because its #status notification banner overlaid the checkout button, triggering element click intercepted on tap.

When the implementation is right and the agent’s test is wrong.

In the Exact-Money Ledger task, all three builds passed 9/9 external BigInt oracle checks (handling values above IEEE-754 float precision, malformed decimals, idempotency conflicts, and corrupted journals). Yet two candidates failed their own test suites: one wrote a test asserting that 10.00 + 0.50 == 1025 cents (instead of 1050) and tested a Unicode file path by passing compte_é to an ASCII-only account-name validator.

Scaling up to a 384-file repository exposed discovery bloat.

We next tested discovery and planning on an existing repository rather than a greenfield task: adding reply.appendHeader and reply.vary on a pinned checkout of Fastify v5.6.2 (384 tracked files, 72,231 lines of JS/TS, 94 base regression tests), first with DeepSeek V4.1 Flash and then with GLM 5.3 Flash.

Fastify v5.6.2 repository pilots (reply.appendHeader and reply.vary)
Pilot & modelPlain Pi outcomeCarthagent outcomePrimary bottleneck observed
Pilot 1: DeepSeek V4.1 Flash (6 cells)Completed both tickets (162.7s and 204.8s; 7/7 behavior, tsd types, 100 & 109 tests pass)0/4 completed (3 hit 180s provider compaction deadlines; 1 hit CURL_56 receive error)Agents spent 19 to 54 tool calls reading files and docs before creating an initial plan.
Pilot 2: GLM 5.3 Flash (6 cells, with startup guide)Completed vary end-to-end (289.5s; 7/7 behavior, 2,152/2,157 full upstream tests pass); appendHeader code passed 7/7 but candidate test hung at 900sReached implementation in 2 cells (both passed 7/7 external behavior & 94 base regressions), but 0/4 reached final handoff before 900sSchema repair retries and candidate test mistakes (using node:assert inside t.plan suites).

The pilots exposed two recurring delays. First, in Pilot 1, the structured runs spent 19 to 54 tool turns reading files and the delivery_plan schema before writing a plan, then timed out when summarizing that discovery history. Second, adding a focused pre-planning startup guide in Pilot 2 got Carthagent into implementation by tool 5 to 10 (producing Fastify runtime code that passed all 7 external behavior checks and 94 regressions), but schema repair loops and slow candidate test runs still prevented a clean handoff within 15 minutes.

Across both Fastify pilots, neither Jev-enabled run reached plan inspection, resulting in zero Jev calls. We do not claim any validated benefit for Jev from these trials.

Cutting delivery context by 26% to 51%.

The saved v4 histories contained repeated design, capture, and revision data in delivery_status responses. Superseded planning exchanges also remained in subsequent requests.

Context projection before and after approved-plan archivalBefore compaction, every turn re-sent full historical plan revisions, captures, and walkthroughs until hitting 400KB. After compaction, superseded delivery-only exchanges become compact archive pointers while code edits, user messages, and open blockers stay verbatim.Uncompacted v4 history (hits 400KB cap)• Full plan v1, v2, v3, v4 JSON echoed every turn• Full inspection captures + walkthroughs inline5 of 6 runs stopped before final handoffCompact projection + archive pointers• Active contract, checks & open blockers only• Archive IDs replace older approved delivery turns26.4% to 50.9% smaller message history
The archive projection described here retains code turns, failed tool outputs, user messages, and open findings verbatim.

In those historical delivery modules (lib/delivery-context.mjs and lib/context-archive.mjs), we implemented three changes:

  1. Compact status projections. delivery_status and plan-review responses returned only active acceptance criteria, check definitions, current open blockers, and compact evidence counts instead of repeating full historical walkthroughs.
  2. Paged capture archives. Large inspection payloads were written to content-addressed archive files rather than placed inline in the prompt history.
  3. Archiving superseded delivery turns. Once a plan was approved, older successful delivery-only exchanges (beyond the two most recent) were replaced with compact archive pointers, while user messages, code edits, failed tool outputs, and unresolved findings stayed verbatim.

Voxel history saved

Carthagent−26.4%
Carthagent + Jev−27.6%

26.4% – 27.6%

Offline byte reduction on saved v4 Voxel histories.

Shop history saved

Carthagent−29.6%
Carthagent + Jev−38.7%

29.6% – 38.7%

Offline byte reduction on saved v4 Shop histories.

Ledger history saved

Carthagent−48.4%
Carthagent + Jev−50.9%

48.4% – 50.9%

Offline byte reduction on saved v4 Ledger histories.

These percentages measure offline byte reductions when applying archive projection to the saved v4 message histories (assuming approved state). They are not replayed paid runs and do not prove that the stopped v4 runs would have completed.

Archive superseded planning history without removing acceptance checks.

In these runs, request limits and timeouts prevented final verification even when the remaining code passed external checks. After these trials, we changed context projection to archive superseded planning exchanges while retaining the acceptance checks. The offline projections reduced saved history bytes; we did not rerun the stopped trials to measure completion.

Research record: nine-cell v4 benchmark (commit e157563), two six-cell Fastify v5.6.2 pilots (f15d4eaba03d86786bec2de7aed1236718c813f9), and offline context-archive projections. Documented in B-icy/Carthagent (docs/deepseek-jev-benchmark-v4-results.md, docs/jev-repository-pilot-results.md, docs/glm-pilot-results.md, and docs/context-efficiency.md).

All research notes