A better test for an agent
Making software experiments easier to trust.

A test line, a quarantine pen and a wall of nine tools.
An illustration built for this site in Blender. It explains the idea; it is not data from the project.
Before asking whether a coding agent improved, it helps to know exactly what it was tested on. This project builds the infrastructure behind that question for a repository-repair agent based on Gemma 4: task provenance, evaluation isolation, deterministic training schedules, and recoverable runs. Its working rule is that no claim should rest on an unverified step.
The job being tested
The agent is given a real open-source repository and a reported issue. It may use exactly nine tools, covering running commands, reading and editing files, checking status, navigating code structure and submitting a patch. Whether the patch is correct is decided by tests it never sees.
A sealed room and two kinds of test
Each task runs in a container with the network switched off. A valid task must fail before the fix and pass after it: the tests that were failing must now pass, and the tests that were passing must still pass. Parsers for five test frameworks read the per-case results and reject summaries whose totals contradict their own cases.
Knowing where each task came from
From 32,079 candidate rows, 4,818 were eligible and 2,966 were selected and frozen into training, development and confirmation splits, separated by repository and by owner so the same project cannot appear on both sides. Licence files were checked at each task’s pinned commit, which caught four repositories carrying copyleft text under a permissive label.
Four gates and two checkers
Training examples begin in quarantine. They are admitted only through four gates: rights and attribution, split isolation, runtime qualification, and task alignment. A separate audit module then recomputes every quantity without sharing code with the loader, so the two must agree independently. The first admitted set holds 51 tasks.
Separating infrastructure from performance
The model is a 4-bit Gemma 4 with a small LoRA adapter. So far it has taken one update on a tiny fixture; the first real training step ran out of memory, was diagnosed, and was fixed in code that has not yet been re-measured on a GPU. No sustained training has run, so the project does not claim that the agent has learned anything yet.
Make an improvement reproducible.
- 01
Track the task’s lineage
Repository tasks carry source provenance and evaluation contracts. Checks for overlap and quarantine keep questionable material out of the training path. This makes the boundary between development data and held-out evaluation something the tooling can inspect.
- 02
Read the tests, not just the headline
The test audit preserves individual case outcomes and rejects contradictory aggregate counts. Failures, errors, and skipped cases remain distinct. A successful process exit or an attractive summary count cannot substitute for the underlying test evidence.
- 03
Translate actions, replay results
Teacher trajectories from another agent framework are converted to the nine-tool format under a fixed set of rules. Tool results are deliberately not translated: they must come from replaying the action, because rewriting them would fabricate runtime output. One untranslatable call rejects the whole trajectory.
These are implemented evaluation and training-preparation mechanisms. The repository does not yet establish learned gains on unseen repository tasks, and this page makes no claim about a competition result.
Read the implementation readiness recordThe honest edges.
- No model has been trained at scale. There is no measured improvement to report.
- All 51 admitted tasks are development-exposed, so they cannot be used as a held-out test.
- Licence checks record what each repository says at a pinned commit. They are observations, not legal clearance.
Where this connects.
ZenithSync Assay PlannerShares its project name, ZenithSync, and its habits: freeze the protocol, keep the negative results.
Language in motionInstrument skepticism in both: a verifier is audited before it is believed.
CollegeAdvisorEach receipt states what it does not establish, the same idea as cite-or-abstain.Research tooling in development. The project does not yet establish learned performance gains on unseen repository tasks.