← All work

A better test for an agent

Making software experiments easier to trust.

ROLE

Project lead & researcher

CONTEXT

ZenithSync Developer Agent

TOOLS & METHODS

Python · Gemma 4 · LoRA fine-tuning · Docker sandboxes · Evaluation tooling

A hand-built island model for A better test for an agent
THE ISLAND, EXPLAINED

A test line, a quarantine pen and a wall of nine tools.

An illustration built for this site in Blender. It explains the idea; it is not data from the project.

THE QUESTION

Before asking whether a coding agent improved, it helps to know exactly what it was tested on. This project builds the infrastructure behind that question for a repository-repair agent based on Gemma 4: task provenance, evaluation isolation, deterministic training schedules, and recoverable runs. Its working rule is that no claim should rest on an unverified step.

9tools the agent is allowed to use
4gates before a training example is admitted
2,966public tasks screened and frozen into splits
51tasks in the first admitted training set
01

The job being tested

The agent is given a real open-source repository and a reported issue. It may use exactly nine tools, covering running commands, reading and editing files, checking status, navigating code structure and submitting a patch. Whether the patch is correct is decided by tests it never sees.

02

A sealed room and two kinds of test

Each task runs in a container with the network switched off. A valid task must fail before the fix and pass after it: the tests that were failing must now pass, and the tests that were passing must still pass. Parsers for five test frameworks read the per-case results and reject summaries whose totals contradict their own cases.

03

Knowing where each task came from

From 32,079 candidate rows, 4,818 were eligible and 2,966 were selected and frozen into training, development and confirmation splits, separated by repository and by owner so the same project cannot appear on both sides. Licence files were checked at each task’s pinned commit, which caught four repositories carrying copyleft text under a permissive label.

04

Four gates and two checkers

Training examples begin in quarantine. They are admitted only through four gates: rights and attribution, split isolation, runtime qualification, and task alignment. A separate audit module then recomputes every quantity without sharing code with the loader, so the two must agree independently. The first admitted set holds 51 tasks.

05

Separating infrastructure from performance

The model is a 4-bit Gemma 4 with a small LoRA adapter. So far it has taken one update on a tiny fixture; the first real training step ran out of memory, was diagnosed, and was fixed in code that has not yet been re-measured on a GPU. No sustained training has run, so the project does not claim that the agent has learned anything yet.

INSIDE THE ENGINEERING

Make an improvement reproducible.

  1. 01

    Track the task’s lineage

    Repository tasks carry source provenance and evaluation contracts. Checks for overlap and quarantine keep questionable material out of the training path. This makes the boundary between development data and held-out evaluation something the tooling can inspect.

  2. 02

    Read the tests, not just the headline

    The test audit preserves individual case outcomes and rejects contradictory aggregate counts. Failures, errors, and skipped cases remain distinct. A successful process exit or an attractive summary count cannot substitute for the underlying test evidence.

  3. 03

    Translate actions, replay results

    Teacher trajectories from another agent framework are converted to the nine-tool format under a fixed set of rules. Tool results are deliberately not translated: they must come from replaying the action, because rewriting them would fabricate runtime output. One untranslatable call rejects the whole trajectory.

These are implemented evaluation and training-preparation mechanisms. The repository does not yet establish learned gains on unseen repository tasks, and this page makes no claim about a competition result.

Read the implementation readiness record
WHAT THIS DOES NOT SHOW

The honest edges.

  • No model has been trained at scale. There is no measured improvement to report.
  • All 51 admitted tasks are development-exposed, so they cannot be used as a held-out test.
  • Licence checks record what each repository says at a pinned commit. They are observations, not legal clearance.
NEARBY ISLANDS

Where this connects.

CONTEXT & FURTHER READING

Research tooling in development. The project does not yet establish learned performance gains on unseen repository tasks.