03/ 03Case study
Autonomous Repository Engineer
A verified tool-using coding agent: the planner only proposes actions, a deterministic state machine decides, and completion requires green test evidence - never model claims.
- Python
- pytest
- Git
Lab/See it run
ANALYZEThe runtime inventories the target repo - file tree, tests, entry points - fully inside the workspace jail. The planner sees only the summary.
01/Overview
What this is.
repo-engineer is a tool-using coding agent in pure-stdlib Python. Given a small repository and a task, it inspects the code, plans, edits through validated tools, runs tests, replans on failure, and only declares success when the verification suite is green. The core rule: the planner proposes, the runtime disposes.
The project treats agent reliability as an engineering problem rather than a prompting problem: malformed tool calls, arbitrary shell access and unverified success claims are made structurally impossible by the runtime, for scripted and LLM planners alike.
02/Problem
What motivated it.
Three failures dominate tool-using coding agents: malformed tool calls that crash the loop, arbitrary shell access that is unsafe by construction, and the model claiming a fix without proof.
The gap this project attacks: what does an agent runtime look like if none of those three are possible - where every call is validated, every mutation is jailed and allowlisted, and completion is a state reachable only through test evidence?
03/System
How it is built.
- state.py
- A nine-state machine (INIT, ANALYZE_REPO, PLAN, ACT, OBSERVE, REPLAN, VERIFY, COMPLETE, FAILED) with an explicit TRANSITIONS table - illegal transitions raise. Serializable AgentState plus dual budgets: max_steps 40, max_replans 4.
- tools.py
- Seven validated tools (list_tree, search, read_file, apply_edit, write_file, git_diff, run_tests) with ToolSpec schema validation, READ_ONLY/MUTATING/EXECUTION permission classes, AUTO/ASK/DENY policies, the workspace path jail, and the run_tests command allowlist.
- planner.py
- Three interchangeable planners: ScriptedPlanner (deterministic recipes for the reproducible benchmark), LLMPlanner (gemini-2.5-flash via an OpenAI-compatible client), RecordedPlanner (test double).
- agent.py
- The AgentRuntime loop: executes proposals, observes results, triggers verification and replanning, and exports a full JSON trace per run.
- benchmark/
- Harness plus four self-contained mini-repos, each run from a fresh copytree workspace with its own task.json and failing or missing tests.
04/Approach
The mechanisms that matter.
Planner proposes, runtime disposes
Planner output is only a tool-call proposal. The state machine alone transitions states, enforces budgets and decides verification - the planner is the only non-deterministic part, and the machine is the auditable safety boundary.
Schema-validated tools
ToolSpec.validate rejects malformed calls with structured errors that return as rejected Observations. Hallucinated calls become feedback in the agent's history, never crashes.
Workspace path jail
Every path argument is resolved and must land inside the workspace: ../, absolute paths and drive-letter escapes are rejected with PermissionError.
Fail-closed permissions
Tools are classified READ_ONLY, MUTATING or EXECUTION with AUTO/ASK/DENY policies. ASK without a callback denies - the headless default is safe.
Allowlisted execution
run_tests accepts only python -m pytest / unittest prefixes, shell=False, a 120 s timeout and capped output. rm -rf and curl-pipe-shell are test-proven rejections.
Unambiguous edits
apply_edit requires exactly one match (zero or multiple occurrences is an error) and write_file refuses overwrites, so every mutation path is explicit.
05/Evaluation
Setup and workloads.
- Four self-contained mini-repos, each with a task.json and a failing or missing suite: an off-by-one loop, a wrong tax constant with a pre-failing test, a slugify function with no coverage, and a rename across two files.
- Each run starts from a fresh copytree workspace; success is decided only by a VERIFY re-run of the full suite exiting 0. Budgets: 40 steps, 4 replans.
- Deterministic mode uses ScriptedPlanner recipes (byte-stable, no network, no keys); live mode swaps in gemini-2.5-flash at temperature 0 through the identical gates.
06/Results
Measured outcomes.
- 4/4
- SWE tasks, deterministic, zero failed tool calls
- 2/2
- live Gemini tasks (2 mid-run 429s absorbed)
- 0
- failed tool calls across every run
- 7
- validated tools behind schema + policy + jail
| Task | Change required | Steps | Replans | Failed calls |
|---|---|---|---|---|
| Off-by-one loop fix | one-line range fix | 5 | 0 | 0 |
| Repair failing test | wrong constant 0.20→0.25 | 5 | 1 | 0 |
| Add missing tests | write tests/test_slug.py | 5 | 0 | 0 |
| Constrained rename | rename across 2 files | 7 | 0 | 0 |
Deterministic results from results/benchmark_results.json (success_rate 1.0, 1 total replan). Live run from results/llm_benchmark_results.json: gemini-2.5-flash completed task_01 and task_03 with two HTTP 429 quota errors absorbed as recoverable planner errors; the committed per-task counters are cumulative snapshots (7 LLM calls true total), an accounting caveat the report itself documents.
07/Findings
What the results say.
- 01
The safety boundary is planner-agnostic: live LLM proposals flow through the identical validate, policy, jail and execute gates as scripted ones.
- 02
Provider failures were absorbed mid-run: two HTTP 429 quota errors surfaced as recoverable planner errors and the task still ended verified - resilience lives in the runtime, not the model.
- 03
Zero failed tool calls across all runs: rejections return as observations fed back into history, not exceptions.
- 04
Completion is unreachable by claim: only a VERIFY re-run exiting 0 opens the COMPLETE transition.
08/Decisions
Why it was built this way.
A state machine around the planner
The planner is the only non-deterministic component; wrapping it makes the safety boundary auditable and identical for scripted and LLM planners.
apply_edit, not diffs
Exact-once string matching rejects ambiguity mechanically (zero or multiple matches are errors) without a diff parser.
Scripted planner for the benchmark
Byte-stable recipes with no keys or network mean the benchmark measures the runtime, not model intelligence.
No framework
Zero third-party runtime dependencies: the validation, jail and policy layer is the actual engineering, and a framework would hide exactly those parts.
Completion needs test evidence
A planner claiming success changes nothing: VERIFY re-runs the suite and requires exit 0.
09/Validation
How the claims are checked.
- 14 test functions (17 collected with a four-task parametrization): illegal transitions raise; unknown tools and wrong-typed arguments are rejected; ../ and drive-letter jail escapes fail; apply_edit ambiguity errors; write_file refuses overwrites; EXECUTION under DENY is blocked; rm -rf and curl-pipe-shell are allowlist-rejected; ASK without a callback fails closed in both directions; LLMPlanner refuses without credentials; budget exhaustion ends in FAILED.
- Benchmark JSONs commit per-task steps, replans, wall time and failed-call counters; the live run records token totals and its 429 recovery events.
- The state machine itself is exhaustive: any transition outside the table raises, so the agent cannot reach COMPLETE except through VERIFY.
10/Limitations
Stated plainly.
- The scripted planner is a fixture: the benchmark measures the runtime, not model intelligence.
- The live LLM benchmark is n=2, one run each - a demonstration, not a leaderboard.
- The jail is per-path-argument, not an OS-level sandbox, and there is no transactional rollback for multi-file edits.
- The LLM planner's context includes file content, so prompt injection is mitigated by prompting only - a documented residual risk.
11/Interview TL;DR
- Built
- A pure-stdlib tool-using coding agent: a ten-state machine around pluggable planners, seven schema-validated tools inside a workspace jail, and completion decided only by green tests.
- Why
- Three failures dominate coding agents - malformed tool calls, arbitrary shell access, and 'the model says it's fixed' without proof. This runtime makes all three structurally impossible.
- How it works
- The planner only proposes tool calls; the state machine validates, jails, executes and observes; VERIFY re-runs the suite and COMPLETE requires exit 0.
- Key decision
- Wrapping the planner in a deterministic state machine: the planner is the only non-deterministic part, the machine is the auditable safety boundary.
- Strongest result
- 4/4 benchmark tasks with zero failed tool calls, plus a live Gemini run that absorbed two mid-run HTTP 429s and still finished verified.
- Main limitation
- The scripted planner is a fixture, the LLM run is n=2, and the jail is per-path rather than an OS sandbox.
- Next
- OS-level sandboxing and transactional rollback for multi-file edits.
Likely questions
Q01The model says it fixed the bug but tests fail - what happens?
Nothing: COMPLETE is only reachable from VERIFY, which re-runs the full suite and requires exit 0. A planner claiming success changes nothing.
Q02Why a state machine around the planner?
The planner is the only non-deterministic component. Wrapping it makes the safety boundary auditable: every call passes validate, policy, jail and execute identically for scripted and LLM planners.
Q03Why apply_edit with exact-once matching instead of diffs?
Ambiguity is rejected mechanically: zero or multiple matches are errors. That gives edits an unambiguous contract without a diff parser.
Q04What happens when a tool call is malformed?
Schema validation returns a structured rejection as an Observation, which enters the agent's history - the loop continues instead of crashing. Zero failed tool calls across all runs.
Q05Why not LangChain?
The validation, jail and policy layer is the actual engineering; a framework would hide exactly the parts that matter. The whole runtime is stdlib Python.