Project essay

Building with Symphony: from a task to a reviewable change

How I let coding agents do real engineering work at Snap eHealth without letting them exceed the authority a human gave them.

Thomas Howard · 7 min read

My role

Architecture & development at Snap eHealth

Focus

Codex · Symphony · Agent orchestration

Project stage

In use · Built our products under human supervision

A coding agent that can only follow instructions is not worth much. The value comes when it can investigate a failure, try a fix, run the tests, and try again. That same freedom is the risk. An agent with real autonomy can edit files it was never asked to touch, keep spending after a crash, or hand over a change that passed tests on a different revision than the one being merged.

In a healthcare software company, any of those is a serious problem. So the question I set out to answer was narrow: how much useful autonomy can an agent have while staying inside exactly what a human approved?

I architect, build, and operate the Symphony-based engineering infrastructure that answers it at Snap eHealth. It turns an authorized engineering task into a reviewable change. It built our products under human supervision, with a person approving every task and merging every change. Running it unattended is the next step.

OpenAI’s upstream Symphony specification supplies the foundation: issue intake, per-issue workspaces, sessions, retries, and reconciliation. My work is everything Snap needed on top of it: who may start work, what context the agent sees, where it may write, how execution survives restarts, and what evidence a reviewer gets.

No agent starts work without a human decision behind it

The easiest way to lose control of an agent is to let it decide what it is allowed to do. So every task starts as a contract. It names one repository and source revision, the allowed changes, acceptance criteria, test commands, and a reviewer.

Admission checks bind that contract to a recorded human decision. A ticket label cannot grant more authority. Neither can a plan the agent wrote for itself.

I built a persistent controller extension that records dependencies, permitted write paths, tests, and finite execution limits. If tests fail, the agent may repair its work, but only inside the original approval and the remaining budget. Changed scope, revoked authority, expired approval, or exhausted limits stop execution. They do not prompt a negotiation.

Split the work so no agent grades its own homework

An agent that writes code and then reviews it will tend to approve it. I separate implementation, testing, and acceptance so each role has a concrete input, a permitted action, and an output someone else can check. One task may pass through several roles as evidence builds.

Implementation

Receives the approved task, repository context, and write scope. Produces a candidate diff and developer checks within that scope.

Testing / QA

Receives acceptance criteria and the candidate. Adds or runs permitted tests and returns reproducible results and coverage gaps.

Review

Inspects the exact diff, requirements, and verification evidence. Returns findings and unresolved risks while preserving the reviewed candidate.

Debugging

Uses a failed check, bounded diagnostics, and retained state to reproduce the problem and identify a targeted correction.

Security

Examines permissions, credentials, data flows, and trust boundaries. Returns findings and required controls for the acceptance decision.

Remediation

Repairs an identified defect within permitted files and submits the changed candidate for fresh verification.

The controller runs dependent tasks in order and keeps one Codex session per task, so debugging and remediation can pick up where implementation left off. Independent review runs outside that session, with its own evidence and its own authority boundary. That separation is the point.

Enforce the limits in the runtime, not the prompt

Telling a model to stay in scope is guidance. Operating-system permissions and host checks are enforcement. Repository context shapes what the agent does; the runtime decides what it can do.

Let the agent edit, but run the tests in a locked room

The native Codex controller applies filesystem and network permissions, turns off external apps, plugins, and MCP servers, and checks the returned workspace and permission profile. Codex manages authentication. A second execution path, the broker, uses a container worker and a host-only model relay instead. Each path has explicit controls for how it runs.

Tests run somewhere else entirely: a pinned, nonroot container with no network, bounded resources, and a read-only copy of the source. That copy leaves out Git metadata, approval records, the journal, and host credentials. The test environment has no network, no host credentials, and no access to approval records or the journal.

Make old permission slips stop working

Approvals need to expire, and revoking one has to take effect even for work already running. The broker issues signed execution leases tied to a separately signed approval, the task, the worker, an expiry, and a stored generation number. Revocation bumps the generation. A signature that was valid a minute ago now fails the authority check, so stale work is fenced off at its next authority check.

The persistent controller does the same job its own way. It ties its journal to the approved plan, checks cancellation and expiry, and confirms the worker’s identity before resuming. Changes outside scope put the run in quarantine for a person to inspect. My control-plane essay goes deeper on the architecture behind these choices.

Never let a restart make uncertain spend look free

Time, turn, token, process, and memory limits each catch a different failure. On the broker path, requests reserve against configured cost bounds before they are forwarded. If the final usage never comes back, a durable hold stays in place until it is reconciled. A restart cannot quietly erase it.

The subscription-backed controller stops new work when observed usage crosses a threshold, though a request already in flight can still go over. Provider work can also outlast a local timeout. The journal keeps those uncertainties on the record. Cancelling a job is not proof it cost nothing.

Accept the exact change that was tested, nothing else

The quiet failure in automated delivery is evidence drift: tests passed on one revision, and a slightly different one gets merged. Here, any change to the artifact invalidates earlier passing evidence. The controller checks workspace scope before and after validation and reruns the combined suite before asking for delivery. A final failure sends the responsible task back to a bounded repair loop.

The hard cases have regression tests of their own: persisted cancellation, recovery of interrupted work, unknown usage, scope quarantine, stale test evidence, final-suite repair, and ambiguous publication outcomes. Controlled fixtures and local processes make those failures reproducible on demand.

Opening the pull request is its own risk. If the network drops mid-write, did the push happen? I built publication as a recoverable protocol. Record the intent first, then read back the remote identity, parent revision, and contents. An uncertain result triggers reconciliation, not a blind retry. The adapter targets an existing draft PR and leaves merge authority with the reviewer.

Snap Workflows adds a second, independent check in hosted CI: canonical CI, then architecture review. The reviewer gets the candidate and evidence through read-only mounts, has no host credentials, and calls models through a constrained inference proxy. Its verdict must match the candidate revision and the passing checks. A human still decides what merges and deploys.

Every failure tightened the contract instead of loosening it

The useful failures exposed gaps the happy path hid. Early on, a runtime check reported Codex initialized correctly, while command execution still lacked a sandbox capability it needed. I changed acceptance to an observable outcome: create the expected file, run the command under the intended restrictions, and keep the matching completion events.

A failed model attempt exposed another gap. Generic error messages and early cleanup were deleting the evidence I needed to debug. The fix kept bounded failure categories and container state until after diagnosis, while still excluding raw prompts and credentials.

Then native filesystem restrictions blocked the host’s Python framework. The tempting fix was to widen the agent’s access. Instead, I kept editing scoped and moved tests into the isolated validator. The failure made the line between implementation and verification sharper.

One record that says what a reviewer is accepting

I treat the task, its authority, the candidate change, and its verification as one operational record. That record answers three questions at any moment: may this work continue, why did it stop, and what exactly is the reviewer accepting? Human authorization stays part of the architecture through implementation, recovery, and release.

Separately, I’m building AES, a personal project that carries these ideas forward.

What I’d tell a founder

Put authority outside the agent. If a prompt, a label, or the agent’s own plan can expand what it is allowed to do, you do not have a boundary. You have a suggestion.

Tie every approval to an exact artifact. Passing tests mean nothing if they ran on a different revision than the one you merge. Make any change invalidate the old evidence.

Treat uncertainty as a state, not a success. Crashes, timeouts, and half-finished writes will happen. The system should record what it does not know and reconcile it, rather than assume the best case and retry.