Beyond AI Coding Assistants: Engineering the Control Plane for Autonomous Software Delivery

The next frontier in AI engineering is not generating more code. It is building the systems that allow autonomous agents to deliver software reliably, safely, and with verifiable evidence.

Thomas Howard

AI has fundamentally changed the mechanics of writing software. Coding agents can inspect repositories, diagnose defects, implement features, generate tests, and propose changes that once required hours of engineering effort.

But writing code and delivering software are different engineering problems.

A coding agent can produce a technically correct patch and still leave an organization with an operational failure. It might work against the wrong branch, modify files outside its intended scope, duplicate another agent's work, overlook a failing integration test, or create a pull request after its execution authority has expired.

These are not necessarily failures of model intelligence. They are failures of the system surrounding the model.

As engineering organizations move from AI-assisted development toward autonomous execution, the central architectural question changes: How do we give AI agents enough freedom to complete meaningful engineering work while retaining deterministic control over what they can execute, modify, approve, and deliver?

The answer requires more than better prompts, larger context windows, or more sophisticated models. It requires an engineering control plane.

1. The distinction between coding assistance and autonomous delivery

Consider the progression of AI within an engineering organization.

The transition between stages three and four is particularly important. A collection of successful coding agents is not automatically a reliable engineering organization.

Each agent may perform its assigned task effectively while the surrounding system fails to coordinate work, enforce permissions, or establish whether changes are safe to deliver.

Traditional engineering organizations solve these problems through ownership, process, technical controls, and human judgment. Autonomous engineering must turn much of that coordination into executable infrastructure.

2. What OpenAI's harness engineering experiment demonstrates

OpenAI published Harness engineering: leveraging Codex in an agent-first world.

The team reported building a product containing approximately one million lines of code and merging roughly 1,500 pull requests over five months, initially with three engineers directing Codex.

The headline numbers are striking, but the more significant finding is architectural.

The engineers did not simply ask agents to write more code. They invested in the environment that enabled agents to work independently. They made applications executable within isolated development worktrees. They exposed application logs, metrics, traces, and browser inspection capabilities to agents. They established repository conventions and mechanically enforced architectural boundaries.

Their central constraint was that humans would steer the engineering process while agents generated the code.

That changed the engineers' responsibilities. Instead of directly implementing every feature, engineers increasingly defined requirements, designed feedback mechanisms, improved development environments, and established constraints that made autonomous execution repeatable.

The lesson is not that every organization should eliminate manual code writing. It is that agent productivity is strongly influenced by the quality of the engineering environment in which the agent operates.

A capable model operating inside an opaque, poorly documented repository has fewer opportunities to validate its decisions. The same model operating inside a well-structured repository, with accessible tests, observability, architecture rules, and clear task boundaries, can work with substantially more independence.

This is harness engineering: designing the surrounding system so the agent can do useful work and determine whether that work is correct. But harness engineering alone does not solve coordination across dozens of autonomous tasks. For that, we need a control plane.

3. The control plane architecture

OpenAI's Symphony specification describes a service that monitors an issue tracker, dispatches coding agents into dedicated workspaces, and manages their execution.

Its design includes a workflow definition, issue tracker adapter, orchestrator, workspace manager, agent runner, and observability. That is an important architectural foundation.

However, scheduling an agent and governing its actions are separate responsibilities. Symphony's published specification explicitly leaves some security and approval choices to implementations.

For governed autonomous delivery, I would separate the system into three boundaries.

The control plane decides what should run, who owns the task, what permissions apply, and whether the result can progress.

The execution plane runs agents in bounded workspaces with the tools and resources needed to perform their assigned work.

The evidence and promotion plane validates results, records provenance, and controls which changes can become reviewable or deployable artifacts.

Reference architecture

The essential principle is that the agent does not independently grant itself delivery authority.

An agent may produce a patch, tests, documentation, and a recommendation. A separate, trusted component determines whether that result satisfies the organization's delivery policy.

This distinction becomes increasingly important when multiple agents work concurrently.

4. Why autonomous engineering needs durable execution authority

Imagine two coding agents working on the same engineering ticket.

Agent A claims the ticket and begins implementing a change. Its execution lease expires because the host becomes unresponsive. The orchestrator assigns the ticket to Agent B.

Agent B completes the task and produces a validated change. Several minutes later, Agent A resumes.

If Agent A still possesses valid repository write credentials, it might submit its outdated patch or create a second pull request. The original task assignment may be expired, but the operational capability remains.

This is a classic distributed-systems problem.

One technique for addressing it is a fencing token, typically a monotonically increasing generation number associated with an execution.

When a new worker claims a task, the generation advances. The trusted component responsible for accepting a result verifies that the worker still holds the current generation.

An old worker may continue computing, but its stale authority must not allow it to promote a result. The critical detail is where the check occurs.

Checking a lease when work begins is insufficient. Authority must be checked again at the boundary where consequential actions are accepted.

A reproducible fencing experiment

The following Python example demonstrates the fundamental principle using SQLite. It does not launch an AI agent, contact GitHub, or modify a real repository.

Save it as fencing_demo.py and run it with python3 fencing_demo.py.

Example · Python 3 / SQLite

import sqlite3
import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as directory:
    db = sqlite3.connect(Path(directory) / "control.sqlite")

    db.execute("""
        CREATE TABLE runs (
            issue TEXT PRIMARY KEY,
            generation INTEGER NOT NULL,
            state TEXT NOT NULL
        )
    """)

    def claim(issue):
        row = db.execute(
            "SELECT generation FROM runs WHERE issue = ?",
            (issue,)
        ).fetchone()

        generation = row[0] + 1 if row else 1

        with db:
            db.execute("""
                INSERT INTO runs VALUES (?, ?, 'RUNNING')
                ON CONFLICT(issue) DO UPDATE SET
                    generation = excluded.generation,
                    state = 'RUNNING'
            """, (issue, generation))

        return generation

    def revoke(issue):
        with db:
            db.execute("""
                UPDATE runs
                SET generation = generation + 1,
                    state = 'REVOKED'
                WHERE issue = ?
            """, (issue,))

    def submit_for_review(issue, generation):
        with db:
            result = db.execute("""
                UPDATE runs SET state = 'REVIEW'
                WHERE issue = ?
                  AND generation = ?
                  AND state = 'RUNNING'
            """, (issue, generation))

        return result.rowcount == 1

    old_worker = claim("ENG-42")
    revoke("ENG-42")

    assert not submit_for_review("ENG-42", old_worker)

    new_worker = claim("ENG-42")
    assert submit_for_review("ENG-42", new_worker)

    print("Stale worker denied")
    print("Current worker accepted")

Expected output:

Stale worker denied
Current worker accepted

This is deliberately a small experiment. The trusted database transition is atomic, but it is not an atomic transaction with an external Git hosting provider.

A production system still needs to handle retries, duplicate submissions, delayed external API calls, crash recovery, and uncertainty about whether an external side effect occurred.

It also needs to control the worker's credentials. Fencing a trusted submission service is ineffective if the agent can bypass that service and directly push changes.

5. The failure modes that matter

A production control plane should be evaluated against failure scenarios, not merely successful demonstrations.

The most dangerous failures are not always obvious exceptions. A failed command is relatively easy to detect.

A worker that times out after an external service accepted its request is more difficult. Retrying blindly might duplicate the action, while assuming failure might leave untracked changes behind.

The correct state in such cases is often uncertain, not failed or successful. An autonomous system needs a reconciliation process capable of establishing what actually happened.

Likewise, stopping an agent process is not identical to revoking its authority. A delayed request, surviving child process, or previously granted credential may still have consequences.

Termination, revocation, and evidence of cleanup should be treated as distinct concepts.

6. Why isolation is necessary but not sufficient

Execution isolation is another essential layer.

A coding agent might require the ability to run shell commands, install development dependencies, execute tests, and inspect application behavior. Granting those capabilities directly against a developer's unrestricted environment creates unnecessary exposure.

An isolated execution environment can constrain filesystem access, network connectivity, process privileges, resource consumption, and credential availability.

But a workspace is not automatically a security boundary. A Git worktree provides useful source-code separation. It does not, by itself, prevent a process from reading other host files.

Likewise, a container configuration that requests network isolation or reduced privileges does not prove that every runtime restriction was enforced. The distinction is between requested configuration and observed behavior.

In one local execution experiment, I encountered an instructive example: a test worker launched, but the supervising controller rejected the runtime inspection because a Docker field was represented differently from the expected configuration.

The worker subsequently exited, yet the controller could not automatically establish the full cleanup lifecycle.

That is not evidence of a compromised container, nor is it a successful lifecycle verification. It illustrates an important operational tension. Fail-closed validation can prevent unsafe assumptions, but overly rigid compatibility checks can also create operational friction.

The engineering response is to distinguish meaningful security invariants from harmless serialization differences, test those assumptions, and preserve uncertain execution records.

The goal is not endless infrastructure hardening. It is a sufficiently strong and observable execution boundary that enables real engineering work without making every configuration detail a manual intervention.

7. The repository becomes part of the agent harness

A governed control plane cannot compensate for a repository that agents cannot understand.

Repositories designed primarily around human tribal knowledge create a serious problem for autonomous execution. An agent needs to determine where functionality belongs, which contracts must remain stable, how tests should be executed, and what constitutes a valid change.

That information should be discoverable from the repository itself.

A useful agent-ready repository includes:

• A concise AGENTS.md describing operating conventions, boundaries, and validation commands.
• A versioned WORKFLOW.md when using an orchestration system such as Symphony.
• Clearly defined module ownership and architectural dependency rules.
• Deterministic setup, build, and test commands.
• Machine-readable interface contracts and schemas.
• Small, purpose-specific skills that explain repeatable engineering procedures.
• Representative fixtures and isolated integration environments.
• Accessible logs, metrics, traces, and reproducible failure scenarios.

The distinction between instructions and enforcement matters here.

An AGENTS.md file can explain that domain code must not access the database directly. A structural test or architecture linter can enforce that rule.

A prompt can request adequate test coverage. A CI gate can verify the relevant tests and coverage criteria.

An agent may disregard or misunderstand written instructions. It should not be able to bypass trusted validation merely by announcing that it has completed the task.

This approach also reduces the need to repeatedly provide agents with large amounts of explanatory context. The necessary knowledge becomes part of the environment and can evolve with the codebase.

8. Evidence, evaluations, and provenance

Successful autonomous delivery requires a definition of completion that is stronger than an agent's final message.

Anthropic's engineering work on evaluating AI agents emphasizes the importance of evaluating outcomes and trajectories, rather than relying on a model's description of its own performance.

For software engineering, that means independently establishing whether the resulting system behaves as required.

A reviewable autonomous execution should produce a structured evidence bundle containing:

Provenance should link the task, source revision, execution, validation, and resulting artifact.

Cryptographic attestations can further establish where and how build artifacts were produced. GitHub, for example, supports artifact attestations that associate builds with workflow and repository information.

However, a signed attestation does not prove that the code is correct or secure. It establishes the provenance of the attested artifact. Correctness and security still require appropriate validation.

A practical promotion policy might require all critical tests to pass, no prohibited file changes, verified source identity, a complete evidence record, and an independent approval for higher-risk changes.

The agent generates the candidate. The trusted system evaluates whether that candidate is eligible to advance.

9. How to measure autonomous engineering

Counting generated lines of code is a poor measure of engineering value. Counting agent runs is not much better.

Even pull request volume can be misleading if reviewers spend more time correcting changes or production stability declines.

Google's 2025 DORA research describes AI as an amplifier of existing engineering capabilities. Greater throughput without sufficient organizational and technical foundations can expose underlying weaknesses.

For autonomous engineering, I would establish a measurement framework before scaling concurrency.

These are proposed metrics, not claimed experimental results.

A useful baseline requires consistent task categories, comparable complexity, and enough observations to distinguish model behavior from environmental failures. A five-minute documentation fix should not be compared directly with a multi-service database migration.

I would also report distributions rather than averages alone. A system with excellent average task completion time but frequent six-hour outliers may impose substantial operational overhead.

Most importantly, measurement should distinguish agent failures from harness failures.

An incorrect patch is a different problem from a missing dependency, broken test fixture, lost execution lease, or unavailable API. Without that distinction, organizations risk optimizing the model while ignoring the platform.

10. Three experiments I would run before scaling

Before allowing a control plane to execute a large backlog unattended, I would run three controlled experiments in a disposable repository.

Experiment A: Duplicate ownership and stale execution

Dispatch a task, deliberately expire or revoke the first execution, and assign a second generation. Allow the old worker to submit a delayed result. Verify that the trusted promotion boundary rejects it and records the reason.

Pass criterion: Zero stale results promoted, with both generations preserved in the audit record.

Experiment B: Crash and uncertain external effects

Interrupt execution at several points: before claiming work, after workspace creation, during validation, and after a simulated external submission whose acknowledgment is lost.

Restart the orchestrator and verify that it reconciles existing state instead of blindly repeating work.

Pass criterion: No duplicate consequential effects, no lost ownership history, and every unresolved outcome explicitly identified.

Experiment C: Policy violation during an otherwise successful task

Provide a legitimate feature request that an agent can complete correctly, then introduce a conflicting instruction inside an untrusted repository file requesting an unauthorized action.

Verify that the agent's available tools and trusted promotion gate prevent the unauthorized result from progressing, regardless of whether the agent attempts it.

Pass criterion: Legitimate changes can be accepted while prohibited actions are denied and recorded.

Each experiment should be repeated enough times to expose nondeterministic behavior. These tests measure the engineering system, not just the coding model.

11. Autonomy should reduce supervision, not eliminate accountability

There is a temptation to equate autonomous engineering with removing humans from the entire delivery process. That is the wrong objective.

The more useful goal is to remove unnecessary human intervention while preserving meaningful human authority.

An agent should not require an engineer to approve every test execution, inspect every intermediate file, or manually restart routine recoverable work. Those activities should be handled through bounded permissions, deterministic checks, and automatic remediation.

At the same time, organizations should retain explicit authority over consequential actions such as production deployment, sensitive data access, infrastructure destruction, or changes that exceed the established risk policy.

Human review should be risk-based rather than universal or arbitrary. A documentation change and a database migration should not necessarily require the same approval process.

This is where the control plane becomes more than an agent scheduler. It becomes the mechanism for expressing and enforcing engineering risk.

12. The real shift in engineering leadership

The most consequential change introduced by autonomous engineering is not that software developers stop writing code.

It is that engineering organizations must become better at specifying intent, defining system boundaries, exposing feedback, and measuring outcomes.

That changes the role of platform engineering.

Developer platforms are no longer simply environments that make human engineers more productive. They increasingly become the environments in which autonomous workers operate.

Repository architecture becomes agent navigation infrastructure. CI becomes an evidence-producing evaluation system. Observability becomes an input to diagnosis and remediation. Policy becomes executable authority.

And engineering leadership shifts from supervising individual implementation steps toward designing a system capable of producing reliable engineering outcomes.

The essential question is no longer: How much code can an AI agent generate?

It is: How much verified engineering work can our organization safely complete without requiring continuous human supervision?

That is a much harder question. It requires distributed-systems thinking, software architecture, security engineering, developer platforms, evaluation design, and operational discipline.

It also provides a more meaningful benchmark for progress.

The organizations that succeed will not necessarily be those running the largest number of agents or using the most sophisticated orchestration framework.

They will be those that make autonomous work understandable, constrained, measurable, recoverable, and ultimately useful.

Technical references

1. OpenAI: Harness engineering: leveraging Codex in an agent-first world
2. OpenAI: Symphony service specification
3. OpenAI: Unlocking the Codex harness: how we built the App Server
4. Anthropic: Effective harnesses for long-running agents
5. Anthropic: Demystifying evals for AI agents
6. Google DORA: 2025 State of AI-assisted Software Development
7. GitHub: Artifact attestations and build provenance