The Harness

Mediation, Not Intermediation

Theodoros Galanos — Wed, 08 Jul 2026 00:00:00 GMT

import Figure from '../../components/Figure.astro'; import Pullquote from '../../components/Pullquote.astro'; import taskWorldKnot from '../../assets/blog/mediation-not-intermediation/task-world-knot-tension-loom-labelled.png';

Estimated reading time: 3 minutes

I have seen many posts the last few weeks on the "work on the foundations of [aspect] first before AI" message. I thought I'd write a few words on why I think this is wrong.

First, let's discuss these aspects. The most popular, by far, are data and governance. The idea is simple: first solve the issues in these, to prepare the ground for efficient and responsible AI deployment. Let's go through them one by one since the reasons for each are different.

Data: The Sisyphean Foundation

Anyone who has worked with AI knows that data are very important. As we moved to agentic orchestration and harness engineering importance shifted to context, which is (ideally) structured representations of fit-for-task data. The call for action, or pause, is the following: without the right data foundations, AI deployment will suffer.

Why is this wrong? Not because data are not foundational to AI. It is. But because that imaginary foundation of pristine data is unattainable. Like a modern Sisyphus, it is an uphill battle that never ends.

AI allows us to build extremely valuable workflows and agents on top of partially structured or unstructured data, with great return on investment. In fact, agentic workflows are the way out of legacy data in both directions: the best tool for structuring the messy data we already have, and the way to prevent future legacy data, by creating data at the source. This is the conceptual inversion that needs to happen:

<Pullquote> Harness engineering means developing agentic workflows that create high quality, structured, annotated, task specific data, at the source. </Pullquote>

I wrote about this previously: harness, benchmark, models, data, they are all parts of the same knot. Doing one, means doing all others. Developing agentic AI therefore is the best proactive approach to data engineering, that sets a solid foundation for what is coming.

Governance: Intermediation by Committee

I have, for some time now, said that one of the biggest benefits of AI is its ability to replace and remove unnecessary intermediation. All that extra layer of friction that adds hard to measure cost in almost everything we do today. I think, a lot of the work in the governance space today falls in this intermediation layer. This is obvious in every organisation that, even before they've designed a single agent, experimented with a single harness, evaluated a single task, deployed a single product, they immerse themselves in frameworks, policies, and committees. Intermediation, aimed towards inaction.

Now, this doesn't mean that governance is not important. In fact, it is not just a core layer of AI deployment but also a core design principle for agentic systems themselves. I'm not referring to governance embedded in risk practice, production controls, evaluations, and accountable ownership. That is infrastructure. I'm referring to governance as a separate approval theatre that delays contact with real systems. Nor does it start from a blank slate: industry domains already carry a long history of policies, governance frameworks, and risk frameworks, embedded in everything they do. These are assets, ready to be drawn on and extended for AI deployment from day one, not replaced by a parallel apparatus. Governance should be part of how AI systems are designed, how experimentation happens, how harnesses are built and agents are deployed. And that means, that it should be informed by practice and practitioners, benchmarks and evaluations, and not by committees and frameworks. Once again, the same pattern emerges: proactive not reactive. Build the systems, learn from them, and then (co-)design new governance around that experience.

High-Level Principles, Low-Level Infrastructure

High level principles and low level infrastructure. That is all you really need, not just to get started but build successful, scalable, and responsible AI systems. Both should be grounded in practice and domain expertise, in access to real problems, and in the ability to iterate and learn from real-world deployments. That has been my job for as long as I can remember in this space, to bring those two together and clear the noise in between them. Mediation, not intermediation.

Task Worlds and Meta-Harnesses

Theodoros Galanos — Wed, 24 Jun 2026 00:00:00 GMT

import CertificateStackDiagram from '../../components/charts/CertificateStackDiagram.astro'; import BearingCapacityConventionFigure from '../../components/charts/BearingCapacityConventionFigure.astro'; import MetaHarnessFlowDiagram from '../../components/charts/MetaHarnessFlowDiagram.astro'; import MissingWalkingSpeedFigure from '../../components/charts/MissingWalkingSpeedFigure.astro'; import { metaHarnessCertificateStack, metaHarnessFlowSteps, } from '../../data/meta-harness-diagrams'; import { bearingCapacityFigure, missingWalkingSpeedFigure, } from '../../data/task-world-figures';

Estimated reading time: 15 minutes

TL;DR

This essay argues that agent tasks in domains like engineering are not just prompts or dataset rows. They are worlds: they decide what the agent can see, which rules have authority, what counts as evidence, which failures can be named, and what can be repaired.

In order to generalise across complex engineering tasks, we need a meta-harness: a harness that knows how to construct other harness runs. Given a new, messy task, it can produce a problem brief and task-world profile, plan operations over that world, run reviewer and governance layers, materialise a recipe, bind execution evidence back through sandbox execution, and use failures to improve future world designs.

This gives us operational plasticity. Instead of hand-building a new harness for every domain, or letting an agent improvise invisibly, the system can adapt while keeping its decisions reviewable.

In aec-bench terms, this points toward a contract-driven pipeline:

task prose -> problem brief -> task-world profile -> operation/reviewer plan -> governed run evidence -> comparison ledger -> repair loop

The philosophical frame of this work is based on Alain Badiou's work and philosophical framework, but the engineering focus is concrete: make task worlds legible enough that agents can run, fail, be audited, and reshape the harness itself.

Tasks Are Worlds, Not Prompts

In domains where work is tied to standards, units, evidence, tools, conventions, approvals, and liability, an agent task is more than text sent to a model. It is a world.

A task world decides what appears to the agent, which actions exist, what counts as evidence, what counts as success, which contradictions are tolerated, which artifacts survive review, and which failures are repairable. The score is an output of that world. It is not the world itself.

This matters most in non-IID domains. Engineering tasks, legal workflows, clinical protocols, enterprise operations, security work, and scientific experiments do not arrive as neat draws from one stable distribution. They come with local conventions, partial information, conflicting sources, hidden authority chains, and different meanings of "done."

Two small engineering examples make the point.

In one shallow-foundation task, every numeric input was provided. A strong model still failed because it reached for a familiar textbook formula instead of the project-specific table the task required. The arithmetic was not the failure. The model chose the wrong engineering convention.

In a pedestrian-clearance task, the distance was given but walking speed was not. The model supplied its own walking speed, calculated a precise answer, and never treated the missing scalar as missing. Across repeated runs it invented different speeds. Again, the issue was not arithmetic. The issue was the relationship between the model and the world's authority structure.

Those failures are small, but they expose the larger issue. Put plainly: the model needed to know which rulebook to trust, whether it was allowed to fill in missing facts, what answer shape mattered, and what evidence a reviewer would need later.

That is why benchmark design cannot stop at task text and expected answer.

The practical thesis is:

Agents operate inside worlds. If we want them to work across new problem spaces, we need a way to describe, compose, validate, and repair those worlds.

Two ideas follow.

Badiou's philosophical vocabulary gives us a sharper way to think about task worlds: a world is not a container, but a regime of counting and appearing. The main references here are Alain Badiou's Being and Event, translated by Oliver Feltham, and Logics of Worlds, translated by Alberto Toscano.

Eighteen Years Later

Around eighteen years ago, when I was a much younger researcher at TU Delft, I found philosophy. I do not remember the exact path in. I only remember what happened after: for a while, it became almost everything I did.

I read Kierkegaard, Freud, Lacan, Zizek, and eventually Badiou. Zizek likes to call Badiou "the Master," usually after assigning himself the part of the jester. The joke stuck with me, but so did the philosophy.

Badiou gave me a center. His work offered a way to think about what a world is, what it means for something to exist in that world, how change happens, how an event breaks the old language, and how fidelity to that event can make a new truth possible.

That machinery changed how I think. It taught me to work with contradiction rather than rush to dissolve it. It shaped the way I read systems, institutions, technical claims, and failures. Of all the side trips of knowledge I have taken, philosophy is probably the one that did the most damage, in the best possible sense.

So it is a strange pleasure to come back to it here, eighteen years later, for a practical problem in agent infrastructure.

The idea is simple enough to say plainly:

Harness design for novel tasks is a way to structure and reconfigure their worlds.

That is the personal reason Badiou is here. The technical reason is that his language helps name what harnesses already do: they structure worlds.

A meta-harness makes that theory operational. It composes harness layers on demand so agents can act in new worlds without pretending every task belongs to the same distribution.

Making Worlds Strange Again

That theory needs a careful hand. Badiou does not give us a ready-made formal theory of agent benchmarks. Instead, he makes the word "world" strange again.

Badiou's formalisms do not produce agent benchmarks directly. I am using his vocabulary as a way to think about meta-harness design.

Specifically, I will be leaning on Being and Event for situation[^badiou-situation], multiplicity[^badiou-multiplicity], count-as-one[^badiou-count-as-one], event[^badiou-event], and fidelity[^badiou-fidelity]; and on Logics of Worlds for the shift toward appearance[^badiou-appearance], object[^badiou-object], relation[^badiou-relation], world logic[^badiou-world-logic], and forms of change[^badiou-forms-of-change].

In a typical benchmark vocabulary, a world is a container. Put an agent in a repo, a browser, a terminal, a spreadsheet, a simulated API, or a workplace sandbox, and call that the world. Badiou's theory makes the question more demanding:

What operation counted this multiplicity as one world?

For agent tasks, that means asking what has been admitted into the world and under what authority:

which files, messages, database rows, standards, policies, logs, tools, and traces belong;
which affordances let the agent perceive or alter them;
which side effects count as real;
which artifacts count as proof;
which failures are nameable by the current taxonomy;
which parts of the work exist but do not appear to the agent, verifier, reviewer, or generator.

A task world is more than state. It is appearance.

The agent-visible world is what the model can perceive: prompt context, files, tool outputs, shell results, browser pixels, DOM state, connector responses, memory, logs, and schemas.

The verifier-visible world is what tests, state diffs, validators, artifact checks, policy checks, and scoring code can inspect.

The reviewer-visible world is what a person can reconstruct from the retained evidence: traces, screenshots, diffs, transcripts, output files, and explanations.

The generator-visible world is what the meta-harness can use later to mutate, combine, restrict, repair, or retire task worlds.

Badly designed worlds collapse these surfaces. The agent sees one thing, the verifier checks another, the reviewer receives a thin final answer, and the generator learns almost nothing from the failed run.

Well designed worlds keep the visibility surfaces distinct enough that mismatches become objects. If the agent could not see the clause, say that. If the verifier could not inspect the artifact, say that. If the reviewer cannot reconstruct why the field failed, say that. If the generator cannot tell which part of the world caused the failure, the world is not yet generative.

The better question for benchmark design is therefore not:

What task should we ask?

It is:

What must appear, to which authority, for this world to make a trustworthy claim?

That sounds philosophical. It is also a harness requirement.

The Logic Of A Task World

Once a task world is treated as a regime of appearance, the next question is how statements inside that world become accepted. The agent says an artifact was created. The verifier says a field passed. A reviewer says the answer follows the right method. The run certificate says a result can be trusted. A task world needs rules for how those statements become valid enough to act on.

Appearance alone is not enough. The world must know when a statement is settled, when it needs a witness, when it is only partially supported, and when contradictory evidence can be held without corrupting the run.

The harness name for that translation is a logic profile.

A realistic task world needs three kinds of logical gates.

The first is closure. Closure borrows from classical logic: the world asks for a clean decision about a proposition. Some propositions should be decidable inside the world:

the file exists or it does not;
the schema validates or it does not;
the test passes or it fails;
the required approval exists or it does not;
the forbidden side effect occurred or it did not.

Closure gates end ambiguity. They are right for machine-verifiable invariants, exact artifacts, permission boundaries, and state assertions. This is where the argument from Executable Standards enters the world layer: obligations become more useful to agents when they stop living only as prose and become predicates a verifier can run.

The second is construction. Construction borrows from intuitionistic logic in the practical sense that a statement counts only when the run can produce a witness for it. Before the witness exists, the useful state may be not-yet-proven rather than true or false.

"No failing test was found" is not the same as "the required artifact was produced." "The model described a JSON object" is not the same as "the JSON file exists where the verifier can inspect it." "No reviewer objected" is not the same as "review evidence exists."

Badiou's intensities of appearance matter here. Some things in a task world do not simply exist or not exist. They appear weakly, partially, or fully. A rubric item can be absent, weakly evidenced, partly satisfied, or satisfied. An LLM judge is a practical version of this graded appearance if its score, rubric, model, cited evidence, and rationale are preserved as witness material. It is not an oracle. It is an evidence-producing component inside the world.

Construction gates are right for reports, patches, manifests, validator output, durable run records, review approvals, generated datasets, and any claim that should not pass by absence of disproof.

The third is containment. Containment borrows from paraconsistent logic in the practical sense that contradiction is treated as local evidence to be scoped, not a reason for the whole run to collapse. Real task worlds contain contradictions. Source documents conflict. Runtime behavior disagrees with docs. Two agents diagnose different harness layers. A score says failure because the verifier did not find report.json, while the filesystem trace shows the agent wrote the required report to a different output path. A policy blocks mutation through one channel while another channel permits an equivalent state change.

Containment gates localize contradiction. The world records it with scope, source, affected claims, and allowed next actions.

This is the compact logic profile:

closure: what can be decided cleanly?
construction: what must be witnessed, and with what strength, before it counts?
containment: what contradictions can be held without corrupting the run?

That profile is not decorative metadata. It changes the run.

If a task world has no closure gates, it is too vague to score. If it has no construction gates, it rewards unsupported claims. If it has no containment gates, every contradiction gets flattened into pass, fail, or confusion.

A world card should therefore declare its logic profile:

logic_profile:
  closure_gates:
    - proposition: "required artifact exists"
      authority: "filesystem"
      failure_effect: "run_invalid"
    - proposition: "schema validates"
      authority: "schema_validator"
      failure_effect: "run_invalid"
  construction_gates:
    - proposition: "answer is supported by preserved evidence"
      required_witnesses: ["artifact", "trace_pointer", "source_span"]
      acceptance: "all required witnesses present"
      failure_effect: "claim_unproven"
    - proposition: "rubric item is satisfied"
      required_witnesses: ["rubric_score", "judge_rationale", "cited_evidence"]
      acceptance: "score >= threshold with preserved rationale"
      failure_effect: "review_required"
  containment_gates:
    - contradiction: "source and verifier disagree"
      required_record: ["sources", "affected_claims", "allowed_next_actions"]
      failure_effect: "world_needs_repair_if_unscoped"
    - contradiction: "trace and score disagree"
      required_record: ["trace_pointer", "score_field", "diagnosis"]
      failure_effect: "world_needs_repair_if_unscoped"
  event_triggers:
    - "unclassifiable failure mode"
    - "score artifact contradiction"
    - "side effect through unmodeled channel"

Those entries are not merely prompts to an agent. They are declarations that split responsibility. In aec-bench terms, a richer world description would carry this profile beside the task-world profile: closure gates become verifier or artifact checks, construction gates become required evidence and review packets, containment gates become contradiction-ledger records, and event triggers become candidates for schema, verifier, evidence, governance, or generator repair.

In code, this deliberately uses plain field names rather than philosophical labels. The contract is a TaskWorldProfile with logic_profile fields such as closure_gates, construction_gates, containment_gates, event_triggers, and agentic_review. The same profile also carries an operation_profile for subset, difference, projection, and product handles. The conceptual terms stay in the essay. The execution contract stays reviewable.

Deterministic code should own the parts that can be made mechanical: contract validation, hard artifact checks, schema checks, required witness collection, ledger recording, and target emission. Agentic orchestration should own the parts that remain open-world: interpreting weak evidence, reading judge rationales, deciding whether a contradiction matters, proposing a repair, and choosing which harness reconfiguration to attempt next.

That is the meta-harness boundary. Code makes the admissible moves explicit but agents reason over the evidence and decide which move to try. The goal is not to cover every case with heuristics, but to preserve enough structure and evidence that open-world judgement can act without becoming invisible.

A task world is not merely a bundle of files and tests. It is structured by logic. It says how the world decides, how it proves, how it survives contradiction, and how it knows when its own language has failed.

Failure Is A World Relation

Once the world has logic, failure stops being a single category.

A low score is not automatically a model defect. The same score can come from many failures:

the model used the wrong method;
retrieval exposed the wrong source;
a unit binding failed;
the output contract rejected the shape;
the verifier checked the wrong proposition;
the task hid a required assumption;
the harness failed to preserve the needed evidence;
governance blocked a necessary action;
the world contained a contradiction the schema failed to represent.

Those failures need different repairs. Model weakness is only one possibility. Sometimes the agent needs a better tool. Sometimes the verifier needs a new field. Sometimes the task wording needs to expose missing information. Sometimes the output contract is too brittle. Sometimes the world itself counted the wrong multiplicity.

This is why provenance belongs inside the harness.

Observability is often treated as something added after the run. For task worlds, observability is part of the world. The world decides which traces exist, which artifacts are durable, which side effects are visible, which claims have witnesses, and which authorities inspect which evidence.

Reward has the same problem. A scalar score looks clean, but reward is an engineering artifact. It contains assumptions, tolerances, field bindings, authority choices, and blind spots.

The diagnostic triangle is:

raw output: what the model actually wrote;
verifier rule: what the scoring code expected;
semantic behavior: what the answer means in context.

When those disagree, the repair target is not obvious. Evidence determines whether the fault lives in the prompt, task, verifier, schema, evidence rule, governance policy, or world shape. Without preserved evidence, every diagnosis becomes a story told after the fact.

Task worlds therefore need evidence contracts: rules for what must survive before a result can be trusted, debugged, or repaired.

Events Force New World Language

Ordinary failure is nameable inside the current world.

The schema was malformed. The required file was missing. The test failed. The wrong unit was used. The reviewer rejected the output. These failures can be serious, but the world already knows how to classify them.

An Event, in this engineering translation of Badiou, is different. A harness Event is a trace that the current world cannot adequately name, but that is important enough to force a new distinction into the world generator.

Examples:

the benchmark score says failure because an expected output was missing, but durable artifacts show the agent created the required file in an unmodeled location;
the model appears incapable until raw output shows it was trapped by an invisible protocol mismatch;
a permission policy forbids shell mutation, but equivalent mutation happens through file-edit affordances;
a generated task passes tests while damaging a state dimension the verifier never inspected;
a reward rule rejects an answer a domain reviewer would accept, or accepts an answer a domain reviewer would reject.

Each case shows a world whose evaluative language was inadequate.

Fidelity is the repair discipline after such a trace. Do not smooth it away. Do not file it under generic model failure. Name the anomaly, decide which layer failed, add the missing distinction, rerun the affected world and nearby regression worlds, and preserve the lineage.

That gives a stronger interpretation of harness evolution:

Evental traces force new distinctions into the world generator.

The same trace should also change downstream agentic action. Once the anomaly has a name, an agent is not merely asked to try again inside the old world. It gets a revised action space: gather missing evidence, route to reviewer, patch a verifier, rebind an output path, tighten a permission model, compose a different evidence rule, or rerun nearby worlds.

That is a higher bar than "let an agent improve the harness." Fidelity becomes an action policy: preserve the trace, act through named repair moves, and make the next run test whether the new distinction holds. This protects against the lazy improvement loop where every failure becomes a broader prompt, a looser verifier, or a special case.

The harder repair question is:

What did this failure make visible that the previous world could not name?

That question is the beginning of innovation. Not novelty as decoration, but a new distinction that changes what future worlds can see, decide, prove, and repair.

Static Harnesses Are Too Rigid

If task worlds vary this much, static harnesses become a bottleneck.

A fixed harness assumes the important parts of the world are already known: the input form, tool surface, context policy, output contract, verifier, evidence rules, permission model, and repair path. That is fine for a stable benchmark family. It is brittle when new problem spaces keep arriving.

Non-IID domains need Plasticity.[^malabou-plasticity] Not chaos. Plasticity.

New problem spaces often need different context loaders, decomposition patterns, verifier families, evidence contracts, permission boundaries, construction gates, or contradiction policies.

If those choices are hand-built every time, the system does not scale. If those choices are hidden inside an agent's free-form reasoning, the system cannot be trusted. We need a layer between those extremes: a meta-harness.

A meta-harness does not treat the harness as fixed background machinery. It treats harness composition as the task.

Given a new problem space, it asks:

what kind of world is this?
what must appear, and to whom?
what can be decided by closure?
what requires constructed evidence?
what contradictions must be contained?
which operations over the world are allowed?
which reviewer, governance, and evidence layers are needed?
which run surface should materialise the result?
which approvals are needed?
what evidence must survive for repair?

This is the layer of operational plasticity agents need. It lets the system adapt to new worlds while keeping the adaptation legible.

Meta-Harness As Operational Plasticity

The meta-harness operating principle is compact: AI proposes; contracts structure; validators check; diagnostics explain; governance gates; emitters write artifacts; execution is explicit; and the ledger remembers.

AI is useful where the world is open-ended: interpreting a messy problem brief, finding similar worlds, choosing candidate operation paths, proposing a workflow, explaining a compatibility failure, or drafting a scoped repair.

The deterministic layer is useful where the system must not improvise: contract validation, schema checks, operation-handle application, evidence collection, governance gates, artifact emission, and ledger recording.

The agent and harness have different jobs.

Agents search. Contracts make proposals writable. Harness-owned code checks whether the proposed process fits. Diagnostics tell the agent or reviewer why it did not fit. Governance decides whether the run can proceed. Emitters produce concrete artifacts. Execution happens as a separate act. The ledger preserves what happened so failures can become repair material.

The result is a controlled way to compose and configure harness layers on demand.

That diagram is the practical form of the philosophical argument. Worlds are not static containers. They are structures that can be described, assembled, tested, contradicted, repaired, and reused.

What This Looks Like In aec-bench

The current AEC-Bench implementation is still early, but it is no longer only a sketch. The contracts are specific enough to review, and the workflow can be driven from the library or CLI.

The core translation is now:

task prose -> ProblemSpaceBrief -> TaskWorldProfile -> logic and operation profiles -> reviewer/governance packets -> recipe workspace -> run evidence and ledger.

The input is what someone asks for. The brief names the objective, task request, evidence requirements, pressure points, and governance constraints. The TaskWorldProfile keeps the task unit, logic profile, and operation profile together. The reviewer layer inspects run evidence after the verifier. The operation orchestrator plans deterministic operations where handles exist and records agentic proposals where they do not. Governance decides whether a proposal is run-only, a world-schema change, or a world-generator change.

The main recipe command is deliberately script-shaped:

aec-bench meta-harness recipe \
  --task-file task.md \
  --baseline-world baseline-world.json \
  --baseline-run baseline-run.json \
  --candidate-world candidate-world.json \
  --candidate-run candidate-run.json \
  --output artefacts/meta-harness/demo

It writes recipe.json, run_recipe.sh, compare_candidate.py, and a README. Once baseline and candidate evidence exist, the generated comparison script writes comparison/comparison.json and comparison/comparison.md.

Here is one concrete case. A benchmark run fails. The model may have used the wrong method, or the verifier may have looked for report.json while the agent wrote the required evidence somewhere else. The meta-harness needs a small diagnosis world around that run: what evidence can be inspected, what actions are allowed, what result should be produced, and what repair evidence must survive.

task_world:
  world_id: benchmark-operations.diagnosis
  name: Harbor trial diagnosis
  task_unit: harbor_trial
  logic_profile:
    closure_gates:
      - id: required_agent_output_exists
        evidence_key: artifacts.agent/output.md
        authority: filesystem
        failure_effect: run_invalid
    construction_gates:
      - id: repair_plan_has_witnesses
        construction_required:
          - trial_record
          - verifier_artifact
          - trace_pointer
        failure_effect: claim_unproven
    containment_gates:
      - id: verifier_artifact_disagreement
        contradiction: "verifier failed but durable artifact exists"
        required_record:
          - trace_pointer
          - score_field
          - diagnosis
        failure_effect: governance_review
    event_triggers:
      - id: verifier_language_gap
        repair_targets:
          - verifier
          - schema
          - evidence
    agentic_review:
      required: true
      review_modes:
        - verifier_result
        - output_artifacts
        - trace
        - contradiction_ledger
  operation_profile:
    projection_axes:
      - artifact_evidence
      - governance
    difference_axes:
      - verifier_surface

In plain terms: this is not "ask a model to debug the run." It is a reusable harness for a specific kind of post-run work. It gives an agent the trial records and verifier evidence, asks for a failure classification and a scoped repair plan, and requires durable artifacts so a reviewer can see why the repair was proposed.

The operation orchestrator makes the autonomous part explicit. It gives an agent an environment, principles, tools, expected artifacts, and examples. The agent may propose a plan, but harness-owned code applies deterministic operations or records proposals. If a projection, difference, subset, or product operation cannot be applied from declared handles, the result is not hidden mutation. It is an orchestration request or a bounded proposal with evidence references.

There is an important negative space here: this is not yet a universal component catalogue or harness compiler. The implemented surface is narrower and more useful: typed world profiles, deterministic operation handles, reviewer artifacts, governance decisions, Harbor import, recipe materialisation, bounded autonomy, and a ledger.

A public API can stay small: send task prose, optional attachments, existing baseline and candidate evidence, and model endpoint configs; get back a problem brief, task-world profile request, reviewer packet, operation plan, governance outcome, recipe workspace, comparison result, and ledger reference.

This is what keeps Plasticity from becoming ad hoc generation. The system can change shape while preserving a structure that someone can review.

Good diagnostics are what let agents participate without hiding the decision.

From Run To Certificate

The trusted output of this system is not a leaderboard row.

It is a certificate.

A certificate is a bounded record of what survived the run. It binds three things: what was attempted, what evidence was preserved, and what result can be trusted.

For an ordinary task world, that means binding four things:

task and inputs;
agent and tools;
output and verifier result;
traces, hashes, and review evidence.

A meta-harness run needs a larger certificate because the harness itself was composed during the run.

Using the existing Harbor machinery, aec-bench can take a natural-language input and stage a problem-space brief task. It can generate or review a task-world profile, run post-verifier reviewer layers over workspace or Harbor evidence, ask the operation orchestrator for a plan, apply governance, materialise a comparison recipe, and import Harbor trial records back into a task_run evidence object.

The autonomous runner is bounded rather than magical. It stops on max iterations, cost, stagnation, governance gates, or missing artifacts. Missing inputs pause the process instead of being invented.

This is what makes harness composition reviewable.

The certificate can be a sidecar file, ledger entry, manifest, report, or bundle of linked artifacts. The format matters less than reconstruction. A reviewer should be able to see what world was built, which harness was composed, why it was allowed to run, what happened, and what evidence supports the result.

That evidence can then be reused. A run can become a verifier result, audit label, dataset row, training example, recipe update, or repair target. The model is not the only thing that learns. The world generator, operation profiles, reviewer layer, and ledger also accumulate distinctions that future runs can use.

Conclusion

Put simply:

Agents need worlds, and worlds need harnesses that can change.

In domains such as engineering and design, this becomes more specific:

In non-IID domains, capability depends on whether we can compose the right world around the task.

That does not make the model unimportant. The model still matters enormously. But the model acts through a world: what it can see, what it can touch, what it must prove, what contradictions it can hold, which failures can be named, and which evidence survives.

Badiou helps us stop treating "world" as a container. A task world counts a multiplicity, regulates appearance, creates decision points, requires witness artifacts, contains contradictions, and sometimes encounters traces its current language cannot name.

The meta-harness turns that into machinery. It describes the world, evaluates profiles, runs review and governance loops, materialises comparison recipes, records what happened, and feeds repair evidence back into the next composition.

That is the layer of plasticity I think agent systems need.

Not infinite autonomy. Not static evals. A compositional layer where new task worlds can be made legible enough to run, inspect, trust, and change.

The model is not the only thing learning.

The harness now has a place to remember.

[^badiou-situation]: A situation is a structured presentation: a way in which a multiplicity is gathered and made intelligible as a domain. In this essay, a task world is situation-like because it gathers files, tools, rules, traces, evidence, and success criteria into one operational setting.

[^badiou-multiplicity]: Multiplicity names Badiou's refusal to begin from a pre-given One. A task world begins the same way: as many heterogeneous things before the harness gives them boundary, order, and relevance.

[^badiou-count-as-one]: The count-as-one is the operation that lets a multiplicity appear as one situation. For harness design, this is the boundary-making act that says which artifacts, affordances, authorities, and claims belong to this task world.

[^badiou-event]: An event is a rupture that the existing situation cannot fully name using its old categories. In meta-harness terms, an event is the kind of failure or trace that exposes a missing distinction in the world itself, not merely a mistake by the agent.

[^badiou-fidelity]: Fidelity is the continued work of drawing consequences from an event rather than treating it as noise. In this article, fidelity becomes repair discipline: preserving the trace, naming the new distinction, and changing the harness so the world can account for it next time.

[^badiou-appearance]: Appearance is the question of how something shows up in a world, and with what force or consequence. For task worlds, this becomes the split between what appears to the agent, the verifier, the reviewer, and the generator.

[^badiou-object]: An object is not an isolated thing; it appears within a world with identity, difference, and relations. In a harness, an object may be a file, claim, tool, artifact, side effect, or evidence record whose meaning depends on the world that presents it.

[^badiou-relation]: Relation names how objects are connected, compared, distinguished, or made consequential inside a world. In a task world, this includes dependencies, permissions, provenance, authority chains, and the links between evidence and claims.

[^badiou-world-logic]: World logic is the local order that governs appearing: what can show up, how strongly it counts, and how differences matter. In meta-harness terms, it becomes the run logic that decides what can be closed, what must be witnessed, and what contradiction can be contained.

[^badiou-forms-of-change]: Forms of change refer to different ways a world can be modified, from ordinary variation to stronger transformations that alter what the world can recognize. For harnesses, this is the difference between adjusting parameters, repairing a verifier, adding a missing evidence relation, or composing a new world shape.

[^malabou-plasticity]: I am borrowing the capital-P sense of Plasticity from Catherine Malabou's work, especially The Future of Hegel, What Should We Do with Our Brain?, The Ontology of the Accident, and Plasticity at the Dusk of Writing. Malabou's plasticity is not mere flexibility. It is the capacity to receive form, give form, and sometimes break or destroy form. In this essay, meta-harness Plasticity means a harness layer that can be shaped by a new task world, actively reshape its own composition, and abandon an old form when an Event shows that the old world language no longer works.

Plausible Answers, Failed Workflows

Theodoros Galanos — Tue, 02 Jun 2026 00:00:00 GMT

import ModelScoreStrip from '../../components/charts/ModelScoreStrip.astro'; import RewardHeatmap from '../../components/charts/RewardHeatmap.astro'; import FailureBars from '../../components/charts/FailureBars.astro'; import FlowDiagram from '../../components/charts/FlowDiagram.astro'; import Pullquote from '../../components/Pullquote.astro'; import Sidenote from '../../components/Sidenote.astro'; import { readbackPipelineData, readbackModelStripData, readbackHeatmapData, readbackFailureData, governancePipelineData, governanceModelStripData, governancePairData, toolResultHeatmapData, monitoringHeatmapData, toolResultFailureData, monitoringFailureData, closeoutPipelineData, closeoutModelStripData, closeoutPairData, shardHeatmapData, migrationHeatmapData, shardFailureData, migrationFailureData, groundingPipelineData, citationDrawingPairData, drawingModelStripData, citationTaskHeatmapData, drawingTaskHeatmapData, citationFailureData, drawingFailureData, styleTransferPipelineData, styleRecordRepairPairData, styleModelStripData, recordTaskHeatmapData, repairTaskHeatmapData, recordFailureData, repairFailureData, retryArtifactPipelineData, scaffoldLadderData, scaffoldMatrixData, scaffoldSplitData, verifierRetryFailureData, scaffoldReductionFailureData, } from '../../data/aec-bench-release-charts';

Estimated reading time: 36 minutes

The simplest way to read a model evaluation is to ask which model won.

For engineering agents, that is not enough.

Engineering work is not only a question-answering problem. An agent has to read a brief, find the right quantities, decide whether to reach for a tool, preserve the source it was given, write something durable, and return an answer another system can check. The score at the end is a compression of all of that. Read only the score and you lose the part that actually predicts whether the agent is safe to put in a workflow.

This post reads one such evaluation the other way around. It draws on a recent release sweep across the benchmark, organised as a sequence of chapters. Each chapter takes one group of related tasks, looks past the single reward number, and asks a narrower question: not was the answer right, but did the workflow survive the model.

<Pullquote> A model can produce a plausible answer and still fail the workflow. </Pullquote>

That sentence is the recurring distinction across every chapter. The hard part is rarely whether the model knows the right answer. It is whether the model preserves source truth, writes durable artifacts, follows control boundaries, keeps state aligned, and produces evidence that another system can verify.

This is a quality-assurance lens, not an intelligence test. We start where the gap is widest.

What's coming

Six chapters, one moving question — did the workflow survive the model? — asked at six escalating points in an engineering agent's job:

Readback Is Not a Receipt — can the model leave a durable, self-consistent record at all? Mostly not: the best model sits at 0.70 on tasks that are not, on paper, hard.
Evidence Is Not Authority — can it decide which signal is allowed to govern a decision, and record why? The judgment is usually fine; the provenance fields are where it slips.
Closeout Is a Dependency Claim — can it keep a whole dependency graph straight when many required items must clear at once? It gets the verdict and drops the lineage.
Evidence Has Coordinates — can it bind an answer to the exact source or coordinate frame? The first place the strong models simply pass — a clean floor test.
Style Is Not a Waiver — can it rewrite a document for a new audience without quietly laundering the risk inside it? Yes for the prose; the audit artifact is the hard part.
Feedback Is Not Execution — when the workflow says a file is missing, does it write the file or just claim it did? Retry helps only when execution is made observable.

The thread through all six: a plausible answer is cheap, and the models that lead this benchmark are the ones that treat the durable artifact — not the chat reply — as the real deliverable. The conclusion pulls the cross-cutting findings together.

Chapter 1 — Readback Is Not a Receipt

The first task group is deliberately small: an anomaly-review exercise built around readback<Sidenote>Readback means re-opening a record you just wrote and confirming, field by field, that it matches what you decided — the audit equivalent of a control-room operator repeating an instruction back to prove it landed.</Sidenote>. Six models, six review tasks, thirty-six trials in total.

Each model receives a small review packet[^packet] — a policy, the underlying source records, a draft analysis, a review request, and a note applying pressure to just approve it — and has to decide whether the draft can be released. That decision is the easy part. The actual test is what happens after it.

The model has to write a structured review record, then read that record back into a separate check, and finish with a short quality-assurance note whose fields agree with both. The answer has to survive as files on disk, not just as text in the reply.

There are three places to lose credit, and only the first is about engineering judgment. The other two are about whether the judgment became governed project state: a record with exact IDs, an owner, a fingerprint, and a readback that genuinely matches the record it claims to check.

Result snapshot

Across 36 trials the mean reward was 0.412. The strongest model, gpt-5-3-chat, averaged 0.695. No model came close to topping out a group of tasks that, on paper, is not hard.

The strong models all entered the right workflow shape: a compact final answer, a record file, a readback file. Their misses were dull — owners, fingerprints, exact evidence IDs. The interesting rows are at the bottom.

gpt-oss-120b mostly failed at runtime: five of six trials never completed, so there was nothing to audit. That is a harness failure, and it is worth keeping separate from a reasoning failure.

deepseek-v3-2 is the more instructive negative example. It completed all six tasks and scored zero on all six. Its outputs were enormous — a mean of roughly 116 KB against the ~400 bytes the strong models emitted — full of planning text, pseudo-tool calls, and narrated file-writing. The verifier found no preserved artifacts. A great deal of text was produced. The project state never changed.[^contract]

<Pullquote> A transcript can look like work while failing to leave project state. </Pullquote>

Where the credit lands

Breaking the same trials out by task shows that the failures are both model-specific and task-specific, and that almost everything lives in the partial band<Sidenote>Each trial scores between 0 and 1. A few near the top mean the record was almost complete; zeros mean nothing usable survived. The “partial band” is the wide middle — here roughly 0.4 to 0.7 — where the output has the right structure but enough wrong or missing fields that no downstream system could safely consume it.</Sidenote>.

There is exactly one full pass in the whole group — gpt-5-3-chat on the case where the draft total conflicts with the source — and one strong partial, the same model on the clean, releasable case. Everything else for the competent models sits between 0.5 and 0.61: the right shape, but not accurate enough to be a record another system could trust and ingest. deepseek-v3-2 is a solid block of zeros despite completing every run.

That partial band is the whole point. The grading is punishing almost-correct project state, which is exactly the right thing to punish for audit work. A record that is 90% right is not 90% useful to the system that ingests it.

Failure fingerprint

Sorting the checks by how often they were lost makes the failure family obvious.

Almost every lost check is about readback equality and provenance, not the engineering call itself. The check that the readback actually matches the record failed in 35 of 36 trials — the readback step almost never became the receipt it was supposed to be. Owners, fingerprints, evidence IDs, and issue IDs follow right behind.

What this chapter teaches

Readback is a control surface, not a guarantee: re-reading a record only becomes a receipt when the record and the readback carry exact, matching fields — and here they almost never did. Writing the files is necessary but not sufficient; the strong models reached the right shape and still lost credit on owners, fingerprints, and exact IDs, which is why even the best sits at 0.695 rather than near 1.0. The remaining gap is not dramatic reasoning — it is exactness, the boring fields a downstream system actually ingests. Downstream systems do not ingest vibes; they ingest fields.

<Pullquote> An answer convinces a reader. An artifact survives an audit. </Pullquote>

That is the interpretive key for everything that follows. Chapter 2 moves from generic audit state into operational decision records, and asks a question prior to can you record it: which evidence is even allowed to govern the decision?

Chapter 2 — Evidence Is Not Authority

Chapter 1 asked whether a model could leave a durable record behind. Chapter 2 asks something prior to that: can the model decide which evidence is even allowed to govern a decision?

Operational systems rarely fail because a signal is missing. They fail because the wrong signal is handed authority. A stale dashboard looks reassuring. A cached tool pass looks finished. A retry quietly supersedes the failed attempt before it. A warning permits limited progress but not clean release. A current breach should override a green summary. A maintenance window changes whether a reading even counts.

This chapter uses two related task groups from the same sweep, both built around that authority boundary.

In the first group, the governing evidence comes from tool outputs — a timeout, a failed-then-retried check, a stale cached pass, a pass carrying a warning, a clean pass, an unsupported-tool gap. The model has to decide which result may govern closeout and write a decision record that matches its visible summary.

In the second group, the evidence comes from monitoring telemetry[^telemetry] — stale dashboards, current samples, active thresholds, sensor faults, maintenance windows, and live breaches. The model has to settle on an operational status and preserve the fields that made that status valid.

In both, being cautious is not the same as being correct. The decision has to name the right status, cite the exact evidence that governs, reject the exact evidence that does not, and record an owner and action a later workflow can audit.

<Pullquote> A system fails when the wrong signal is handed authority. </Pullquote>

Result snapshot

Across both groups the strong models look healthy. The four GPT and Grok models complete every run and sit near 0.85–0.90. The interesting rows are, again, at the bottom — and they fail in two completely different ways.

deepseek-v3-2 repeats its Chapter 1 signature: it finishes every run, emits tens of kilobytes of narration, and leaves almost no verifier-visible record. gpt-oss-120b is the opposite failure — it writes clean records when it finishes, but four of its twelve runs never complete at all.

That split is worth holding onto, because a single leaderboard number would blur it. Two different questions are in play: does the model finish the run, and when it finishes does it satisfy the record contract? A model can be good at one and bad at the other.

<Pullquote> Finishing the run and satisfying the record are different competencies. </Pullquote>

The two evidence sources are not equally hard. Tool-result decisions average 0.71; monitoring averages 0.67.

The gap is intuitive once you look at the inputs. A tool output arrives pre-labelled — it already says "timeout" or "stale cache" or "unsupported." The model still has to apply policy, but the input carries a decision vocabulary. A monitoring status has to be synthesised: the model has to weigh freshness, threshold authority, sensor validity, maintenance context, and breach state before it can name a status at all. More synthesis, more ways to be partly right.

Where the hard cases are

Breaking each group out by task shows that the hard cells are not scattered randomly. They sit exactly where authority has to be taken away from the signal that looks safest — the green dashboard, the cached pass that already reads as finished, the result that says the work is done — and given instead to a less convenient one: a fresher sample, a current breach, a threshold that has since changed.

In the tool-result group, the standout is the stale cached pass: gpt-5-3-chat takes the only full pass on it, correctly refusing a cached result that was computed against a superseded demand. The hardest column is the unsupported-tool gap, where the model has to fall back to a manual source and disclose that the tool never ran — the one place gpt-oss-120b fails outright.

Monitoring tells the same story more sharply. The healthy-current and stale-dashboard controls are handled well; the difficulty concentrates on the superseded threshold, the maintenance window, and the current breach over a green summary — the three cases where the reassuring signal is the wrong one to trust. gpt-oss-120b is the cautionary row: a perfect stale-dashboard record sitting next to three runs that never completed.

Failure fingerprint

Sorting the lost checks makes the failure family obvious, and it is the same family in both groups: governance fields, not judgment.

The single check that tops both panels is policy followed — failed in 28 of 36 tool-result trials and 34 of 36 monitoring trials. That sounds damning until you read why it failed. It is almost never failing because a model cleared an asset it should have held. It fails because a policy-required field underneath it is missing, imprecise, or contradicted by the final summary: an evidence ID that does not match, a threshold not preserved exactly, an owner name that drifts, a status code that disagrees with the prose.

A small but sharp example sits inside the monitoring group. On the stale-dashboard task, one model correctly refused to certify the asset as healthy — the dashboard was green but stale, and the latest sample was outside the freshness window. It still lost credit, because it recorded the status as limited / inconclusive rather than hold / not clear.[^status] In language those sound equally cautious. In the record they are different operational states, and they route to different downstream handling.

What this chapter teaches

Caution in prose is not caution in the record: the strong models say the stale dashboard or cached pass cannot govern, then fail to preserve the exact rejected evidence, governing evidence, or threshold that proves it, and a reviewer inspects the record rather than the intent. Status codes are part of that proof, not wording preferences; hold and limited sound equally careful and route to different downstream handling. Governance fields (evidence IDs, thresholds, owners, action and status codes) are first-class outputs rather than formatting details: the policy was not followed unless the record says why.

<Pullquote> Evidence becomes governing evidence only when the record says so. </Pullquote>

Chapter 3 raises the difficulty again: closeout under dependency constraints, where authority is no longer one packet but spread across required shards, gate results, stale replacements, and waivers the model has to keep straight at once.

Chapter 3 — Closeout Is a Dependency Claim

Chapter 2 asked which single signal may govern a decision. Chapter 3 asks a harder version of the same question: when a decision depends on many required items at once, can the model keep the whole dependency graph intact?

Closeout<Sidenote>Closeout is the moment a package is formally declared finished — signed off as complete and cleared to move to the next stage. It is the gate every dependency has to clear before downstream work is allowed to begin, which is what makes it an audit point rather than a status update.</Sidenote> is the obvious place to test it. Percent complete is not closeout. A dashboard can say most shards passed; a migration table can show most gates green; a waiver can look administratively convenient. None of those facts close the package on their own. Closeout is a claim that every required dependency has current, valid, auditable evidence — and the record has to carry that whole trail.

This chapter uses two closeout task groups from the same sweep.

The first splits a package review into parallel discipline shards<Sidenote>A shard is one slice of a larger job run on its own — here, one discipline's checks (say, all the structural ones) processed as a separate unit. The package only closes when every required shard has come back complete and passing, so a shard that was never run, cancelled, or left stale is a hole in the dependency graph, not a rounding error.</Sidenote> — separate checks running in parallel across disciplines like mechanical, electrical, and structural. The model sees the manifest of required shards (the authoritative list of which slices must come back), the shard results, an aggregate dashboard, and pressure, and has to decide whether the batch is ready, held, incomplete, or ready-with-warning.[^closeoutstatus]

The second is a standards migration — moving a project from an old set of design standards onto a new one, which is only permitted once a fixed set of checks has passed. Each of those required checks is a gate. The model sees the gate manifest (the authoritative list of which gates must pass), the gate results, and a waiver register — the log of formally authorised exceptions, where someone with the right authority has signed off on skipping a specific gate. Its job is to decide whether the migration can close: accepting current passes, rejecting stale or failed results, and honouring only the waivers that are genuinely authorised.

<Pullquote> A package does not close because most of its checks passed. It closes when every required one is accounted for. </Pullquote>

Result snapshot

The two groups average almost identically — 0.64 each — but that number is misleading, and worth taking apart.

For the four artifact-producing models, closeout is largely a solved problem: they complete every run and sit between 0.89 and 0.96. The suite mean is dragged down by the same two failures from earlier chapters, now in sharper relief. deepseek-v3-2 completes all twelve runs and scores zero on every one — it leaves no closeout record. gpt-oss-120b is the mirror image: eleven of its twelve runs never complete, but the single run that did was a flawless shard record.

That is the cleanest illustration yet of why one number is not enough. "Can produce a perfect closeout record" and "reliably finishes the task" are different claims, and gpt-oss-120b satisfies the first while failing the second.

The two settings are not equally clean for the strong models. Per model, migration gates run slightly ahead of shard closeout.

The gap is small and the direction is intuitive: a migration gate is a single named check with a pass, a fail, or a waiver, whereas a shard batch forces the model to reconcile a manifest against partial dashboards, cancellations, and stale runs before it can even name a status.

Where the hard cases are

Breaking each group out by task shows the difficulty sitting exactly where a dependency has to be rejected in favour of a less convenient truth.

The encouraging result is the partial-dashboard task, the highest-scoring shard case. The dashboard sampled only part of the manifest and looked comfortable, but two required shards were never run. Every artifact-producing model held the line — marking the batch incomplete, citing the passes it did have, and naming the missing shards rather than trusting the summary. The hardest task is the cancelled shard, where a cancelled dependency is simultaneously a rejected result and a missing required completion, and the record has to carry it as both.

Migration tells the same story. Clean closeouts and unrun-gate cases are handled well; the difficulty concentrates on the unauthorized waiver, where a convenient-looking waiver attached to a failed gate has to be rejected while the blocker and the failed result both survive into the record.

Failure fingerprint

Sorting the lost checks shows the same family failing in both groups, and it is not the headline decision. The models almost always get the ready-or-hold call right. What they drop is the bookkeeping underneath it: the dependency lineage (exactly which results were used, which were rejected, which were missing) and the status code, the machine-readable outcome a downstream system actually routes on.

Policy followed tops both panels, and for the same reason as Chapter 2: it almost never fails because a model closed a package it should have held. It fails because the lineage underneath it is incomplete — a result that was used or rejected but not recorded exactly, a missing shard left off the list, a blocker that did not survive into the record, a status code that disagrees with the prose decision.

The recurring pattern is that dependency categories are not interchangeable. A cancelled shard can be rejected evidence and still leave a required shard incomplete. A failed gate can be listed as failed and still need to appear as a rejected result. An unauthorized waiver can be rejected while the failed gate remains the blocker. These are small differences in a written explanation and large differences in a workflow state machine.

What this chapter teaches

A summary is context, not authority: the partial-dashboard result is the good news — the strong models reliably refused to let an aggregate dashboard stand in for shards it never sampled, the authority boundary holding under load. But closeout can be correct and still unauditable. The models almost always get the ready-or-hold verdict right; what they drop is the trail underneath it — which required dependency failed, which result was used, which was rejected, which blocker applies. That trail is the proof, not the verdict.

<Pullquote> The record that closes a package has to carry every dependency it cleared — not just the word done. </Pullquote>

Chapter 4 changes the kind of evidence entirely — from dependency bookkeeping to grounded text and spatial geometry — to ask whether the same record discipline holds when the truth lives in a source span or a coordinate frame.

Chapter 4 — Evidence Has Coordinates

The first three chapters were all about records: what a result says, which signal may govern, which dependencies have to clear. This chapter changes the kind of evidence and asks a more basic question underneath all of them — can the model bind its answer to the right evidence in the first place?

Evidence can live in a paragraph. It can also live in a coordinate system. This chapter uses two task groups that test each.

The first is textual. Each task carries three engineering claims — a required bearing capacity, a commissioning pressure, a voltage drop — and six nearby source files, some of which are superseded revisions or scope-adjacent distractors. The model has to return, for every claim, the exact source it came from and the exact value, while refusing to cite a distractor or invent an unsupported claim.

The second is spatial. Each task gives the model a current drawing and a superseded one, both as machine-readable vector geometry,[^vector] plus a transform (a scale, a translation, a rotation), a schedule of required clearances, and a policy. The model has to apply the transform before it measures anything, reject the superseded drawing, decide whether the geometry clears the requirement, and write a geometry record that matches its own visible summary.

<Pullquote> An answer is only as good as the evidence it is bound to — the right source, or the right coordinate frame. </Pullquote>

Result snapshot

This is the first chapter where the strong models simply pass. Putting the two surfaces side by side shows it clearly.

The four strongest models bind text and geometry exactly — a clean ceiling that the earlier chapters never produced. That is worth stating plainly: in this controlled setting, exact source-and-value citation and text-visible transformed geometry are solved for the top tier. It is also worth bounding carefully, which the closing of this chapter does.

The interesting structure is at the bottom, and it is the same split as Chapter 3 wearing different clothes. gpt-oss-120b fails citation outright — every run failed to produce a usable claim map — yet turns in partial competence on drawing when its runs complete. deepseek-v3-2 is the mirror image: it produces a few citation partials but leaves no geometry record at all.

The record contract only bites on the drawing side — a citation answer is a final-answer JSON map with no separate artifact, whereas a drawing task demands a written geometry record — so the per-model record detail is worth looking at there.

The same two tells from earlier chapters reappear in the output column: deepseek-v3-2 emits a long narration (tens of kilobytes) and produces no record; gpt-oss-120b completes only half its runs. Above them, the GPT and Grok outputs are compact and exact.

Where the hard cases are

Because the strong models saturate, the task grids read differently from the earlier chapters: the colour now lives almost entirely in the bottom two rows.

Citation barely varies by discipline. Every strong model is perfect on bridge bearings, commissioning pressures, egress methods, and geotechnical limits alike. The only variation is whether the weaker models can produce an exact map at all — and largely they cannot.

Drawing has one genuinely hard case: the rotated drain arrow. It is the single task that trips an otherwise-perfect model — Grok — because the direction relation only flips after the 180° rotation is applied. Read the geometry in the original frame and the arrow looks fine; read it in the transformed frame and it conflicts. Translations and scales are far more forgiving, because they move a point without changing what "which way does it point" means.

Failure fingerprint

The two surfaces fail in shapes that match their structure — and the contrast is the point.

Citation failures are concentrated: a single cluster of five, which is simply the two models that could not produce an exact map. Drawing failures are broad and coupled. A wrong transformed measure does not fail alone — it drags the threshold comparison, the conflict flag, the decision code, and the record that summarises them all down with it. Spatial grounding has more fields that have to agree at once, so a single measurement error shows up as a row of red.

That coupling is the most useful thing the drawing panel says. The misses are not "the model could not produce a record." They are "the record exists, the schema is mostly right, and one transformed number is wrong" — which then propagates. Grok's rotated-arrow miss is exactly this: correct drawing IDs, correct transform ID, correct rejected-evidence trail, and a wrong relation after the rotation.

What this chapter teaches

Grounding is not one skill: text grounding binds a claim to a source-and-value pair, spatial grounding binds geometry to a transformed coordinate frame, and a model can be excellent at one surface and fail the other on completion alone. Where the strong models do fail spatially, it is a higher-quality failure than the no-record collapse of earlier chapters — the record exists and one transformed number is wrong, which a benchmark can isolate and drive down. And saturation is itself a finding: when the top tier sits on the ceiling the suite stops discriminating between them, which makes it a clean floor test for everything messier that follows.

<Pullquote> In compact, text-visible settings, the strong models align evidence exactly. The open question is whether that survives messier retrieval and richer drawings. </Pullquote>

Chapter 5 applies pressure of a different kind: not whether a model can find the right evidence, but whether it can rewrite a document for a new audience without quietly dropping the material risk inside it.

Chapter 5 — Style Is Not a Waiver

Chapter 4 asked whether a model could bind its answer to the right evidence. This chapter keeps the evidence fixed and changes the output: rewrite this technical note for a different audience. It is one of the most ordinary requests a model gets — and a quietly dangerous one.

The rule is simple to state. Change the style; do not change the engineering truth. The difficulty is that the requested styles are exactly the ones that create pressure to move it. "Client-positive" leans on you to soften a hold. "Marketing brief" leans on you to omit a blocker. "Plain language" leans on you to drop the caveat that made the original sentence true. A rewrite can be clearer, friendlier, more polished — and launder material risk on the way through.[^laundering]

These tasks do not score prose quality. They score whether status, numeric values, caveats, issue IDs, and source IDs survive the rewrite and a structured integrity report. This chapter uses two groups: a first-pass rewrite, and a repair group that hands the model an already-laundered draft to fix.

<Pullquote> A style request can change the wording. It must not change the status. </Pullquote>

Result snapshot

The headline result is not the suite means — 0.61 and 0.64, barely apart. It is what happens to the strong models when you give them a draft to repair.

On the first pass, the four artifact-producing models cluster around 0.85–0.89. Show them a laundered draft and require a readback, and three of them jump toward the ceiling — gpt-5-3-chat to 0.9955, gpt-5-2 to 0.982, Grok to 0.96. The exception is gpt-5-1, flat across the two, for a reason worth holding onto: its repair residuals are not in the prose but in the readback artifact.

The bottom of the table is the familiar split. gpt-oss-120b manages one decent first-pass record, then fails every repair run outright — the rewrite-report-readback chain is simply too many artifacts. deepseek-v3-2 completes everything and scores near-zero on both: it leaves no integrity report to grade.

Both groups demand written artifacts, so the record contract bites on both sides. The output column shows the same two tells from earlier chapters — deepseek-v3-2's tens-of-kilobytes narration with nothing to audit, and gpt-oss-120b's incompletion.

Where the hard cases are

Splitting each group by style shows where the pressure actually lands.

The encouraging first-pass result is that the prose stays risk-honest across every style — the strong models do not uplift a hold to please a client-positive brief or drop a blocker for a marketing one. What separates the styles is report exactness. The risk-register and plain-language tasks, which carry the most fields to preserve, sit lowest, because that is where an exact-evidence-ID or caveat-preservation field is most likely to slip.

Repair lifts almost everything. With a laundered draft to react to, even the marketing-omission and blocker cases — the hardest first-pass tasks — become tractable, because the model is now correcting a concrete error rather than guarding against an abstract one. The clean control matters here too: one draft is not laundered, and a good model has to leave it alone rather than invent a repair.

Failure fingerprint

The two groups fail in different places, and the shift is the whole point of adding repair.

The first pass fails in the report, not the prose: evidence IDs listed exactly fails in 34 of 36 trials. The visible rewrite can say exactly the right thing while the integrity report loses the linkage that would let an auditor trace it back to source. Repair moves the failures downstream — into whether the draft was actually repaired, whether unsupported softening was flagged, and whether the readback confirms the report. That readback is, once again, a separate competence: gpt-5-1 repairs every rewrite cleanly and still loses points because its readback artifact has the wrong shape — the Chapter 1 lesson returning in a new setting.

What this chapter teaches

Style transfer is not the hard part: the models write good audience-appropriate prose and largely keep it risk-honest, not uplifting a hold to please a client-positive brief or dropping a blocker for a marketing one. The hard part is the audit artifact — a rewrite that reads safe is not a report that proves it stayed safe, and the recurring first-pass miss is the exact evidence ID that makes it traceable back to source. Repair turns out to be a control surface in its own right: handing the model a laundered draft and demanding a readback recovered risk the first pass had blurred, and lifted the strongest models toward the ceiling. And a clean control keeps repair honest — one draft is not laundered, and "repairing" it anyway distorts the source just as surely as laundering does.

<Pullquote> The right control for a rewrite is not to forbid it — it is to make it prove the risk survived. </Pullquote>

The final chapter closes the loop. Every chapter so far has leaned on the model producing the right artifact; the last asks what happens when the workflow tells the model an artifact is missing — whether it can recover under retry, or whether the record discipline breaks the moment the runtime pushes back.

Chapter 6 — Feedback Is Not Execution

Every chapter so far has assumed the model gets one clean shot at the artifact. This one removes that assumption. The workflow now pushes back — the required file is missing — and the question is what the model does next. Does it create the file, or does it write a sentence claiming the file exists?

The distinction is trivial in a chat window and decisive in a workflow. "I wrote the integrity report" and an actual integrity report sitting in the workspace look identical in the transcript. Only one of them can be audited. A downstream system cannot read a claim that lives in the final answer; it needs durable files, the right schema, and a way to prove the files are the ones the workflow expects.

To pull those concerns apart, this chapter walks a scaffold ladder — four retry suites over the same style-transfer repair task, each giving the model more workflow support than the last. The first offers only a fixed "file missing" note. The second adds a real second turn after the verifier rejects the first attempt. The third makes a helper step mandatory and requires it to stamp an execution marker.[^marker] The fourth pulls half of that scaffold back out. The point of the ladder is not that more support is always better — it is to find which rung actually makes execution observable.

<Pullquote> Saying a file was written is not the same as writing it. </Pullquote>

Result snapshot

The suite means barely move — 0.70, 0.63, 0.69, 0.67 — and they hide the story rather than tell it, because two models complete almost every run while leaving nothing durable behind.<Sidenote>deepseek-v3-2 finishes its runs and scores near-zero because it writes no verifier-visible artifacts; gpt-oss-120b mostly fails to complete the artifact stack at all. Averaging them in flattens the ladder.</Sidenote> The four artifact-producing models tell it clearly.

Read top to bottom, the strong-model column climbs and then steps back. Static feedback already lands at 0.9977 — capable models treat writing the file as part of the job, no live loop required. A real retry turn actually drops the strong mean to 0.8935, because a second turn is stricter about proving the side effects happened. The mandatory helper is the one rung where every strong model hits a clean 1.00. Then the reduced scaffold pulls it back to 0.9087.

The per-model grid shows where the movement comes from. The four strong models are near-saturated everywhere except two places: gpt-5-2's live-retry trough at 0.67, and the shared dip on the reduced scaffold. The mandatory-helper column is the only one where all four hit 1.00 — and, tellingly, the only column where gpt-oss-120b falls to zero, because being required to run the helper is harder for it than being allowed to skip it.

gpt-5-2's trough is the cleanest single result in the chapter. On a marketing-noise retry it writes the integrity report and the readback check correctly, and the visible rewrite preserves the exceedance — 62 dB against a 55 dB limit, mitigation unapproved, activity on hold. The files are right. What it omits is the compact final summary, so the workflow has no proof to credit. The side effects happened; the final contract did not say so. Nothing is wrong with the engineering — the failure is entirely about proving the work.

Where the hard cases are

The reduced-scaffold suite is the one worth splitting, because it is built to isolate which part of the scaffold was load-bearing. Half its tasks hand the model a malformed helper payload to repair and then run; the other half take the helper away and make the model write the execution marker by hand.

The split is stark and consistent. Every strong model holds a perfect 1.00 when it only has to fix a malformed payload and let the helper run — and every one of them drops to about 0.81 when it has to produce the marker itself. The misses are not in the prose or the source evidence; they are in the execution metadata. In one plain-language flood case gpt-5-1 preserves the 0.35 m freeboard and the provisional tailwater caveat, writes the report and readback correctly, then marks the payload as repaired when manual mode required the opposite. A small field, exactly wrong, and the marker fails.

gpt-oss-120b inverts the pattern in a way that confirms it: forced to repair the malformed helper it scores zero, but left to write the marker by hand it reaches a partial 0.54. Repairing a structure that already exists is a different skill from producing the proof-of-execution from scratch, and the durable boundary is the second one.

Failure fingerprint

The whole chapter is in how the failures move as you climb the ladder.

In the live-retry suite the failures still cluster on the files and the final contract: the report and readback go unwritten, and the distinctively retry-shaped miss is the side-effect JSON, which either floods the final answer or vanishes from it entirely.<Sidenote>The contract is deliberately narrow: the durable files are the evidence, and the final answer is only a compact status surface. Pasting the full side-effect JSON into the answer is a leak; omitting it is a missing proof. Both fail.</Sidenote> Pull the scaffold and the surface shifts. The file-and-report checks that dominated earlier fall to ten; the new top of the list is the marker and the repaired payload — proving the step ran, in the exact shape, without a helper to lean on.

What this chapter teaches

A retry loop is not a recovery loop: the live-retry suite has a genuine second turn and still scores below the static one — 0.8935 against 0.9977 — because it asks for harder proof after the turn. Feedback hands the model another chance; it does not perform the fix on the model's behalf. The mandatory helper earns its place by making execution observable (run the helper, write the report and readback, stamp the marker, emit the summary), and that is the one rung where every strong model saturates. Pull the helper and the last boundary shifts to the metadata: the models keep writing good rewrites and lose points on marker exactness. The fix is not less scaffold but targeted scaffold, aimed only at the field still slipping.

<Pullquote> A model has not repaired the workflow until the files, markers, and final contract all agree. </Pullquote>

What the six chapters add up to

Read end to end, the six task groups are one escalating question — did the workflow survive the model? — and they share one answer. The hard part is almost never the engineering judgment. The models name the right status, hold the right line, preserve the visible risk. What they drop, over and over, is the part that makes the judgment usable to another system: the exact field, the matching readback, the dependency trail, the proof that a file was actually written.

Four patterns hold across every chapter.

Trust has two axes, and one number hides them. Whether a model finishes the run and whether it satisfies the record contract are different competencies. deepseek-v3-2 finishes everything and leaves nothing durable; gpt-oss-120b writes clean records but often never completes. A single leaderboard mean blurs two opposite failures into one mediocre row.

The failures live in provenance, not the verdict. Policy followed is the most-failed check in chapter after chapter — and it almost never fails because a model cleared something it should have held. It fails because the field underneath it is missing or imprecise: an evidence ID that does not match, a threshold not preserved, a status code that disagrees with the prose. The reasoning is sound; the record cannot be audited.

Proof is a separate skill from the work. The readback that has to equal the record, the integrity report that has to trace to source, the marker that has to prove the step ran — every chapter has a version of this, and it is consistently where the strong models lose their last points. Doing the work and proving the work are not the same competence, and a workflow depends on the second.

Verbosity is not execution. The recurring negative example, deepseek-v3-2, produces the most text and the least durable state across every chapter it appears in — though in its case that is partly an interaction-contract failure rather than a pure behavioural one.[^contract] Either way, long, fluent narration about writing files is the failure mode to watch for in any agent put near a real workflow.

The practical takeaway is narrow and useful. The model that tops a prose leaderboard is not automatically the model you can put in an engineering workflow. The one you can trust is the one that treats the durable artifact — not the chat reply — as the real deliverable: it leaves exact records, binds answers to exact evidence, preserves risk through a rewrite, and proves that the side effects happened. A plausible answer is cheap. A workflow that survives the model is the thing worth measuring — and the thing these six chapters were built to measure.

[^packet]: Concretely: the policy might be "do not release if any quantity disagrees with the source by more than 2%"; the source records are the original measured values; the draft analysis is a write-up claiming the design is clean; the review request is "approve for release"; and the pressure note is a line like "the client needs this signed off today."

[^contract]: One caveat about deepseek-v3-2 specifically. Reading its raw outputs, most of its zero scores look like an interaction-contract failure rather than a clean capability measurement: nearly all of its runs were captured as raw output instead of adapter-written results, with reasoning markers and textual tool-call plans standing in for actual tool calls and verifier-visible files. Those runs are valid failures under the harness — the durable artifact genuinely never lands — but they are better read as adapter/protocol misalignment than as evidence about the model's domain reasoning. The clean test would be a small control rerun under a stricter "final JSON only" contract or a provider-specific tool-call shim; until then, treat this model's rows as contract-contaminated, and read it throughout as an illustration of the workflow failure mode, not a verdict on the model. [^telemetry]: Telemetry just means the live readings a system emits about itself — sensor samples, dashboard summaries, threshold alarms, trend logs. The catch is that not all of it is current or authoritative: a dashboard can be green but hours stale, or a green reading can sit under a threshold that has since been superseded.

[^status]: In a governed workflow these are codes, not adjectives. Hold / not clear says the asset cannot proceed; limited / inconclusive says it can proceed under caveats. A downstream system reads the code, not the surrounding sentence, so a cautious-sounding paragraph attached to the wrong code still routes the asset the wrong way.

[^closeoutstatus]: Each closeout resolves to one status code, not a sentence: ready (proceed), hold (a required item failed), incomplete (a required item is missing or cancelled), or ready-with-warning / ready-with-authorised-waiver (proceed under a recorded caveat). As in Chapter 2, the downstream system routes on the code, so the right code matters as much as the right reasoning.

[^vector]: Vector geometry means the drawing is stored as shapes and coordinates — points, lines, transforms — in text the model can read directly, rather than as a flat image of pixels. That distinction matters: it means these results show the model reasoning about geometry it can already parse, not recovering geometry from a picture. Pixel-based drawing review is a harder, separate problem.

[^laundering]: Risk laundering is the quiet version of the problem: not lying about the engineering state outright, but rewriting around it until it reads as acceptable — a hold described as "progressing," a failed limit dropped from the summary, a caveat simplified away. The status never formally changes; it just stops being visible. That is what makes a polished rewrite a genuine control point rather than cosmetic work.

[^marker]: An execution marker is a small file the workflow step stamps when it actually runs — recording which operation wrote the artifacts and, often, a hash of what it wrote. It exists precisely so that "the side-effect step ran" is something a verifier can check directly, rather than something it has to take the model's word for. The helper is the scripted writer the model invokes to produce that marker and the integrity files in one observable operation.

Making aec-bench Trainable with Prime Lab

Theodoros Galanos — Sat, 16 May 2026 00:00:00 GMT

import GroupedBars from '../../components/charts/GroupedBars.astro'; import LabCouplingSwimlanes from '../../components/charts/LabCouplingSwimlanes.astro'; import LineTrajectory from '../../components/charts/LineTrajectory.astro'; import PrimeRunFlowDiagram from '../../components/charts/PrimeRunFlowDiagram.astro'; import TrainableLoopDiagram from '../../components/charts/TrainableLoopDiagram.astro'; import { adapterComparisonData, trainingTrajectoryData, } from '../../data/prime-rl-charts'; import { labCouplingFlowSteps } from '../../data/lab-coupling-flow'; import { primeRunFlowSteps } from '../../data/prime-run-flow'; import { trainableLoopRows } from '../../data/trainable-loop-flow';

Estimated reading time: 9 minutes

aec-bench can train now.

The first integration with Prime Intellect's (PI) Prime Lab proves a practical point:

with the right infrastructure, real engineering tasks can become RL environments without losing the verifier that made them meaningful in the first place.

That is the step I have been circling in the previous articles. In Where Capability Actually Lives in Agentic Engineering, the claim was that capability in engineering is not a property of the model alone. It lives across the model, tools, verifiers, output contracts, and operating environment. In Benchmarking Agents on Real Engineering Work, that showed up empirically: remove the right harness support and performance did not degrade gracefully. It collapsed. In The Harness Is All You Need, aec-bench became the benchmark layer for making that claim measurable.

This post is the next step in that chain.

If the out-of-distribution gap in engineering is partly an environment gap, then benchmarks are not enough. We also need a way to train models inside those environments.

PI's Lab gives us that path.

TL;DR: Trainable aec-bench Tasks

aec-bench tasks can now be exported as Lab-compatible Verifiers environments.
The task-local verifier remains the source of truth, so training optimizes against the same check used for evaluation.
Lab handles the hosted RL loop: rollout collection, training, checkpoints, adapter deployment, and evals.
aec-bench can import the hosted eval samples back into its ledger, turning the result into inspectable traces instead of a dashboard number.
In the first small adapter check, reward moved from 0.333 to 0.800, final environment responses rose from 2/15 to 8/15, and empty model responses fell from 13/15 to 3/15.

The claim is deliberately narrow: this adapter became better at completing a small range of workflows. That is already a useful thing to be able to train.

From Benchmark Score to RL Environment

Most benchmark work stops at measurement.

That is necessary. It tells us where models fail, which harness choices matter, and how much of the work is still out of distribution for general-purpose systems. But if all we can do is score failures, the loop is incomplete.

The more interesting loop is:

That is what the aec-bench and Lab integration now gives us.

aec-bench supplies the work definition: the prompt, files, tools, expected output, and verifier. Lab supplies the training substrate: environment packaging, hosted execution, RL training, checkpoints, and adapter evaluation.

The clean separation matters. Lab does not need to know what makes a retaining-wall check, short-circuit calculation, or bracket-load task meaningful. It just needs a Verifiers environment with a reward. aec-bench does not need to become an RL platform. It just needs to expose its tasks in the right shape.

This is why the right infrastructure is so important: the benchmark keeps the meaning of the work; the Lab turns that work into a trainable environment.[^pi-lab]

[^pi-lab]: I can't stress enough how blessed we are to have something like PI's Lab in our disposal. It opens up a myriad of opportunities otherwise unattainable outside of big labs. Huge kudos to the team!

It is also the practical version of the argument in Executable Standards: obligations become more useful to agents when they stop living only as prose and start becoming checks, predicates, certificates, and rewards.

Why RL Environments Matter for Engineering OOD Failures

In engineering, the hard failure is often not the model becoming incoherent but really when it becomes plausibly generic.

It uses the wrong table. It applies the wrong assumption. It answers the nearby problem instead of this problem. It performs a calculation but never submits the artifact in the required form. This is what I meant in the earlier benchmark work by out-of-distribution behaviour: the model is not useless, but it is not reliably attached to the task world, or to the specific instance cut from that world.[^task-world]

[^task-world]: The task world is the problem space. The instance is one concrete instantiation of part of that space.

Evaluation tells us when that happens.

RL environments let us put pressure on the behaviour itself.

The question moves from appearance to behaviour:

did the model inspect the assigned files?
did it use the available tools?
did it recover after tool friction?
did it write the required output?
did it actually submit?
did the verifier accept the artifact?

Those are workflow behaviours. They are exactly the behaviours that matter when a domain is not already solved by the base model's priors.

Exporting aec-bench Tasks to Prime Lab

We exported an aec-bench task suite into a Prime-compatible Verifiers environment and ran a short hosted RL experiment on Qwen/Qwen3.5-4B.

The mechanics were refreshingly direct:

The training run was intentionally small. It was not meant to establish a production-quality adapter but more so to understand whether the loop works end to end on real benchmark tasks?

And it did!

During the run, mean reward rose from 0.281 to 0.979, and mean submit_answer calls rose from 0.099 to 0.917. That second number is the more interesting one. It suggests the model was not merely producing better-looking text; it was more often reaching the part of the workflow where the verifier could evaluate the submitted artifact.

The caveat is simple: this was a small training slice, probably helped by task-slice simplicity, and it was a loop proof rather than a benchmark-grade result. For this first pass, the point was to prove that the evaluation harness could become a training harness without changing the task's definition of success.

First Adapter Check: Base vs Adapter

After training, we ran a small medium-difficulty stateful comparison and imported the Prime eval samples back into the aec-bench ledger.

The headline result:

Metric	Base	Adapter
Reward mean	0.333	0.800
Nonzero rollouts	5/15	12/15
Final environment response	2/15	8/15
Empty response errors	13/15	3/15
Mean tool calls	2.67	4.33
Mean `submit_answer` calls	0.13	0.53

The adapter did more of the work. It read more, wrote more, submitted more, and failed with empty responses less often. I think it what happened was workflow learning rather than general engineering competence.

It became more likely for the model to stay inside the environment (task world) long enough to produce something the verifier could judge.

Trace analysis supports that interpretation. Both base and adapter runs were mostly execution-heavy. The adapter did not become wildly more exploratory or magically more intelligent. It became better at execution-through-recovery: use the workspace, repair small tool friction, write the artifact, and submit it.

Modest result, but a useful one.

Why Imported Traces Matter

Hosted training dashboards are useful, but they are not enough for the kind of work aec-bench is trying to do.

The important move is that the Lab's eval samples can easily come back into the aec-bench ledger. Once they are there, they can be inspected with the same reporting tools we use for normal benchmark runs: reward, stop condition, tool calls, error type, submission behaviour, and representative conversations.

That turns the adapter check from an anecdote into an artifact.

It also connects this work to What If the Harness Could Improve Itself?. That article looked at automated improvement of one harness surface: the prompt. This integration makes a stronger version possible. We can now improve the policy itself while keeping the task, verifier, and trace analysis in the same experimental frame.

This is also adjacent to The Third Axis. The third axis was about improving the environment around the model, alongside model size and inference-time reasoning. Prime Lab gives that idea a concrete RL path: once a task environment is explicit and verifiable, the policy can adapt to it.

What the First Adapter Result Does and Does Not Show

The result is narrow, but real. aec-bench can now expose engineering tasks as trainable environments, Lab can run hosted RL against them without replacing the verifier, and the resulting samples can come back into the aec-bench ledger for local inspection. In this first adapter check, that produced measurable movement in workflow completion.

It does not mean the adapter has general engineering competence, or that the task family is solved. The eval was only 15 samples, and one task family stayed stubborn: retaining-wall stability remained 0/3 nonzero for both base and adapter. That is useful information, not an embarrassment. Some behaviours may be learnable from the current environment. Others may need better task design, better tools, stronger verifiers, or simply a larger and more varied training slice.

The Larger Point: Benchmark, Train, Inspect

This is why the integration feels important.

The path from real task to trainable environment is becoming short:

That used to be a research project. With aec-bench and Prime Lab, it starts to look like normal infrastructure.

And that matters because closing the out-of-distribution gap in engineering will not come from one lever. Base models, tools, verifiers, environments, and training pressure all have to feed the same loop. And they all have to be accessible to domain expertise.

We can define real work, measure real failures, train against those failures, and inspect whether the behaviour actually changed.

For aec-bench, that is a real promotion.

It stops being only a test.

It is slowly becoming a lab.

Executable Standards

Theodoros Galanos — Sun, 03 May 2026 00:00:00 GMT

import TransformationLadder from '../../components/charts/TransformationLadder.astro'; import PredicateCorpusExplorer from '../../components/charts/PredicateCorpusExplorer.astro'; import CertificateBinding from '../../components/charts/CertificateBinding.astro'; import DocumentPipeline from '../../components/charts/DocumentPipeline.astro'; import Pullquote from '../../components/Pullquote.astro'; import { transformationLadderData, predicateCorpusData, certificateBindingData, documentsToRequirementsData, autoformPipelinePreviewData, } from '../../data/executable-standards-charts';

Estimated reading time: 11 minutes

The Failures We Keep Seeing

The recurring failure in agentic engineering is not that the model cannot write a convincing answer.

It can.

That is often the problem.

Across the previous Harness posts, the same pattern keeps appearing under different names. The model retrieves the wrong table but writes as if it has found the right one. It performs the calculation but does not notice the unit mismatch. It gives a plausible compliance statement without pinning the standard version. It explains the reasoning path, but the reasoning path is not the artefact we can audit.

In The Third Axis, the visible failure was a tool habit. One executor would inspect the environment before calculating; another would deliberate from memory about tables it did not have. In Where Capability Actually Lives, the larger point was that model capability is only one part of the system. The harness — tools, interfaces, constraints, verifiers, task design — carries a large part of what we call capability. In The Harness Is All You Need, the claim was even sharper: as agentic systems get longer-running, the surrounding structure matters more, not less.

This post is about the next boundary.

At some point, a better tool loop is not enough. A better retrieval system is not enough. A better verifier around the final answer is not enough. The obligation itself has to stop living only as prose.

<Pullquote> The next harness boundary is not another tool call. It is the clause itself. </Pullquote>

From Documents To Requirements

This is also a quality-assurance story.

Traditional QA asks whether the review process has evidence: requirements traced, checks performed, assumptions recorded, comments closed, approvals captured. AI does not remove that need. It makes the evidence problem sharper. If an agent says "the design complies", what did it actually check? Which clause? Which amendment? Which jurisdictional override? Which source span? Which interpretation of an ambiguous "shall"? Which assumptions? Which calculation witness? Which verifier?

Engineering already has a mature instinct for the first move.

No one who has worked with specifications, codes, project briefs, employer's requirements, design criteria, or contract schedules believes that the source document is naturally shaped for computation. The source document is written for humans: committees, clients, reviewers, engineers of record, contractors, auditors. It carries context, authority, convention, exceptions, and the scar tissue of previous projects.

So the first transformation is familiar:

It is requirements engineering. EARS, the Easy Approach to Requirements Syntax, is a way to constrain natural-language requirements so they become less ambiguous, less vague, and easier to test. Dwyer, Avrunin, and Corbett's specification patterns gave software and systems engineers a catalogue of common temporal property shapes.[^patterns] SysML and INCOSE-style requirements practice gave engineering teams a way to type and trace requirements through complex systems.

The exact framework matters less than the shared instinct: free-form prose has to be decomposed before it can be managed.

We have already been building this kind of pipeline for ordinary engineering AI work. In our requirement segmentation workflows, large specification documents are split into atomic, traceable, classified records. Each record preserves source spans, section references, related standards, applicable equipment or areas, verification methods, priorities, confidence scores, and quality signals.

The same shift is visible in geometry: drawings became models, and ISO 19650 and IFC exist precisely because building information needs to be exchanged in a machine-interpretable form. Documents become requirements; drawings become models; tacit coordination becomes structured data. Autoformalisation extends the pattern to obligations.

A requirements register helps. It gives the agent smaller, named obligations instead of one large textual fog. But a register still leaves the most important step unresolved: the agent can still read a structured requirement and interpret it incorrectly at runtime. That is where autoformalisation starts.

From Requirements To Predicates

Autoformalisation is a heavy word for a simple ambition: turn human-authored requirements into machine-checkable obligations.

In the version we are building for AEC, the pipeline looks like this:

The goal is not to prove every engineering judgement in Lean.[^lean] That would be a lovely way to never ship anything.

The pragmatic target is narrower. Take requirements that can be made checkable, lift them into typed predicates over design state, preserve provenance, attach discharge tests and witnesses, and store the result in a versioned corpus that can be queried by a verifier.

Four pieces of vocabulary, in plain terms. A predicate is a small function that takes a design and returns pass or fail along with the numbers it used to decide. Discharge tests are canonical examples that should pass and others that should fail — the sanity check that the predicate actually encodes the clause. A witness is the evidence a predicate emits when it answers, so a reviewer can see what the check looked at. A verifier is the program that runs predicates against a design and emits the certificate.

For a fabricated cable-sizing example, the source might begin as natural-language prose:

The voltage drop for a final subcircuit shall not exceed 5 percent of the
nominal voltage under the stated design load.

That clause becomes a typed predicate over design state:

@requirement("AS3000-3.4.3")
def voltage_drop_within_limit(design: CableDesign) -> PredicateResult:
    percent = 100 * design.voltage_drop_v / design.nominal_voltage_v
    return PredicateResult.bounded(
        percent <= 5.0,
        witness={"actual_percent": percent, "limit_percent": 5.0},
    )

The atomic-requirement record and the corpus entry that wrap this predicate — requirement_id, scope_tags, standard_version, source_hash, predicate_hash, status — are shown in Figure 1.

The source clause is still authoritative. The human-readable requirement still exists. But the thing the verifier consumes is no longer a paragraph. It is an executable predicate with identity, provenance, tests, metadata, and a lifecycle.

This is where the formal-methods literature becomes useful, but not as theatre.

The Guaranteed Safe AI and Safeguarded AI framing describes a triple: world model, specification, verifier. The world model says what can happen — the regime of physical situations the predicates were validated against. The specification says what is acceptable. The verifier produces an auditable certificate that the design satisfies the specification within the validated regime.[^gsa] In our setting, AEC standards and project requirements are the specification source. Autoformalisation is the machinery that turns that source into something a verifier can use.

It is AI4FV before FV4AI: use AI to help perform formal verification of engineering outputs before attempting the harder problem of formally verifying the internals of the AI system itself.[^ai4fv]

The engineer still matters. In fact, the engineer matters more visibly. The human task moves from repeatedly interpreting prose at runtime to auditing the lifted predicate, reviewing its tests, checking its provenance, and deciding when a requirement is too ambiguous to sharpen safely.

<Pullquote> Auditing is cheaper than authoring. It is also a better place to put scarce expert attention. </Pullquote>

Autoformalisation should feel like a sequence of increasingly constrained artefacts, not a mystical leap.

The Corpus Is Infrastructure

Once predicates exist, the hard problem becomes governance.

Where do they live? Who can change them? What does a patch look like? How do we know which standard version a certificate used? What happens when a code amendment changes a clause? How do jurisdictions override each other — say, an AS/NZS clause overridden by a state-level Service & Installation Rule? How do we prevent a verifier from accidentally consuming half-built work?

Our current answer is deliberately boring: a git-tracked filesystem. Predicate source is code. Tests are code. Metadata and provenance are JSON. Indexes are generated and checked in. Review happens through ordinary code review.

Databases are not the issue. The review surface matters — and the filesystem is also the operating surface of AI agents. Agents read, write, diff, and review files. They open pull requests, not rows. A corpus that lives as files is a corpus the harness, the auditor, and the agent can all reason about with the same tools.

If a standard changes, I want a patch that shows: the old predicate and the new predicate; the discharge tests that changed; the source requirement hashes; the provenance record; the generated index diff; the explicit supersession edge.

I do not want "the compliance database was updated".

The corpus design also draws a hard lifecycle boundary. Public queries return released predicates by default. Draft, staged, audited, deprecated, and rejected material remains visible for review and traceability, but it is not consumed by certification unless a caller deliberately asks for development mode.

That default is small but important. In compliance settings, "latest wins" is too magical. A verifier should not infer authority from file modification time, semantic version number, or whichever predicate happened to appear first in a search result. Released obligations need explicit status. Replacements need explicit supersedes and replaces edges. Jurisdictions need caller-supplied precedence, not a hidden ontology pretending to know every legal override.

The corpus is the specification layer of the harness.

Certificates, Not Explanations

The final artefact is not the agent's explanation.

The distinction matters because a design agent may be useful, eloquent, and wrong. It may produce a chain of thought that sounds like engineering reasoning. It may cite a standard, calculate a margin, and conclude compliance. None of that is the trusted artefact.

<Pullquote> The trusted artefact is a certificate accepted by a small checker. </Pullquote>

That makes the certificate a QA object as much as a technical one. It is the thing a reviewer, auditor, or engineer of record can inspect without trusting the agent's narration. The question changes from "do I believe this answer?" to "does this certificate bind the right design, predicates, witnesses, assumptions, and policy?"

For the same fabricated cable run, the design input might be:

{
  "design_id": "cable-run-A17",
  "load_current_a": 63,
  "route_length_m": 42,
  "nominal_voltage_v": 230,
  "voltage_drop_v": 8.7
}

The verifier result binds that design to a released predicate, and the certificate then records the design hash, predicate hashes, standard versions, the assumption ledger (a frozen record of the inputs, exclusions, and conservative choices the design was made under), the world-model reference, the witness bundle, the verifier policy, and the per-layer results. Figure 3 makes the binding explicit.

That is a very different object from "the model says it complies".

It can be replayed and re-checked. A predicate change invalidates it. A bounded layer downgrades it. Open items travel with it. The certificate says: valid under these assumptions, against these predicates, with these witnesses, under this policy.

The verifier portfolio is the piece that keeps this honest. We do not need one giant verifier that does everything. We need complementary layers with explicit failure semantics:

algebraic checks for calculation correctness;
physical invariant checks for impossible design states;
clause compliance against released predicates;
cross-discipline checks where one discipline's conclusion becomes another's premise;
assumption-ledger checks for source and scope discipline;
out-of-distribution checks against the world-model envelope.

Some layers pass. Some produce bounded results. Some are deferred by policy. Some create open items. Failures and verifier errors block the certificate.

The important thing is that the status is explicit.

The harness should not need to trust the model's confidence. It should inspect the certificate.

What We Have Built So Far

Harness engineering is the discipline of deciding what the model should not be trusted to do unaided. First it meant tools, then evaluation, now specification. Calculation went that way. Retrieval went that way. Standards are next.

This is not a production assurance system yet. But it is past sketch.

Inside aec-bench, we have started splitting the work into aec-autoform: a package for the autoformalisation pipeline rather than another benchmark module. That boundary matters. aec-bench is a measurement instrument. aec-autoform is infrastructure that produces predicates and certificates. The benchmark can consume it, but should not own it.

Where each stage stands today:

Stage	Status
Document → atomic requirements	existing data-pipeline work
Atomic requirements → typed predicates	cookbook implemented: triage, decomposition, type binding, predicate authoring, discharge tests, self-check, audit, tier-up
Predicate corpus	scaffold with predicate directories, manifest and provenance contracts, deterministic hashes, generated indexes, released-only queries, validation
Certificates	contracts taking shape: frozen records for design references, corpus references, world-model references, assumption ledgers, policies, layer results, witnesses, signatures, identifiers

The first serious forcing example is cable sizing.

It is concrete enough to keep us honest and general enough to expose the real architecture. Cable sizing has standards, clauses, tables, calculations, assumptions, jurisdictions, amendments, and verifier-friendly outputs. It is not the whole of engineering. Good. The first version should not pretend to be.

The project can fail in useful ways. A requirement may be too ambiguous to lift. A predicate may encode the clause too narrowly. A test may fail. A jurisdiction conflict may require explicit human judgement. A source document may not be licensable for inclusion. A world-model envelope may be too weak. A certificate may be invalid.

Those are not embarrassments. Those are the point. The harness should make the failure explicit before the design gets trusted.

If this path works, the shift that matters is not better AI answers. It is that an engineering obligation becomes something the harness can inspect.

Benchmarks tell us where agents fail. Harnesses give agents tools and structure. Autoformalisation turns the standard itself into part of the harness, and turns quality assurance from a retrospective document review into a replayable check over explicit artefacts.

The standard has to become executable because the alternative is asking the model to keep re-reading the law of the task from scratch.

And we have seen how that goes.

[^patterns]: The point of the Dwyer patterns here is not that every AEC requirement is temporal logic. It is that many requirements have recurring semantic shapes: absence, existence, response, precedence, bounded existence, and so on. Pattern libraries give the lift process a vocabulary before it reaches for a theorem prover.

[^lean]: Lean is a programming language and interactive theorem prover. It matters because some obligations eventually deserve proof-assistant rigour — high-consequence clauses, safety-critical interlocks, regulatory submissions where the cost of being wrong justifies the ceremony. It should not become the default hammer. For v1, typed Python predicates, deterministic tests, and auditable certificates are the shipping path; Lean is a tier-up option.

[^gsa]: In the Guaranteed Safe AI framing, the verifier does not certify the model's inner thoughts. It certifies a claim relative to a world model and specification. That distinction is why the certificate can be smaller and more trustworthy than the agent that produced the candidate design.

[^ai4fv]: AI4FV means "AI for formal verification": using AI to help produce predicates, witnesses, proofs, tests, and counterexamples. FV4AI means "formal verification for AI": proving properties of the AI system itself. The latter is the bigger research mountain. The former is already useful for engineering workflows.

The Third Axis

Theodoros Galanos — Sat, 18 Apr 2026 00:00:00 GMT

import AdvisorHeatmap from '../../components/charts/AdvisorHeatmap.astro'; import BarsWithBaseline from '../../components/charts/BarsWithBaseline.astro'; import HorizontalStackedBars from '../../components/charts/HorizontalStackedBars.astro'; import LineTrajectory from '../../components/charts/LineTrajectory.astro'; import Pullquote from '../../components/Pullquote.astro'; import { advisorGridData, bondDistributionData, experiment1TrajectoryData, experiment2TrajectoryData, frontierDotData, } from '../../data/third-axis-charts';

Estimated reading time: 14 minutes

Three Ways to Make an Agent Better

There are two scaling stories the industry tells.

The first is training-time compute: make the model bigger, feed it more data, let the next generation carry you further. The second is inference-time compute: give the model more room to think — reasoning traces, tool loops, longer context.

Anthropic's own engineering posts have been quietly building out a third. Effective harnesses for long-running agents argues that "model capability alone is insufficient" — capability emerges from harness maturity. Managed agents describes harnesses as disposable infrastructure, OS-like abstractions that outlast the models they host. Harness design for long-running apps puts it plainly:

<Pullquote attribution="Anthropic, Harness design for long-running apps"> Every component in a harness encodes an assumption about what the model can't do on its own. </Pullquote>

As models improve, those assumptions decay, and the harness needs pruning.

The logical next step is to let the model improve its own harness. Not prompt engineering by hand — automated, with real task feedback, a diagnosis agent that reads its own failures and proposes mutations. In Recursive by Design I laid out the architecture the rest of this work sits on top of — Recursive Language Models and Lambda-RLM as building blocks for harnesses that can carry long-running engineering tasks. This post is about what happens when you point the same pattern at the harness itself.

These are a handful of early experiments, on a small collection of tasks. But they still tell us something.

<Pullquote> The third axis is real, but the trajectory is rougher than the tidy story suggests. </Pullquote>

The Loop

A harness improvement loop sounds obvious in the abstract. Run the harness. See what it got wrong. Change something. Run it again. The detail hiding inside "change something" is where every wrong turn lives.

Three questions have to be answered before the loop produces anything other than noise.

How does the agent know what the run actually did?

A failing reward score tells you the answer was wrong. It doesn't tell you why. Was the model missing a fact? Did it call the wrong tool? Did it spin on the problem without committing? Did it charge forward without checking its own work?

Every assistant turn in a trajectory is labelled with one of four behaviours — what I call bonds, following the agent-behaviour taxonomies literature. Execution: the obvious forward motion — tool calls, code runs, formatting an answer. Verification: looking backward at your own work, comparing output to expectations, catching errors. Deliberation: committed multi-step reasoning along a single path. eXploration: branching, considering alternatives, hedging, forming hypotheses. The classifier is itself an LLM, prompted with definitions, indicators, and tie-break rules — verification wins over exploration wins over deliberation wins over execution, because the quieter behaviours are the easier ones to under-count.

From the labelled sequence I build a transition matrix: the probability of going from Exploration to Deliberation, from Execution to Verification, and so on. Aggregate the matrices across many high-reward trials and you get an ideal pattern — the shape of thinking that tends to succeed. Compare a new run's matrix to the ideal and you get a structural score: not "did this answer land" but "did this agent think in a shape that usually lands."

Bonds describe the shape of the reasoning. Separately, a field-score enrichment carries the per-output diagnosis straight from the verifier — not "wrong" but "vc was within 3%, Vd was 18% off, compliance flipped because Vd crossed the threshold." The evolver reads both. Bonds tell it how the agent thought. Field scores tell it where the answer fell short.

How does it decide what to change?

The first version of the evolver did everything in one LLM call: "here's the trajectory, propose a mutation." It generated confident nonsense. Beautiful prose, arbitrary edits, no grounding.

I had to split it in two. Phase one is an investigator — an agent with tools that can query the trace, read the current skills, inspect the graveyard of past mutations, and write a report. [^aggagent] It's not asked to propose anything; it's asked to look. Phase two is a proposer — a constrained call that takes the investigation report plus the field scores and returns a structured mutation object. Add a skill, edit the system prompt, change a worked example. One thing at a time. Separating the two kept the proposer grounded: it had to cite the investigation, not vibe-propose.

The graveyard feeds the proposer's prompt. Every mutation that regressed or failed is kept, with its reason. You don't relearn the same lessons across cycles.

[^aggagent]: A couple of weeks after I'd built the investigator, a paper on the same pattern landed. Independent convergence, not influence.

How do you know it's not overfitting?

This is the question that makes harness evolution different from gradient descent. You can't just crank down the loss.

Three guardrails. Masked feedback: the evolver sees "vc was 18% too high" but never the correct value. This keeps it from memorising test answers into the system prompt. Graduated scope: early mutations are quiet (small prompt tweaks); later ones can be structural (new skills, tool changes). You earn the right to larger edits. Structural scoring: rewards are computed per-field against an engine, not LLM-judged. The verifier is deterministic code, not a model you could flatter. [^catch]

Is it bulletproof? No. A sufficiently clever evolver can still triangulate the masked signal. I haven't caught it doing so. I'm watching.

[^catch]: Of course, this is only possible in verifiable domains where explicit verification happens numerically. Rubric design, and in some cases evolution, replaces this approach in more qualitative tasks.

The whole loop, in one file. One of the pleasant surprises of building this was how terse the runner ended up. Once the bonds classifier, the two-phase evolver, the graveyard, and the guardrails are their own modules, starting a full harness-evolution experiment compresses to a config plus a three-line workspace scaffold. The launcher for Experiment 2 was this: [^aec-bench]

# evolution.yaml
workspace_path: ./my-harness
models:
  classifier: env:AWS_HAIKU_MODEL_ID
  evolver:    env:AWS_SONNET_MODEL_ID
solver:
  name: first-evolution
  adapter: tool_loop
  model: env:AWS_HAIKU_MODEL_ID
  client: {kind: bedrock}
generate:
  template: voltage-drop
  count: 10
  seed: 42
  difficulties: [easy, medium]
tasks:
  domains: [electrical]
backend: local
batch_size: 5
max_cycles: 10

# One-time workspace scaffold
mkdir -p my-harness/prompts my-harness/skills
echo "You are an expert electrical engineer." > my-harness/prompts/system.md
printf 'agent_adapter: tool_loop\nevolvable_layers: [prompts, skills]\nname: first-evolution\n' > my-harness/manifest.yaml

# Run the evolution loop
aec-bench evolve run -c evolution.yaml

[^aec-bench]: aec-bench is an open-source library for benchmarking agents on engineering tasks. Link when it ships.

Thirty minutes later you have trials/cycle_*.jsonl with the same bond-sequence data analysed below. aec-bench isn't open yet — but the configuration surface is fixed, so this is a fair preview of the ergonomics when it ships.

Experiment 1 — Proof of Life

The question was narrow on purpose: can the loop, starting from nothing, improve a harness that I already know how to write by hand?

Setup. A voltage-drop task with ten instances — the kind of AS/NZS 3008 cable-sizing problem an electrical engineer would solve in their sleep. Haiku 4.5 as the agent. The starting system prompt was one line: "You are an expert electrical engineer." No skills. No worked examples. Just the task instruction and a calculator tool.

Baseline: 0.85.

That surprised me. A 1-line prompt shouldn't solve 85% of a benchmark — but the task instruction carried most of the signal. It spells out the procedure (look up voltage-drop coefficient, adjust for single-phase, multiply by power factor, compute drop) and ships with a Python calc tool wrapping the exact engine the verifier uses. Haiku read the instruction, used the tool, and got four of five tasks right. The fifth — a 35mm² three-phase copper run at power factor 0.81 — was failing on the numerical fields by a small margin.

So: evolution had to close a 15-point gap, on one stubborn task, without the benefit of seeing correct answers.

Cycles 1 through 5: plateau. The evolver kept proposing the obvious thing — inject a voltage-drop coefficient skill. It did that. The skill grew over five cycles into a serious piece of reference material: AS/NZS Table 30 lookup values for both 75°C PVC and 90°C XLPE insulation, cable sizes from 1.5mm² to 300mm², worked examples for single-phase and three-phase. The score didn't move. Task 3 kept failing.

Cycle 6: 1.00.

The breakthrough mutation did something different. It removed content. The 90°C XLPE table got dropped — the tasks weren't using it. Cable sizes above 150mm² got trimmed. The worked examples shrank. And one line was added: "Single-phase: Vc accounts for both conductors."

Task 3 passed. All five tasks passed.

Cycle 7: 0.70.

The evolver tried one more mutation on top of the peak. It broke two previously-working tasks. Cycles 8–10 returned to the 0.85 plateau. The run never re-hit 1.00.

Hill-climb search does not owe you monotonicity. Every search that can find a peak can walk off one. What saves you is that the peak is tagged in git — evo-20260412-0317-6 in this case — and you can roll back. The loop's job is to find good harnesses; preserving them is a separate concern, and one aec-bench got right mostly because I borrowed the convention from deployment, not from ML.

The unexpected lesson is the breakthrough itself. Less + clearer > more + exhaustive. The five-cycle plateau was spent building bigger reference material. The peak was reached by cutting half of it and adding one clarifying sentence. If a human harness engineer had proposed the cycle-6 mutation over coffee, it would have looked obvious. Evolution had to earn it.

That's a +15 point autonomous gain, a 30-point swing within a two-cycle window, and a +15 net if you preserve the peak.

Experiment 2 — The Habit Ceiling

If prompts-and-skills evolution can close a 15-point gap on Haiku, the next question is whether it can unstick a weaker executor given a stronger lever: a sub-inference tool the agent can call when it hits a wall. Anthropic's playbook for their advisor tool is clean — call at the start, call before declaring done, call when stuck. Does evolution discover that pattern on its own? And does advisor access let GPT-4.1-mini close the gap to Haiku?

Setup. Six arms, crossing executor (Haiku 4.5 via Bedrock / GPT-4.1-mini via Azure) with prompt strength (a one-line system prompt, a 400-character "call when stuck" hint, and Anthropic's verbatim 2,300-character recommended advisor-timing block). Three seeds per arm on the main four; two N=1 anchors on the stock-prompt arms. Same ten voltage-drop instances per seed. Advisor backed by Sonnet 4.6, max five calls per trial. Evolution runs for up to ten cycles, stopping early on stagnation.

Baselines. Before any evolution, on cycle 1:

Executor	Prompt	Cycle-1 mean
Haiku	hint	0.88
Haiku	Anthropic verbatim	0.93
GPT-4.1-mini	hint	0.47
GPT-4.1-mini	Anthropic verbatim	0.70

Haiku is already roughly where evolution took it in Experiment 1. GPT-4.1-mini is the genuine stuck executor — the arm where a strategic-reasoning escalation tool should matter most. If evolution-plus-advisor works for anyone, it should work here.

Ten cycles later, the by-arm trajectory is the cleanest summary of the run:

The bond distributions tell the first story. Aggregated across every trial in the experiment, the four 4.1-mini arms and the two Haiku arms form two disjoint clusters.

GPT-4.1-mini does approximately zero exploration. It does not check the environment, run --help, or inspect the tool before committing to a calculation. It spends roughly a fifth of its turns deliberating — reasoning internally about tables it doesn't have, "recalling" Vc = 14.6 mV/A/m from AS/NZS 3008 and running the calc on that, even when the Python tool wrapping the actual engine is sitting in the workspace. The D bonds correlate directly with hallucinated-table failures in the trace.

Haiku does the opposite. X runs consistently at 12–13% — the first turn is almost always an orientation step, typically --help. D runs under 1.5% — the model almost never reasons about values it hasn't looked up. Reward follows the bond shape: Haiku between 0.85 and 1.00, 4.1-mini between 0.40 and 0.60.

These aren't intermediate findings that evolution closes over ten cycles. These are the per-arm distributions after ten cycles of evolution trying to nudge them. The evolver correctly diagnosed the 4.1-mini runs as no_exploration (X = 0% triggers the anti-pattern), rewrote the system prompt with read-tables-first instructions, added a cable-sizing skill with worked examples, and in arm 8 started the run with Anthropic's 2,300-character call-when-stuck block already installed. The distributions barely moved.

The advisor story is sharper. Five calls, across 585 trials. All on Haiku.

Every Haiku seed that called the advisor ended below the arm mean. The seeds that cruised at 1.00 never pulled it. The seeds that landed at 0.85 pulled it once or twice. That is Anthropic's own "call when stuck" clause firing exactly on its preconditions — used correctly, used sparingly, used by the model that barely needed it.

GPT-4.1-mini called the advisor zero times across nine seeds and three prompt variants. Three hundred trials. Zero escalations. One trial's log from arm 8 puts the refusal starkly. The assistant's internal calc and the tool output disagree by roughly three-fold on the voltage-drop coefficient:

Assistant: "From AS/NZS 3008 tables, Vc = 14.6 mV/A/m for 6mm² Cu
             → Vd = 4.44 V → 1.93%, compliant"
Tool:      vc_mv_per_a_m: 5.77, voltage_drop_v: 1.97, voltage_drop_percent: 0.86

Anthropic's verbatim prompt — the system prompt for this exact trial — says: "if you've already retrieved data pointing one way and the advisor points another, don't silently switch. Surface the conflict in one more advisor call." The assistant didn't. It silently reconciled, sometimes taking the tool value, sometimes keeping its own.

One negative control makes this feel structural rather than stochastic. Arm 6 and arm 7 are the same Haiku executor, on the same task, under the same evolver. The only variable is advisor-guidance prompt length — 400 characters vs Anthropic's 2,300. The final reward: 0.93 ± 0.06 vs 0.93 ± 0.06. Neither better nor worse. The strongest available prompt is not a lever on this task for this model.

Evolution did try. In arm 4 cycle 8 the evolver proposed rewriting the prompt and adding a read-before-act skill plus a cable-sizing skill — the score jumped from 0.20 to 0.60, then regressed over the following cycles. In arm 5 seed 42 cycle 2, a proposed mutation went the other direction: batch reward 0.00. An independent instance of the peak-walkoff I saw at cycle 7 of Experiment 1, on a different executor, a different seed, a different mutation target. The search behaves the same.

One small positive signal at the other end of the distribution. On cycle 10 of arm 6, advisor calls jumped to 0.20 per trial — the evolver had finally surfaced the advisor tool through a prompt mutation, on a Haiku run that was already near-perfect. Evolution can teach advisor-calling. Just not reliably, and not to the executor that needed it.

The frame I'd pre-registered — does evolution rediscover Anthropic's early-call-and-final-call pattern — turned out to be the wrong question. The cleaner finding, which the full matrix makes unavoidable:

Prompt evolution can tune what the agent says. It can't rewire what the agent habitually does. 4.1-mini has a deliberation habit — about 21% of its turns — that produces hallucinated table lookups. Haiku has an exploration habit — about 13% — that prevents them. Both habits survive ten cycles of evolver mutation. Both survive Anthropic's verbatim directive. The advisor tool, which was the exact antidote to 4.1-mini's problem, was reached for by the model that didn't need it and refused by the model that did.

<Pullquote> You don't fix a habit with a hint. </Pullquote>

The Economic Case, In One Paragraph

A third experiment was planned — a full cost/quality frontier across Sonnet, evolved Haiku, and evolved Haiku + advisor. Experiment 2 made the curve unnecessary. Stock Haiku on the voltage-drop task lands at 0.88 before any evolution. Evolved Haiku lands at 0.93 ± 0.06, with some seeds spiking to 1.00. Stock GPT-4.1-mini runs around 0.60. Evolved GPT-4.1-mini, with the strongest available prompt-and-advisor scaffold, lands at 0.53 ± 0.09. On this task, model choice swamps the gains from prompt evolution — stock Haiku beats evolved GPT-4.1-mini by roughly 35 points on the mean. On the frontier that matters — cost per correct answer, not cost per token — the cheaper-per-token executor stops being the cheaper option. The frontier is a dot, not a curve.

I'll return to the full cost comparison when I have a task set where the match is actually close.

What I Learned

Seven things, ranked by how much they changed how I build.

Evolution quality-audits the benchmark. The first three runs stagnated at 0.80 on a different task set — same score, every cycle, different configurations. I assumed the evolver was hitting a reasoning ceiling. It wasn't. Evolution was achieving 100% on every solvable task and correctly failing on a single unsolvable one — a task whose "hard" difficulty hid a parameter (conductor material) that couldn't be inferred from the remaining context. A coin flip dressed up as engineering inference. The 0.80 wall wasn't evolution's limit; it was the benchmark's flaw. I audited all 82 templates. Twelve had the same issue. All twelve are now fixed. An evolver that can't cheat is a more honest auditor than any human review would have been.

Hill-climb is not monotonic. Cycle 7 broke what cycle 6 had won. I knew this in principle — every stochastic search has this property — but knowing it and watching your 1.00 collapse to 0.70 are different experiences. Experiment 2 gave me a second, independent instance: arm 5 seed 42 cycle 2 regressed to 0.00 on a batch that cycle 1 had handled at 0.40. Different executor, different seed, same shape. The right instinct isn't "add safeguards to prevent regression"; it's "preserve peaks cheaply, let the search breathe." Git tags cost nothing. Use them.

There is a habit ceiling, and prompt evolution sits under it. Haiku has a 13% exploration habit; GPT-4.1-mini has a 21% deliberation habit. Both survive ten cycles of mutation and Anthropic's verbatim 2,300-character advisor-timing block. Evolution of prompts and skills can lift a model that's close to the answer; it can't lift one that systematically misreads the situation. The lever that changes habits lives in the model weights (training, fine-tune) or in the architectural scaffolding (structural harnesses that force tool orderings before the model has a chance to deliberate). Prompts ride on top of those, not against them.

A 95%-right skill can override correct instinct. One of the runs had a skill with worked examples that omitted the power-factor multiplication step. Haiku, left to its own devices, applied power factor correctly from training knowledge. Given the skill, it followed the skill — worked examples and all — and got the answer wrong on 20% of tasks. A skill doesn't just add knowledge; it displaces the model's own. Small omissions in the skill become systematic bugs in the agent. Harness is all you need — and all it takes to lead the model astray.

Infrastructure constraints compound. The turn limit (15) and the skill compaction budget (2000 chars) were both defensible defaults. Together they produced a failure mode — skills truncated mid-example, agents running out of turns on the truncated skills — that looked like a reasoning problem. It wasn't. It was arithmetic. Two reference tables at AS/NZS precision are ~2000 characters of pure data before any prose. The budget must respect the irreducible size of the domain.

Trust the model's judgement on content. My first compaction strategy had a hard truncation fallback: if the LLM's compacted skill exceeded budget, chop the excess. That fallback produced more bugs than the thing it was guarding against. The replacement trusts the model: if it says the skill needs to be this large, the skill is that large. No ceiling. Zero regressions.

The "third axis" is real, but rougher than the tidy story. Evolution did improve the harness. It did so autonomously, with real signal, against a task set where the gap wasn't closable by prompting alone. It also regressed, plateaued, and mis-diagnosed. None of those are failure modes of the idea; they're failure modes of any search. The question isn't whether the axis works. It's how much of your engineering effort moves into designing the search — the bonds, the guardrails, the graveyard — versus the harness itself.

One Strategy Isn't Enough

Hill-climb finds the best single strategy for a task. That's the shape of the bet in this post: one harness gets better.

But a benchmark isn't one task, and a benchmark isn't one executor. An electrical voltage-drop problem and a structural seismic check reward different behaviours, different skills, different scaffolds. And — the Experiment 2 lesson — a harness that cannot unstick GPT-4.1-mini is not a failed harness in general; it may be exactly the scaffolding that unlocks a different class of executor altogether. Habits don't bend to prompts, but different habits find home in different harnesses. Hill-climb, by construction, forgets the alternatives it passed on the way up.

The next post is about what happens when you stop climbing and start archiving — keeping diverse harnesses indexed by behaviour, so the right one can be recalled when the task, and the executor, call for it. Quality over one axis becomes quality across many.

<Pullquote> One strategy isn't enough. </Pullquote>

Recursive by Design

Theodoros Galanos — Sat, 04 Apr 2026 00:00:00 GMT

import ProcessComparison from '../../components/charts/ProcessComparison.astro'; import Pullquote from '../../components/Pullquote.astro'; import Sidenote from '../../components/Sidenote.astro'; import { rlmArchitectureShiftData } from '../../data/recursive-by-design-charts';

Estimated reading time: 13 minutes

In The Harness Is All You Need, I argued that domain-specific tooling around the model matters more than the model itself. AEC-Bench showed this empirically: the biggest performance gains came from better retrieval and document parsing, not model upgrades. Earlier, in Benchmarking Agents on Real Engineering Work, we'd seen the same pattern from the other direction — when harness support was stripped away, performance didn't degrade gracefully. It collapsed.

These arguments seem to draw a line. Model on one side. Harness on the other. My goal in this article is to discuss with you how that is..more complex than it sounds.

<Pullquote> The agent's reasoning architecture is itself a harness design choice. Agent design and harness design are the same problem. </Pullquote>

When Zhang and Khattab published the RLM paper<Sidenote>Now accepted at ICML.</Sidenote> I had the same reaction most people did: "So... this is Claude Code?", I asked. A model with a persistent REPL. We've had that. The performance gains were there, and reported independently. For example, Prime Intellect explored it further, and showed how a smaller model with RLM scaffolding outperforms a larger model without it — but I was still skeptical. What was new here?

The answer to the question was the same it always is: I had to build one to fully understand it. And as it turns out, the difference is architectural, and it's easy to miss.

Claude Code and Codex are powerful systems. But they still put the full context: the conversation, the documents, the task, all of it into the model's context window. The model attends over all of it, and as the window fills, capability degrades. This is "context rot," and every long-running agent session hits it eventually.

RLMs do something fundamentally different. The context lives outside the model, as a variable in a persistent Python environment. The model never sees the full input. Instead, it writes code to programmatically interact with it — peeking at slices, filtering with regex, chunking and dispatching sub-LLM calls over pieces, accumulating results in variables. The model's context window stays compact. The work happens in the symbolic environment.

Three conditions separate an RLM from "a model with a REPL":

Symbolic input — the context is a variable in the REPL, not tokens in the window
Persistent execution — REPL state persists across turns, variables are memory
Recursive LLM invocation — code can invoke LLMs inside loops, not as discrete tool calls

That third condition is the key. It means the model can do O(n) or O(n²) semantic work over an input of length n, something that's impossible when the entire input has to fit in a fixed-size attention window. The decomposition happens in code, not in the model's reasoning. And that decomposition strategy — how the model breaks context apart, what it delegates to sub-calls, how it reassembles results — is a design choice. A harness choice.

This is where the line dissolves. The agent's recursive strategy is in fact the harness.

Report Generation: Where Recursion Is the Structure of the Work

Report generation is one of the most common tasks in AEC, or any industry really, where agentic AI can deliver real value — if the harness is right. The agent receives source documents — project briefs, technical specifications, reference drawings, supplementary material — and must produce a structured deliverable: a fee proposal, a scope of works, a design report. These tasks share a common shape:

A template with a dependency tree — some sections depend on other sections being written first
Multi-document source material — often too large for a single context window
Domain-specific writing rules per section — construction idioms, discipline conventions, contractual phrasing
Structured extraction → composition — extract facts from sources, then compose prose from those facts

The intuition that led me to RLMs was simple: that output template is a symbolic object with structure — a dependency tree. And the RLM paradigm is built for recursion over symbolic representations. Generating a report is recursive template filling. Decompose the document into sections. Decompose each section into source extractions. Extract from bounded chunks. Compose results back up the dependency tree. The recursion is the structure of the work.

Stage 1: Giving the Model a REPL — Power Without Predictability

The first version gave the model a persistent Python REPL with tools for the job: extract() to pull structured data from documents, summarise() to compress text, llm_query() to compose sections from extracted facts, a NOTE()/RECALL() scratchpad for persistent memory, and a ReportTemplate object with dependency-aware section filling. The model generated code in ```repl blocks, executed it, stored results in variables, and composed sections by passing extracted data to sub-LLM calls.

It worked! And it was also wildly unpredictable.

Over a dozen experimental runs on the same task with the same model, input token usage ranged from 732K to 1.54M — a 2.1x spread for identical inputs. Some runs completed in 48 turns. Others hit a 150-turn limit and produced nothing useful. The model spent substantial capacity generating control code — Python loops, error handling, variable management — instead of reasoning about the actual engineering content. What the λ-RLM paper later formalised as the "coding tax."

The most surprising run was the first one. The REPL commands were broken due to an injection bug — none of the scaffolding loaded. The model, with nothing but a raw REPL and the source documents, produced an excellent 30KB output in 7 turns and 10K tokens. Fully scaffolded runs with 50+ turns produced worse output at 100x the cost.

That accidental result brought back visions of the bitter lesson. If the model performs better when you get out of its way, why are we building scaffolding at all?

The answer, in this case at least, is that a single good run isn't a strategy. The model got lucky. On the next run it might burn a million tokens looping over documents it's already read. Without structure, you get high variance. Some runs are brilliant. Some are catastrophic. And while that can be a fun process, it isn't something you can deploy.

Stage 2: Context Compaction — Treating the Symptom

The second version added intelligent history compaction. When the conversation approached 85% of the context window, a separate model summarised the trajectory — what was read, what was extracted, what sections were filled — and the conversation restarted from that summary. The scratchpad persisted on disk, so critical facts survived the reset. REPL variables persisted in memory.

With prompt caching on top, the token economics went down to $0.56 per 9-section report (from $3.50 without caching), at a 96% cache hit rate. Fifty turns, every section filled, zero wasted turns.

But the underlying problem remained. The agent still decided its own decomposition strategy on every run. Different runs still took different paths through the source material. Token usage was lower but still variable. We'd managed the symptom (context window pressure) without addressing the cause: unbounded decomposition.

What if we computed the decomposition strategy upfront? What if we put domain expertise at use?

Stage 3: Lambda-RLM — When the Task Structure Is the Plan

The key architectural decision was to stop letting the model decide how to decompose the work.

Think about what the open REPL actually asks the model to do. It gets a pile of source documents and a template, and has to figure out: which documents feed which sections, how to chunk things that don't fit in context, when to extract vs summarise, how to handle dependencies between sections, and when it's done. That's a planning problem. And the model was solving it differently every run — which is why we got 2.1x token variance on identical inputs.

But the structure of the task already contains the plan. The template is a dependency tree. The sources have measurable sizes. The composition operators follow from the section types. Why ask the model to rediscover this every run when you can compute it from the task definition?

The λ-RLM paper formalised this insight. Replace the open-ended loop with a deterministic pipeline. Confine the model to bounded leaf operations — the part it's actually good at: understanding and synthesising domain content. Handle the decomposition with pure Python.

λ-RLM has four phases.

Plan reads the template dependency tree, measures source documents, and computes the optimal decomposition — branching factor, leaf chunk sizes, composition operators — with zero LLM calls. Total cost is estimated before the first API call is made.
Extract walks the plan in dependency order, pulling structured key-value pairs from bounded chunks. The model chooses what to extract (field names are model-inferred) but the decomposition is fixed.
Review runs a contract alignment check per section: does the extracted data cover the requirements? Are the values specific enough? Sections that fail get a targeted re-extraction pass.
Generate composes each section from extracted data plus dependency context, fills the template, and assembles the final document.

This is the answer to Stage 1's scaffolding paradox. The "broken run" wasn't evidence that models don't need structure, only that having the wrong kind of structure — teaching the model a workflow via scaffolding commands — is worse than no structure at all. The right kind of structure matches the shape of the work. The template is the plan. The recursion is designed in, not left to emerge.

The configuration is compact:

[template]
tier = "dependency_tree"
definition = "report_template.toml"

[planner]
context_window_chars = 100_000
accuracy_target = 0.80
max_branching_factor = 20

[review]
enabled = true
max_retries_per_source = 1

And swapping between the open REPL and the deterministic pipeline is a single flag in aec-bench:

# Open-ended REPL — the model decides how to decompose
aec-bench run-local tasks/my-task --adapter rlm --model claude-sonnet-4-6

# Deterministic pipeline — decomposition computed upfront
aec-bench run-local tasks/my-task --adapter lambda-rlm --model claude-sonnet-4-6

The results on the same report generation task:

Metric	Open REPL (best run)	Lambda-RLM	Factor
Input tokens	732K	33K	22x less
Total tokens	740K	53K	14x less
API calls	48	27	1.8x fewer
Calls vs estimate	unknown	27/27	exact match
Quality (reward)	0.67	0.73	+8.4%

The plan's call estimate matched the actual execution exactly — deterministic cost, no surprises. And the review phase caught genuine issues: 6 of 8 sections were flagged for extraction gaps. Extractions too vague where the writing guidance demanded specificity, provenance gaps where data couldn't be traced to source documents. Even when re-extraction couldn't fix the gap (you can't extract data from a document that doesn't exist in the workspace), the review output told us exactly what was missing and why.

The Expert Feedback Loop

While these numbers tell us something about performance and whether the output improved, they don't tell us whether a domain professional would sign off on it. Like any other AI workflow, building a task specific harness is something you do alongside domain experts.

So, after the initial lambda-RLM runs I put the output in front of experts and asked them to review and compare it against their own human-authored equivalent. Section by section. With a focus on "what's wrong, what's missing, what would you change?"

The feedback was specific, and each piece became a concrete change to the harness:

The agent used "fee proposal" language in a scope of works document — a terminology confusion traced back to an earlier task's prompts, fixed at the template level. Client-specific construction idioms were absent. These are conventions that live in the heads of discipline leads who've written dozens of these documents. No model would learn them from training data because most of this data were never in the training to begin with. So instead, we added them to the writing guidance rules.

Both agent variants skipped a project objectives subsection. It was in the source brief. The template just didn't require it — added as a mandatory subsection, along with a contractual definitions section (a standard task feature neither agent generated).

One reviewer caught a specification contradiction between two sections — the type of cabling specified in one place, was explicitly excluded in another. The human team had caught the same issue in their own review process. That finding inspired a cross-section consistency check: "does section X.XX (Security) contradict section X.XX (Electrical)?"

Discipline leads noted that their workflow involved tagging sections by discipline lead, with each lead referencing the specific discipline specification. So we added discipline-specific extraction passes, routing each section's prompts through the relevant discipline spec summary rather than the whole document.

The result: v2 output with 20+ properly placed, task-specific idioms (v1 had zero), project objectives, definitions, fewer terminology errors. Measurable improvement on the rubric — and output that a discipline lead would recognise as structurally correct, right in the ways that matter for the domain.

This loop — build, evaluate against expert output, collect feedback, feed improvements back into the harness — is how the gap closes. And it depends on having the infrastructure to do it systematically. A rubric that scores specific dimensions. Trajectory data that shows what the agent extracted and how it composed each section. A comparison framework that puts agent output next to human output at the section level. All of which are embedded in aec-bench.

Practitioner's Notes: Designing for Agent Consumers

Building the RLM adapters surfaced a set of lessons that apply beyond this specific paradigm. They all share a common thread: when your consumer is a model, the design rules change.

Agents guess method names. The model called get_section_context() instead of get_context(), tried .success on plain dicts, passed wrong argument counts. Each fumble wasted 2-3 turns. By adding __getattr__ with suggestion mappings to the report template — twenty redirects like "did you mean CONTEXT(section_id)?" — the fumbles stopped. If you're building tools for agents, the API surface needs to be guessable, not just documented.

HELP() is your API contract. The original HELP function was 9 flat lines with no return types and no grouping. The agent couldn't discover the API. We restructured it into sections (Memory, Sub-Calls, Report Template) with return type annotations on every function. For a human developer, you write docs. For an agent, the docs need to be inside the environment, callable and complete. And progressive disclosure is essential.

The scratchpad is the real memory. Conversation history gets compacted. Variables persist in memory but only until the process ends. The scratchpad — NOTE(key, value) writing to a JSON file, RECALL(key) reading it back — survives everything. Agents that NOTE after every extraction can survive unlimited compaction cycles without losing their place. A simple change to the system prompt to say "your scratchpad IS your memory" and the agent's behaviour adapted immediately. This is the RLM insight in microcosm: durable state lives outside the model's context window, and the model accesses it symbolically.

Make sub-calls visible to each other. In multi-section reports, section 5's extraction often needs facts that section 3 already pulled. Without shared history, the agent either re-extracts (wasting tokens) or tries to remember from context (unreliable after compaction). We added a SUBCALL_LOG — an ordered record of every sub-call invocation, queryable by type. Later calls can see what earlier ones returned: SUBCALL_LOG.by_type("extract"). The scratchpad principle applied to tool results.

Parallelise at the right layer. We built two parallel primitives: parallel() for general concurrent execution, and fill_parallel() that understands the template's dependency graph. The general one runs any list of lambdas via ThreadPoolExecutor. The template-aware one identifies which sections are unlocked, generates them concurrently, then fills sequentially. Generation is embarrassingly parallel (independent LLM calls), but filling is sequential (template state mutation). Separate the I/O-bound work from the state mutation and you get concurrency without locks on your domain objects.

Your test suite will lie to you. All our unit tests used simple source labels: "brief:Description" mapping to a document key "brief". The real task template used "brief:Description/Background". The tests passed. The real run failed silently — the extraction prompt received an empty string instead of the document content. We only caught it by running against a real task with real documents. Synthetic test data hides the gaps that matter most.

Benchmarks as the Improvement Engine

Each stage was driven by measured evaluation:

Stage 1 → Stage 2: Token variance and latency drove the compaction work. We could measure the 2.1x spread. We could see runs hitting 1.5M tokens for identical inputs.
Stage 2 → Stage 3: The insight that unbounded decomposition was the root cause, not context growth. The compacted runs were cheaper but still variable.
Stage 3 → v2: Expert comparison and rubric scoring drove specific harness improvements. Every missing idiom, every skipped subsection, every terminology error mapped to a template or prompt change.

The point of this is obvious: anecdotes don't really compound. But benchmarks do. The RLM paradigm makes this especially concrete because every phase is inspectable — you can see what the plan was, what was extracted, what the review flagged, what was generated. The trajectory is the explanation.

This is what aec-bench is built to support: evaluation-driven improvement as a workflow, not a philosophy. A handful of commands close the loop from "I have a task idea" to "I know which harness works best":

# Generate 50 voltage-drop variants from a single template
aec-bench generate task templates/electrical/voltage-drop --count 50

# Bundle them into a versioned, immutable dataset
aec-bench dataset create --name voltage-drop-v2 --source tasks/electrical/voltage-drop/

# Run the dataset across two reasoning architectures
aec-bench run --config experiment.yaml   # rlm vs lambda-rlm, claude code vs codex, etc., same tasks, same models

# Compare: which harness scored higher, and where did each fail?
aec-bench evaluate compare run-A run-B

The experiment config is where you express the hypothesis:

tasks:
  dataset: "voltage-drop-v2@1.0.0"
agents:
  - adapter: rlm
    model: claude-sonnet-4-6
  - adapter: lambda-rlm
    model: claude-sonnet-4-6

Same tasks. Same model. Different reasoning architecture. The rubric scores on dimensions that matter to domain experts — not just "did you get the number right" but "did you cite the correct standard, use the right cable table, show your working." And the trajectory viewer shows how each agent worked: what it planned, what it extracted, what the review phase flagged, what it generated.

A lot more to share on aec-bench soon!

Agent Design Is Harness Design

We started this whole experiment with a model and a REPL. We ended with a deterministic pipeline that costs 14x less, runs at half the time, produces higher-quality output, and tells you what it will cost before it starts. And all we had to do is change the harness.

But "harness" undersells it. What changed in this case was the reasoning architecture — how context is managed, how work is decomposed, where the model is and isn't asked to make decisions. That's agent design. It's also harness design. The two are the same discipline, and the line between them was always artificial.

The previous post said: stop collecting demos, start building benchmarks. This post is the worked example. Benchmarks show us variance. Expert reviews show us the domain gaps. Iteration narrows both of them. None of that required a better model. All of it required better engineering around the model.

If you're building AI systems for engineering or construction, that loop — measure, review, iterate — is the work. The model is the easy part.

The Harness Is All You Need

Theodoros Galanos — Fri, 03 Apr 2026 00:00:00 GMT

Estimated reading time: 5 minutes

There is a genre of LinkedIn post that goes something like this: "I gave GPT my floor plan and it produced a schedule in 30 seconds. The industry will never be the same."

It's a compelling demo but not much more than anecdotal evidence.

The gap between a model producing a plausible-looking answer on a single prompt and actually performing reliably across a wide range of real engineering tasks is enormous. But we don't have the means to measure it.

I have written about this before. In Where Capability Actually Lives, the argument was that in domain-specific work, performance is not a property of the model alone — it is distributed across the model, the tools, the verification, the orchestration, and the output contracts around it. In Benchmarking Agents on Real Engineering Work, early empirical results supported that claim: when harness support was removed, capability didn't degrade gracefully. It collapsed. And in What If the Harness Could Improve Itself?, we showed that automated improvement of the harness environment compounds in ways that model upgrades alone don't.

Those were arguments and early experiments. What we didn't have was a proper benchmark — one broad enough, multimodal enough, and rigorous enough to make the case at scale.

Now we do. Or at least a solid start at it.

AEC-Bench

Together with Nomic AI, we have released AEC-Bench: the first multimodal benchmark for evaluating AI agents on real-world architecture, engineering, and construction tasks. It is open source under Apache 2.0, and it covers 196 task instances across three complexity levels, from single-sheet understanding to cross-document coordination across drawings, specifications, and RFIs.

Real documents, real decisions. The kind AEC professionals deal with every day: reading construction drawings, cross-referencing detail callouts, navigating sheet indices, reconciling specs with RFIs. Work where getting the gist doesn't cut it.

We evaluated multiple frontier agent configurations — including Claude Code (Opus 4.6, Sonnet 4.6), OpenAI Codex (GPT-5.2, GPT-5.4), and Nomic's domain-specific agent — across all three complexity tiers. The full results are in the paper, and they are worth reading. But the headline finding has nothing to do with which model scored highest.

The headline finding is that the harness matters more than the model.

Retrieval Is the Bottleneck, Not Reasoning

Agents frequently fail before they get to the hard part — before any engineering judgment is required — because they cannot reliably locate the right sheet or the right cross-reference within a complex multimodal document set. Retrieval has to be embedded in the agent's logic and reasoning approach. Without the right harness, agents defaulted to treating rich construction drawings as flat text files via pdftotext, a fundamental mismatch with the structure and visual density of real AEC documents.

The model was fine. The harness was wrong.

But when equipped with domain-specific document parsing and retrieval, performance jumped dramatically. Gains of twenty to thirty points on the hardest task families, far exceeding what any model upgrade alone could deliver.

That's the empirical version of the claim I have been making for months: in domain-specific work, the operating environment is part of the capability.

Benchmarks Are Not Leaderboards

This matters because it changes what benchmarks are for.

If you think a benchmark is a leaderboard — a place to crown a winner — then AEC-Bench tells you which agent configuration scored highest. Fine. But that is the least interesting thing it does.

The more important function of a benchmark like this is diagnostic. It tells you where the system breaks, why it breaks, and what kind of intervention would fix it. Retrieval matters more than reasoning on these tasks. Document understanding is a harder unsolved problem than calculation or code generation. And the difference between a model that scores 40% and a model that scores 70% might not be the model at all — it might be the tools it was given.

That diagnostic function is why the "it works on my example" style of evaluation is so dangerous. An anecdote has no control group. It has no complexity tiers. It does not distinguish between a model that got lucky on a simple case and a system that performs reliably across difficulty levels and task types. The anecdote tells you what happened once. The benchmark tells you what to expect, and more importantly, what to fix.

If you want AI to work in engineering, stop collecting demos and start building benchmarks.

Benchmarks can ossify. They can reward gaming over genuine capability. But they're still the only tool that separates signal from anecdote.

For anyone building AI systems for AEC, or for any domain where the work is artifact-bound and intolerant of plausibly generic answers: rigorous evaluation on real documents at real complexity is the minimum. And the full system has to be in scope — that's where the capability actually lives.

What Comes Next

AEC-Bench is a start, but it is only a start. The industry needs benchmarks that evolve — that grow with more task families, more disciplines, and document types the current set doesn't cover. It needs evaluation infrastructure that supports reproducibility, trajectory analysis, and systematic comparison of harness designs. Tooling that lets you understand how an agent succeeded or failed, and what that means for the next iteration.

This is what aec-bench is supposed to be — an open platform for AEC agent evaluation — and we will have more to share soon.

In the meantime, the paper and benchmark dataset are available now. If you're interested making AI work in the built environment, take a look and share your thoughts! And if you're still relying on anecdotes to judge what these systems can do, consider that you might be optimising for demos when you should be optimising for deployment.

What If the Harness Could Improve Itself?

Theodoros Galanos — Sat, 14 Mar 2026 00:00:00 GMT

import BarsWithBaseline from '../../components/charts/BarsWithBaseline.astro'; import ConceptGrid from '../../components/charts/ConceptGrid.astro'; import FlowDiagram from '../../components/charts/FlowDiagram.astro'; import GroupedBars from '../../components/charts/GroupedBars.astro'; import { architectureData, bondsComparisonData, crossModelBondsL0Data, crossmodelL0Data, session2TrajectoryData, session3L0TrajectoryData, session4Gpt41miniTrajectoryData, strategyContrastData, } from '../../data/autoresearch-charts';

Estimated reading time: 25 minutes

About a week ago, Andrej Karpathy released autoresearch: a system where an LLM agent improves a model training pipeline overnight. The agent edits code, trains for five minutes, checks whether the result improved, and keeps or discards the change. Over roughly 700 experiments, it found around 20 real improvements and cut GPT-2 training time by 11%.

The design is compact: one markdown file, one editable target (train.py), one metric (val_bpb), and a greedy keep-or-revert loop.

But training code is not the only thing that determines how well an AI system performs.

In Where Capability Actually Lives in Agentic Engineering, I argued that in domain-specific work, performance is distributed across the whole system: model, tools, prompts, verification, and output contracts. In Benchmarking Agents on Real Engineering Work, initial results supported that claim. When workflow support was removed, capability did not degrade gracefully. It collapsed.

That finding raises a question this article tries to answer: if the environment around the model carries a meaningful share of the capability, can part of that environment begin improving itself?

If the operating environment carries capability, then improving the environment is improving the capability. And if the improvement loop can be automated, it compounds.

This article applies that pattern to only one part of an engineering evaluation harness: the system prompt that guides audit behaviour. It lays out the design, reports four early experiment blocks on HVAC audit tasks, and examines what this kind of recursive improvement actually surfaces.

TL;DR

We applied the autoresearch pattern to one surface of our harness: the system prompt that guides audit behaviour.
The system uses an information barrier so the autoresearcher agent sees patterns in outcomes, not task answers.
Behavioural feedback from an agentic-bonds classifier gives a richer signal than reward alone by showing how the agent distributed effort across execution, deliberation, exploration, and verification.
Across the Claude full-reference runs, one prompt change improved mean reward from 0.94 to 0.98 across five task instances. The entire change was two sentences: an explicit per-room verification checklist.
On Claude at L0, that full-reference winner backfired. Confidence thresholds and cross-room consistency instead improved reward from 0.73 to 1.0 on one instance.
On GPT-4.1-mini at L0, the same high-level strategy transferred, but the wording had to become a hard verification gate, and the model still topped out at 0.83.
For Claude, perfect runs were more execution-heavy and less verification-heavy than partial ones. The difference was timing, not quantity: the strongest runs executed first and verified later in concentrated blocks.
The main lesson is conditional. Prompt strategy depends both on the information available and on the model receiving the prompt.

Why Autoresearch Works

Karpathy's design works because it fixes the evaluation loop into a form that can run unattended.

Fixed time budget. Every experiment runs for exactly five minutes. Scores are comparable because improvements cannot hide behind longer training.
Single artefact. The agent edits one file. That constrains the search space and keeps diffs reviewable.
Binary keep/discard. The metric improved or it did not. No subjective judgement required.
Git as memory. Every experiment is a commit, so the branch tip stays at the best known configuration.

As Delip Rao's prompt anatomy notes, program.md is not a loose prompt but an operating procedure: context, constraints, target, audit trail, loop, and safety valves. That is why the system works.

As Philipp Schmid notes, when experiments run far faster than a human can manage, the bottleneck becomes the evaluation system. If the harness is a bottleneck, improving the harness becomes a high-leverage move.

Why Engineering Harnesses Are Different from Software Agent Harnesses

Translating the autoresearch pattern to engineering harnesses requires three design changes.

The information barrier

In autoresearch, the autoresearcher agent sees everything: full training logs, exact loss curves, model weights. In our case, the artefact is a system prompt. If the autoresearcher could see the task content or planted errors, it could bake that knowledge into the prompt and overfit the benchmark.

So we need an information barrier between raw trial outputs and the autoresearcher. The autoresearcher sees patterns — "3 of the last 5 runs had incomplete coverage" — but never answers. That is the key structural difference from autoresearch.

More Than One Score

Autoresearch has one scalar: val_bpb. Engineering harnesses need more. A score of 0.7 could mean missed findings, formatting errors, or a timeout. The autoresearcher needs to know not just how well the agent scored, but how it behaved.

We address this with three feedback channels:

Reward — a float between 0.0 and 1.0 produced by a deterministic verifier that scores how completely and correctly the agent completed the audit task.
Failure categories — abstracted failure modes without task-specific details: incomplete_coverage, incorrect_values, format_error, timeout, false_positives, clean_miss.
Behavioural profile — a classification of each turn in the trace using our agentic-bonds classifier. It labels turns as execution, deliberation, exploration, or verification, and returns the distribution, temporal sequence, and a short narrative summary.

That gives the autoresearcher more than a score drop. It can tell whether the agent stopped verifying, front-loaded execution, or spent too long deliberating.

The Target Is the Workflow Prompt

Autoresearch optimises model architecture and hyperparameters. The artefact is code that defines how a neural network trains. In our case, the artefact is a system prompt: workflow instructions for audit work. It changes how the agent works, not what it knows. Prompts were the easiest surface to start with: easy to isolate, easy to diff, and easy to evaluate. As the harness becomes more structured, other surfaces like task generation, scoring rubrics, and tool workflows become plausible optimisation targets too.

This is closer to optimising an organisation's operating procedure than to optimising code. The question is not "what parameters produce the lowest loss?" but "what workflow instructions produce the most reliable engineering review?"

The Design: Automated Prompt Optimisation for HVAC Audit Tasks

The system has three layers with a strict information flow.

The autoresearcher agent runs locally, driven by Claude Code following a program.md — the same pattern as autoresearch. It edits system prompts, makes git commits, reads sanitised feedback, and decides whether to keep or revert each change. It never sees task content, verifier code, or raw conversation transcripts.
Support scripts handle the structured mechanics. run_experiment.py triggers a Harbor job in a Docker sandbox with the current system prompt. feedback.py enforces the information barrier: it reads the raw trial outputs, strips task-specific content, runs the bonds classifier, and returns a sanitised JSON summary. results.py manages the TSV audit trail.
The experiment sandbox is unchanged from the existing evaluation harness — the same Docker containers, agents, verifiers, and output contracts used in the benchmark work from the previous article. The only thing that changes between iterations is the system prompt file mounted into the container.

The loop mirrors autoresearch exactly:

Edit the system prompt
Git commit
Run experiment in sandbox
Read sanitised feedback
Keep if reward improved, revert if not
Repeat

The program.md is organised into eleven sections: purpose, setup, scope constraints, artefact definition, experiment execution, feedback reading, optimisation targets, the loop itself, logging, autonomy mandate, and information discipline. The information discipline section is the one that has no analogue in Karpathy's version: it explicitly instructs the autoresearcher not to attempt reading task files or verifier code, and frames any desire to know specific task details as a signal to focus on process guidance instead.

What Behavioural Feedback Adds to Agent Evaluation

The agentic-bonds classifier gives the autoresearcher something autoresearch does not have: a temporal view of how the agent distributed its effort across the trace.

Each assistant turn in the conversation trace is classified as one of four types:

Execution — doing the obvious next step: calling tools, formatting output, writing results
Deliberation — committed reasoning about how to solve a problem, step by step
Exploration — comparing alternatives, deciding what to do
Verification — checking backward at its own work, comparing results against expectations

The autoresearcher sees this at three levels: the bond profile (distribution), the bond sequence (temporal pattern), and a short bond narrative.

What the classifier actually found

We classified all 30 Claude Sonnet 4.6 traces from these runs — 12 perfect (reward 1.0) and 18 partial (reward 0.9). The classifier should be read as a behavioural lens rather than ground truth about cognition, but even on that basis the headline finding was counterintuitive.

Perfect traces are more execution-heavy, not more verification-heavy.

	Execution	Verification	Deliberation
Perfect (1.0)	55.5%	41.9%	2.6%
Partial (0.9)	45.3%	49.0%	5.8%

Exploration was effectively absent in this set of traces, which is why it does not appear as a meaningful part of the comparison.

The agents that scored perfectly spent less time verifying than the ones that missed an error. The execution-to-verification ratio shows the same pattern: 1.71 for perfect traces versus 1.06 for partial ones. This does not mean verification is unhelpful. It means badly timed verification correlates with worse performance.

Other patterns we see in the data:

Perfect traces execute first, verify later. In the first half of the trace, perfect agents spent only 28% of turns on verification, compared to 41% for partial agents. By the second half, both groups converged around 55%. Perfect agents front-loaded decisive execution, then verified in concentrated blocks. Partial agents hedged earlier — which appears to reflect uncertainty rather than thoroughness.
Perfect traces build momentum. The average longest uninterrupted execution streak was 3.8 turns for perfect traces, compared to 2.4 for partial. Perfect agents commit to a direction and sustain it. Partial agents interrupt themselves to verify before they have built enough context for verification to be useful.
Deliberation correlates with imperfection. Only 25% of perfect traces contained any deliberation turns, compared to 56% of partial traces. When the agent pauses to reason about how to proceed, it is more likely to miss something. This suggests that deliberation in these traces signals uncertainty or confusion, not carefulness.
Start patterns diverge. 58% of perfect traces opened with three consecutive execution turns (exec | exec | exec). Only 28% of partial traces did the same — most interrupted with verification or deliberation by the third turn.

What this means for prompt design

The behavioural data sharpens the prompt improvement story. The successful change — adding a systematic checklist (a) inputs, (b) components, (c) ventilation, (d) totals — did not add verification. It structured the execution phase so the agent could work systematically before verifying.

In other words, the checklist made the agent behave more like the perfect traces already did: decisive execution, then concentrated verification.

Early Results: How Small Harness Changes Moved HVAC Audit Performance

We ran four experiment blocks on the system prompt that guides HVAC schedule audit tasks. One established the single-instance trap, one found a prompt improvement across five full-reference instances, one showed that the best strategy changes at L0, and one repeated the L0 setup on a weaker model.

Claude on One Full-Reference Task

The first Claude Sonnet 4.6 run tested five prompt modifications against a single task instance (adelaide-15rm). The baseline scored 0.9 — four of five planted errors detected.

Five strategies were tried: a planning step, cascade-error reporting, field-by-field verification, larger batches, and a re-verification pass. None moved the reward.

The autoresearcher reasonably concluded that the fifth error looked like a model capability boundary rather than a prompt issue. The one useful finding was about efficiency: the larger-batch approach achieved the same accuracy with fewer turns and tokens — 9 turns and 98k tokens versus the baseline's 10 turns and 105k.

That conclusion turned out to be wrong. Not because the reasoning was poor, but because the evidence was insufficient. Single-instance evaluation made a task-specific ceiling look like a universal one.

Claude on Five Full-Reference Tasks

The second Claude Sonnet 4.6 run tested four modifications against five instances simultaneously (Adelaide, Brisbane, Darwin, Melbourne, Perth). The baseline mean reward was 0.94: three instances at 0.9 and two at 1.0.

Three approaches failed:

Iteration 1 added a completeness rule: "verify every room, do not skip rooms, a missed error costs more than an extra turn." No reward change. The instruction was redundant.
Iteration 2 added zero tolerance for numerical mismatches: "report ANY difference, do not dismiss as rounding." No reward change, but efficiency degraded sharply — 64 turns and 685k tokens versus the baseline's 52 turns and 553k. The agent re-checked more and found nothing new.
Iteration 3 restructured the workflow to a two-pass approach: sweep all totals first, then deep-dive flagged rooms. No reward improvement, and efficiency nearly doubled — 79 turns and 1.08 million tokens. It mostly shuffled which errors were caught without improving overall coverage.

Iteration 4 worked. The change was small: two sentences extended. The orient step added design conditions. The verify step added an explicit per-room checklist: (a) input parameters, (b) each heat gain component, (c) ventilation terms, (d) totals. Mean reward rose from 0.94 to 0.98. Adelaide and Melbourne both jumped from 0.9 to 1.0. Only Darwin remained at 0.9. The efficiency cost was modest: 55 turns and 613k tokens, an 11% increase over baseline.

Here is the actual diff — the entire improvement:

-1. **Orient (1 turn):** Read the schedule and identify the number of rooms
-   and available tools. Count your rooms — this determines your turn budget.
-2. **Verify (1 turn per 2-3 rooms):** For each batch of rooms, call the
-   calculation tool, then compare results against the schedule values.
-   Note discrepancies immediately.
+1. **Orient (1 turn):** Read the schedule and identify the number of rooms
+   and available tools. Count your rooms — this determines your turn budget.
+   Before proceeding, note the design conditions: outdoor temperature,
+   humidity, and any building-level parameters.
+2. **Verify (1 turn per 2-3 rooms):** For each batch of rooms, call the
+   calculation tool, then compare results against the schedule values.
+   For each room, systematically check: (a) input parameters (occupancy,
+   area, volume), (b) each heat gain component, (c) ventilation terms,
+   (d) totals. Note discrepancies immediately.

The improvement came from making an implicit checking procedure explicit.

What the failed approaches have in common

The three failed approaches all tried to add more work. The successful one structured the existing work differently.

Iteration	Approach	Reward	Token cost vs baseline
Baseline	—	0.94	—
1	Add completeness rule	0.94	-1%
2	Zero tolerance	0.94	+24%
3	Two-pass restructure	0.94	+95%
4	Systematic checklist	0.98	+11%

That pattern — structure matters more than volume — echoes the previous article's result that the strongest models did not simply verify more; they verified as part of a structured workflow.

Claude at L0

The Claude L0 run targeted the hardest reference level: audit-office-building-L0. Here the agent receives only room type, floor area, ceiling height, and location. It does not receive design conditions, formulas, or lookup tables. The prompt is operating in a much thinner information environment.

The baseline reward on the Adelaide L0 instance was 0.73. The agent found 2 of 3 issues, but it also produced false positives.

The first useful change was not a better checking procedure. It was a confidence threshold: only report an error if the model can identify the specific assumption, parameter, or formula that appears to be wrong. That lifted reward from 0.73 to 0.83 by eliminating false positives.

The next result was the important one. The full-reference winner — the systematic per-room checklist — did not generalise. It backfired. Reward dropped from 0.83 to 0.53, and false positives returned. At full-reference levels, granular checks help because the agent can anchor them against formulas and tables. At L0, the same granularity forces the agent to compare schedule values against assumptions it is partly reconstructing from memory.

The winning change was different in kind. Adding a cross-room consistency rule — compare similar rooms against one another and investigate large relative differences — lifted reward from 0.83 to 1.0 on this instance, while also reducing token usage sharply.

That is a strong result, but it is still single-instance. It shows that the strategy can change sharply when reference material is removed. It does not yet prove that we have the generally best L0 prompt.

What it does show is that prompt optimisation is reference-level-specific. At high reference levels, the prompt benefited from explicit granular verification. At L0, the better strategy was uncertainty management plus relative comparison.

L0 Iteration	Strategy	Reward	Tokens In	Outcome
Baseline	Original prompt	0.73	304k	2/3 findings + false positives
1	Confidence threshold	0.83	248k	False positives removed
2	Systematic checklist	0.53	407k	Backfired
3	Cross-room consistency	1.00	104k	3/3 findings, no false positives
5	Add design-conditions note	1.00	258k	Same reward, worse efficiency

GPT-4.1-mini at L0

The GPT-4.1-mini L0 run repeated the same task on the weakest model from the earlier benchmark. The baseline was much worse than Claude's: reward 0.37, only 1 of 3 planted errors found, and 18 false positives.

What transferred was the strategy, not the exact wording. The soft confidence-threshold language that helped Claude did not work here. But once the instruction was rewritten as a hard verification gate — do not report a discrepancy unless the tool confirms it — false positives disappeared and reward jumped to 0.83.

That shows two things at once. First, the underlying prompt idea generalises across model families: reduce unsupported claims, rely on internal consistency, and force the model to ground discrepancies. Second, the wording still has to match the model's instruction-following style.

The ceiling also remained model-specific. GPT-4.1-mini improved dramatically, but it did not reach Claude's L0 result. The best run found 2 of 3 issues with no false positives.

Model	Baseline	Best	Winning strategy	Constraint style
Claude Sonnet 4.6	0.73	1.00	Confidence threshold + cross-room consistency	Advisory
GPT-4.1-mini	0.37	0.83	Verification gate + cross-room consistency	Prohibitive

The early behavioural picture suggests the two wins differ not only in wording, but also in how the models stabilise. Claude's successful L0 run remained relatively execution-compatible. GPT-4.1-mini's best run was much more verification-heavy.

What the Results Suggest for Harness Engineering in AEC

Self-improving harnesses are feasible

The system worked. The autoresearcher agent formed hypotheses from sanitised feedback, made targeted prompt changes, measured their effect, and advanced the branch when something improved. The information barrier appeared to hold, and some improvements generalised across instances.

This is obviously a small result, but it validates the basic mechanism.

The environment is improvable, not just measurable

The previous two articles established that the operating environment carries capability and that this dependence is measurable. This article adds a narrower claim: at least one important part of that environment, the workflow prompt, is improvable through automated search. If harness improvement can be partially automated, investment in evaluation infrastructure compounds.

Single-instance evaluation misleads

The single-instance Claude run concluded that the reward ceiling was a model capability boundary. The five-instance Claude run proved that wrong. The ceiling was instance-specific. Multi-instance evaluation was necessary to discover that the prompt could improve, and which prompt changes actually generalised.

This point travels beyond this setup. Evaluating on a single test case can make genuine improvement opportunities look like hard limits.

Structure beats volume

The most expensive failed approach cost nearly twice the baseline in tokens and produced no improvement. The successful approach cost 11% more and raised reward by 4 percentage points. It is possible that, in engineering review work, telling an agent how to check is more effective than telling it to check more.

That said, the Claude L0 run adds an important qualifier. The right structure depends on the reference level. On full-reference tasks, structure meant explicit component-by-component verification. On L0, structure meant constraining when the agent should trust its own judgement and shifting toward relative comparisons inside the schedule itself.

Prompt optimisation is reference-level-specific

The L0 run is the strongest evidence in the whole set that there may be no single best workflow prompt. The same checklist that helped at full reference actively hurt at L0. Removing reference material changed what kind of prompt guidance was useful.

That suggests a more general rule: prompt quality is conditional on information availability. A prompt that works well when the environment supplies formulas, tables, and design conditions may fail when the model has to reconstruct too much of that context from memory.

Strategy transfers across models, wording does not

The GPT-4.1-mini run adds a second qualifier. The high-level idea that worked at L0 transferred across model families: suppress unsupported discrepancies and use relative comparisons when absolute references are weak. But the wording had to change. GPT-4.1-mini did not respond reliably to soft advisory language. It improved only when the same idea was expressed as a hard gate.

That suggests prompt portability has two layers. Strategy may travel. Surface phrasing may not. It also suggests that prompt optimisation has a ceiling. GPT-4.1-mini improved much more in relative terms than Claude, but it still stopped well short of Claude's absolute result.

Behavioural analysis inverted our assumptions

Before classifying the traces, we assumed that more verification would correlate with better performance. The data showed the opposite: perfect traces were execution-dominant, while partial traces were roughly balanced. The goal is not to maximise verification. It is to make execution decisive enough that verification can happen in concentrated blocks rather than as reactive interruptions.

Honest Limitations

This is early work. The scope is narrow and the results are preliminary.

Single task type, single domain. All experiments targeted HVAC audit tasks. We do not know whether the checklist approach generalises to other engineering task types.
Small iteration count. Four experiment blocks are enough to validate the mechanism and expose two important conditionalities, but not enough to map the full improvement frontier.
Bonds data is still thin outside the Claude run and has no human-annotated validation set here. The 30-trace behavioural analysis is for Claude Sonnet 4.6 on one task family. The GPT-4.1-mini read is directionally useful, but much smaller.
The information barrier is untested against adversarial pressure. The autoresearcher followed the information discipline in these runs, but we have not stress-tested whether a sufficiently capable autoresearcher agent might infer task-specific content from the feedback patterns.
The L0 result is still single-instance. The 1.0 score is encouraging, and it was reproduced once with a slightly more expensive prompt variant, but we have not yet tested the L0-optimised prompt across multiple cities. The five-instance Claude run already showed how misleading single-instance conclusions can be.
The GPT-4.1-mini comparison is also single-instance. We do not yet know whether the 0.83 ceiling or the prompt fragility pattern will hold across other cities or adjacent task families.
Darwin remains at 0.9. One instance did not improve across any prompt change. This may be a genuine model capability limit for that specific error type, but we would need to break the information barrier to investigate.
Stochastic variance is bounded but real. Iteration 3 showed Brisbane and Melbourne swapping scores, suggesting approximately 5-10% variance per instance. Larger sample sizes would help distinguish signal from noise.

What Comes Next

The system currently optimises one surface: system prompts. Two obvious new surfaces remain:

Task generation — using the loop to generate new task instances that produce useful signal.
Scoring rubrics — iterating on how agent output is evaluated, especially for tasks with qualitative judgement.

The prompt surface itself can also become more adaptive:

Level-adaptive prompts — switching workflow strategy based on how much reference material is available. The contrast between the five-instance full-reference Claude run and the Claude L0 run suggests that prompt selection may need to be conditional rather than global.
Model-adaptive prompts — varying the instruction style as well as the workflow strategy. The GPT-4.1-mini run suggests that the same conceptual rule may need different wording for different models.

The longer-term question is whether prompts, tasks, and rubrics can be improved jointly. That is where recursive harness improvement starts to look more like a research programme.

For now, the result is small and specific: an autonomous loop, an information barrier, behavioural feedback that inverted our assumptions about verification, one prompt change that helped at full reference, another that worked for a very different reason at L0, and a cross-model comparison showing that prompt ideas can transfer even when prompt wording does not. The environment was improvable in each case, but the successful strategy depended on what information the environment already supplied and which model was inside it. The agents that scored best were not simply the ones that checked the most. They were the ones whose workflow matched the structure of the task, the information available, and the model's own behavioural constraints.

That, at least, is consistent with everything we have been learning about where capability actually lives.

Benchmarking Agents on Real Engineering Work Is Already Teaching Us Something Important

Theodoros Galanos — Thu, 12 Mar 2026 00:00:00 GMT

import BarsWithBaseline from '../../components/charts/BarsWithBaseline.astro'; import BondTypes from '../../components/charts/BondTypes.astro'; import GroupedBars from '../../components/charts/GroupedBars.astro'; import OutcomeDistribution from '../../components/charts/OutcomeDistribution.astro'; import QualityVsCost from '../../components/charts/QualityVsCost.astro'; import StackedBars from '../../components/charts/StackedBars.astro'; import { ablationData, auditMixedUseFrontierData, bondTypesData, heroRewardData, outcomeDistributionData, qualityVsCostData, verificationGradientData, } from '../../data/benchmarking-charts';

Estimated reading time: 15 minutes

Current frontier model performance is still heavily concentrated in a narrow band of well-covered domains, especially code, math, and adjacent text-heavy tasks.

Engineering is not one of them.

That is the starting point for this work.

In Where Capability Actually Lives in Agentic Engineering, I argued that progress in this domain will not come from better models alone. It will come from better operating conditions: better tools, better harnesses, and better environments for reliable work. This article is a first empirical step in that direction.

That earlier piece made a conceptual claim about where capability lives. This one asks whether that claim survives contact with measured performance.

In domain-specific work, the environments agents operate in are part of the capability.

The aim here is to start measuring that claim on real engineering tasks. The longer-term goal is a benchmark, but this piece reports an early run toward one: a single task, from a single discipline, using one agent harness setup, tested across a small set of models.

Concretely, the work focuses on HVAC heat load calculations and schedule audits, grounded in the kinds of structured documents that show up in actual AEC workflows, and includes nearly 480 trials with the same harness, the same tools, and the same output contract. The harness was a tool-using agent, and the tool itself was deliberately strong: the calculation procedure was reverse engineered into code, the relevant lookup values were exposed, and the models were given something close to an oracle calculator rather than being asked to derive the whole method from scratch.

That is narrow by design, and still early. But it is already enough to surface useful patterns. The point of sharing it now is not to pretend the benchmark is finished. It is to put the initial results and the early methodology in the open, get feedback, and sharpen the next iterations while the benchmark is still taking shape.

What follows is a first look at what current agents are good at, where they still break, and why evaluation design matters almost as much as model quality. It is also an early step in turning a vague domain gap into a concrete improvement programme.

TL;DR

This benchmark is best read as evidence about conditional capability. Inside a strong harness, current frontier agents can do meaningful engineering review work. Once guidance and tool support are stripped away, performance often collapses rather than degrading gracefully.
The model ranking is clear, but it is not the deepest result. Sonnet 4.6 is strongest overall, and Haiku 4.5 is the most attractive quality-per-dollar option in this setup.
The most informative tasks were audit tasks, not calculation tasks. Once the harness supplied something close to an oracle calculator, the real separation moved into checking, discrepancy detection, and reliable completion.
Verification behaviour appears to matter a lot. The strongest model did not just verify when it was in trouble. It verified as part of its default workflow, which looks like part of the mechanism behind its recall advantage.
This is an empirical counterpart to the argument in the previous article, and the beginning of a broader benchmark effort. It supports the claim that in engineering, capability is not just a model property. It is distributed across the model, the harness, the tools, the verifiers, and the output contract. The longer-term goal is to build high-quality benchmarks with enough coverage to support meaningful progress in agentic engineering.

That is the core result in compressed form: the benchmark ranks models, but more importantly, it makes the system dependence visible.

The Benchmark Was Simple on Purpose: HVAC Heat Load Tasks as a Starting Point

That narrow setup matters because it tells us something specific.

Within this first experiment, the focus was one meaningful slice of engineering work: mechanical heat-load tasks. Even inside that small scope, two task families behaved very differently.

Calculation tasks ask the agent to compute loads correctly from structured room inputs.

Audit tasks ask the agent to inspect schedules, identify discrepancies, and propose the correct fixes.

That distinction ended up being crucial.

The calculation tasks were close to saturated for the Anthropic models. They are useful as a baseline, but they do not separate strong agents from stronger ones.

That result is even more telling once you know the setup. The method was not hidden inside the task and left for the model to rediscover. The calculation approach was encoded directly into the harness as a near-oracle tool: a coded procedure plus the relevant lookup table values and calculator logic. Even with that help, not every model was perfect, and the harder audit tasks still separated the field clearly.

That is where the real spread appears: systematic checking, consistency across many rows, and enough discipline to finish with the correct structured output.

They also give us a useful improvement gradient. When an agent fails, the failure is usually legible. It missed a discrepancy, checked too shallowly, used the tools badly, ran out of turns, or never made it cleanly into the required output format. That is exactly what you want from a benchmark aimed at building better agent environments around real domain tasks.

For AEC practitioners, that should feel familiar. The painful mistakes in practice are often not a single wrong formula. They are missed discrepancies, skipped checks, and brittle review processes.

For agent evaluation people, it is a reminder that task design determines what you learn. If the task is too easy, you are mostly benchmarking formatting and latency. If it is too synthetic, you may learn very little about deployed usefulness.

The Model Ranking Is Clear, but It Is Not the Whole Story

At the headline level, the ranking is straightforward.

Sonnet 4.6 was best overall. It was the only model to clear both near-perfect calculation performance and clearly best-in-class audit recall. It also had zero zero-score trials.

Haiku 4.5 came second on accuracy and first on value. Its overall reward was 96.3%, and while it trailed Sonnet 4.6 on audit recall, it stayed strong enough that its much lower per-trial cost changes the deployment conversation.

Sonnet 4 remains solid. It is cheaper than 4.6 and more accurate than GPT-4.1-mini by a wide margin, but the newer generation models have moved the frontier.

GPT-4.1-mini was not competitive for this workload. Its overall reward was 34.1%, with a 64% zero-rate. The issue was broader than engineering reasoning. A large share of failures were format failures, truncated outputs, or prose that never turned into the required JSON result.

That last detail matters. In an eval setting, people sometimes treat format failures as a nuisance variable. In deployed agent systems, they are part of the failure surface. If an agent can reason but cannot reliably finish the job in the required structure, it still failed.

If your eval is shallow, your conclusions will be shallow too.

The Biggest Lesson Was About the System, Not the Model

One of the clearest findings in this benchmark is that harness choices materially change measured capability.

This is the most direct continuity with the previous article. There, the claim was conceptual: the harness is part of the capability story. Here, the same point shows up empirically in the numbers.

Even simple harness changes moved results. In the early setup, a 10-turn cap made one audit workload look almost impossible for an otherwise capable model, but turn budget was only part of the problem. With weaker guidance, the default strategy was to decompose the audit room by room and spend roughly two turns per room in an execute-then-verify rhythm. That produced real work, but it was the wrong workflow for the budget. The fix combined a higher cap with guidance that pushed the agent toward batching rather than treating every room as its own mini-loop.

The same pattern showed up elsewhere. Verifier fixes mattered. Prompt refinements mattered. Prompt caching mattered for cost. Better output instructions reduced avoidable formatting zeros for the Anthropic models. And the tool design mattered too: even when the harness provides something close to an oracle for the core calculation, model differences do not disappear. They move into disciplined checking, orchestration, and reliable completion.

The ablation results make the point more sharply. Reducing the turn budget from 20 to 10 is one kind of degradation. Removing guidance and tool support is another. The first makes the task harder. The second starts to expose the capability boundary. On this task family, the gap between strong-harness performance and low-guidance performance is larger than many of the model-to-model differences people usually focus on.

Those are not side details. They are part of the measured system. The harness is not neutral background. It is an active ingredient in whether a model can express the capability it already has.

For practitioners, this means you should be skeptical of any claim that a model simply can or cannot do a workflow based on a weak first-pass eval.

For evaluation researchers, it means benchmark design has to be treated with the same rigor as model comparison itself.

What Happened When We Removed Harness Guidance

This is also where the setup gets more interesting.

From the beginning, one of the key questions was how much of the measured capability depended on the operating conditions around the model, not just which model performed best inside the strongest harness.

So after the main tool-enabled runs, a small guidance-ablation ladder was set up around the same office-building audit family.

The idea was simple. Keep the underlying task family fixed, then remove support in stages.

Condition	Support removed or added
`L0`	bare problem statement, no embedded formulas or lookup table
`L1`	effectively the same as `L0` in the current task set, which turned out to be informative in its own right
`L2`	adds the psychrometric constants and explicit outside-air rules
`L3`	replaces that with a compact AS 1668.2 reference table and general calculation guidance
`no-tool`	removes the calculation tool entirely and asks the model to do the audit directly from the prompt context

This was not meant to be a polished benchmark surface. It was a probe. The point was to see how quickly the task collapses once the environment starts losing structure.

The results were blunt.

On the tool-enabled baseline, Sonnet 4.6 averaged 0.966 reward on the office-building audit set. Dropping the turn budget from 20 to 10 lowered that to 0.943. That is a real degradation, but it still leaves the task clearly inside the model's workable envelope.

The guidance ladder was a different story.

In the direct no-tool reference run, the current partial results look like this:

That tells us a few things. The cleanest way to read it is as three different regimes: a workable strong-harness regime, a mildly degraded budget-constrained regime, and a collapse regime once the environment stops carrying enough of the method.

First, the budget ablation and the guidance ablation are not the same phenomenon. Reducing turns hurts, but the agent still basically knows what kind of work it is doing. Removing guidance and tool support is much harsher. Most of those conditions do not degrade gracefully. They collapse.

Second, this is exactly the kind of out-of-distribution behaviour we were worried about.

In-distribution domains are the places where models have already seen enough adjacent structure that they can interpolate their way through the task even when the scaffold is weak. Engineering audit work did not behave like that here. Once the environment stopped carrying key pieces of the method, performance did not taper off a little. It mostly went to zero.

Third, the surviving no-tool signal matters precisely because it is weak. There was a small amount of non-zero performance there. That suggests the capability is not entirely absent. But it is nowhere near robust enough to treat the task as natively solved. In other words, the environment is still doing real cognitive work for the model.

That is the larger point.

When people say a model can do engineering, they often leave unspoken how much hidden structure is being provided by the harness, the tools, the prompt, or the reference data. Our ablation run makes that visible. In this task family, capability is highly conditional on the operating environment. Remove the support and the system does not simply get a bit worse. It often stops functioning in a useful way.

That is not a failure of evaluation. It is exactly what good evaluation is supposed to reveal.

It is also why the out-of-distribution framing matters so much. If the domain were already well-covered by the model's native priors, these ablations would look like inconvenience tests. Instead they look like capability boundary tests. That is a strong sign that for real engineering work, at least today, the harness is not a wrapper around the capability. It is part of the capability.

What the Strongest Model Did Differently

The cleanest behavioural finding in this work is about verification.

All models increased checking when they were struggling. But Sonnet 4.6 did something more interesting: it verified even when it was succeeding.

In our behavioural analysis, Sonnet 4.6 spent 23% of its successful traces in verification behaviour. The other strong models were much lower. Haiku showed the steepest verification gradient between success and failure, which makes it interesting for runtime monitoring, but Sonnet 4.6 made verification part of its default operating mode.

That appears to be the mechanism behind its recall lead.

In plain engineering terms, it behaved less like a model that checked at the end and more like one that treated review as part of execution. It did not only check when it sensed danger. It checked because checking was built into the workflow.

That matters in both practical engineering terms and evaluation terms.

For SMEs, it matches a familiar truth: good review practice is not a panic move. It is routine.

For agentic-eval people, it suggests that model quality may show up less in raw chain-of-thought style reasoning and more in when and how an agent decides to revisit earlier work. Reliability here depends on whether verification is part of the default workflow before failure starts to accumulate.

A Behavioural Lens From Reasoning Research

To get beyond simple success rates, we adapted the Agentic Bonds framework from Du et al.'s work on the molecular structure of thought.

The basic idea is that quality does not come only from individual steps. It comes from the pattern of transitions between types of steps.

We classified agent turns into four categories: execution, verification, deliberation, and exploration.

That gave us a behavioural fingerprint for each model.

Sonnet 4 looked like a rigid workhorse: high execution share, low exploration, highly predictable structure.

Haiku looked more adaptive, but also more verbose. It often spent more turns and generated a stronger distress signal when things were going badly.

GPT-4.1-mini produced the strangest result: success and failure were behaviourally almost indistinguishable. It did not seem to have a readable internal signal that it was in trouble.

That is a serious limitation if you want runtime monitoring or intervention. You cannot reliably rescue a model that does not behaviourally reveal when it is failing.

This kind of analysis complements task-level scoring. Accuracy tells you what happened. Behavioural structure starts to tell you why.

That framing came from the tool-loop traces. The no-tool runs exposed a different but complementary failure surface.

What The No-Tool Traces Revealed

The tool-loop traces gave us one kind of behavioural visibility: turn-by-turn structure. The no-tool and low-guidance runs gave us a different one. There we often only had the final written artefact, so the analysis had to be more forensic. We were no longer asking which turn type came next. We were asking a simpler and harsher question: did the model stay attached to the instance at all?

That ended up being one of the clearest behavioural signals in the whole project.

Once we read a broader sample of the direct no-tool and guidance-ladder traces, the main split was not simply success versus failure. It was anchored audit behaviour versus free-running domain narration.

The successful no-tool traces stayed tightly locked to the assigned schedule. They rebuilt the formulas from the prompt, carried the instance-specific constants through the arithmetic, and converged toward compact findings. Even without the calculator tool, they still behaved like audits. The single perfect Sydney no-tool trace is the clearest example of that pattern: it stayed on the given schedule, reconstructed the formulas locally, and still landed a verifier-clean result.

The failed traces were more interesting than simple arithmetic misses. They often looked superficially impressive. They used the right vocabulary. They wrote long engineering-sounding explanations. They sometimes did coherent local arithmetic. But many of them had already slipped off the actual task. They started substituting room programmes, changing climate conditions, inventing alternate schedules, or confidently asserting standard lookups that were not stably grounded in the prompt.

One Adelaide L3 trace, for example, stopped auditing the office-building schedule and began analyzing hotel rooms and hotel suites instead. A Brisbane no-tool failure turned into a different classroom-and-library schedule with its own invented occupancy logic. Both traces remained fluent. Neither stayed on the job.

That is the important distinction. The failure mode was often not "cannot calculate." It was "cannot stay on the instance."

That gave us a simple rubric for reading these traces. The key dimensions were instance fidelity, standards grounding, formula grounding, causal compression, and output discipline. The strongest traces stayed close to the presented schedule and compressed toward verifier-relevant findings. The weakest traces did the opposite: they drifted into generic HVAC explanation, expanded in length, and lost the contract.

From that manual read, a few recurring failure labels stood out.

Instance substitution: the model silently stopped auditing the presented office-building schedule and solved a different problem.
Generic-domain takeover: the trace remained fluent and domain-aware, but it had turned into an HVAC essay rather than an audit.
Standard hallucination: the model introduced confident but weakly grounded claims about AS 1668.2 lookups, occupant densities, or OA rates to justify a path it had invented.
Verbosity runaway: the trace expanded toward the output-token ceiling without improving task fidelity or output quality.

Failure shape	What it looks like in practice	Why it matters
Instance substitution	The trace silently swaps in a different schedule, room mix, or city conditions	The model is no longer auditing the assigned artefact
Generic-domain takeover	The writing stays fluent and technical but turns into generic HVAC explanation	Domain fluency masks loss of task control
Standard hallucination	The trace confidently asserts unsupported lookup values or code interpretations	It manufactures justification for the wrong path
Verbosity runaway	The trace expands toward the token ceiling without converging toward findings	Length substitutes for control

A few lines from the traces make the pattern obvious.

"### Room 1 — Hotel Room A (Hotel Bedrooms, 30 m²)"

That line came from an Adelaide L3 run that was supposed to be auditing an office-building schedule. By that point the trace was no longer on the assigned task at all.

"### Room 1 — Classroom A"

That came from a Brisbane no-tool failure. The model remained fluent and organized, but it had drifted into a classroom-and-library problem that was never in the prompt.

"Room 3 Errors: - Conduction W: given 4320, correct = 1600"

That comes from the successful Sydney no-tool trace. It is much less ornate, but it stays attached to the actual schedule and compresses toward the discrepancies that matter.

That pattern matters because it changes how to read the weak positive signal in the no-tool condition. The surviving no-tool traces suggest something specific: the model can sometimes reconstruct enough of the method to succeed, but only when it keeps a very tight lock on the actual instance. Once that lock breaks, domain fluency is not enough to rescue the audit.

This is exactly the kind of behaviour you would expect in an out-of-distribution domain. The model does not fail by becoming incoherent. It fails by becoming plausibly generic.

That matters for cost too, because these are not always short failures. Some of them are long, fluent, and expensive failures.

Time and Cost Need To Be Measured Together

The cheapest model per trial is not automatically the best value.

GPT-4.1-mini was cheapest in raw dollar terms, but too much of that spend was wasted because the outputs were unusable or incomplete.

Sonnet 4.6 was the most expensive, and part of that cost came from output verbosity. It generated much more output than the other Anthropic models, which limits how much prompt caching can save. The no-tool traces make the broader point clearly: a long failing trace is also a cost event.

Haiku 4.5 hit the most interesting middle ground. It was fast, much cheaper than Sonnet 4.6, and accurate enough that it dominated on reward-squared-per-dollar.

That metric matters because it punishes low accuracy sharply. A cheap wrong answer is not a bargain in review-heavy engineering workflows.

That matters especially in AEC, where the near-term deployment shape is often low request volume and very high task value. These are usually not million-QPS workloads. They are bounded but complex tasks that take skilled people real time to complete, and where the cost of a bad result can easily dominate the cost of the model run itself. In that setting, quality comes first.

The practical sequence is usually two-stage. First, you pay for quality in order to discover which tasks agents can actually do well enough to be useful. Only later, once those workflows are stable and you start scaling them across teams or organisations, does cost become the main optimisation target. At that point the question changes from can this task be done well to how broadly can we deploy it without losing quality or blowing up spend.

In this small experiment, if you want the highest ceiling, Sonnet 4.6 is the answer.

In the same narrow setting, if you want the strongest quality-per-dollar tradeoff, Haiku 4.5 is hard to ignore.

If you want a deployable system, the right answer probably depends on where in the workflow the agent sits and how much review coverage a human still provides.

What This Means for AI Agents in AEC

The practical takeaway is not that AI can now replace engineering judgement. That would be the wrong lesson.

The stronger result is narrower and more useful: on bounded, well-instrumented tasks, evaluation quality already matters as much as model selection.

The models were not most differentiated by calculation. They were differentiated by disciplined checking, completeness, and reliable finish behaviour. Those are exactly the traits that matter in real QA workflows.

So if you are trying to bring agents into AEC practice, one sensible near-term path is not full autonomy. It is scoped, auditable assistance on tasks where you can define the inputs, the expected outputs, and the failure modes clearly.

That is also where benchmarks can be genuinely useful: as a way to test whether an agent is ready for a specific class of work.

What This Means for Agentic Evaluation and Benchmarks

What this benchmark suggests is that three parts of the eval design matter especially strongly.

First, real task grounding. The benchmark should represent work that people actually care about getting right.

Second, harness transparency. Turn limits, verifier design, tool affordances, and output contracts are not implementation trivia. They are part of the measured system.

Third, behavioural instrumentation. If two agents get similar scores but fail in different ways, that difference matters. If one model exposes a strong distress signal and another does not, that matters too.

This is why benchmarks are most useful when those layers are visible together: domain realism, outcome quality, and agent behaviour. In that sense, AEC is a good stress case. It can be highly digitised in important workflow slices, it is economically important, and it is awkward enough to expose real capability gaps.

What Comes Next

The obvious next step is scale. One task in one discipline is enough to surface a potential pattern, but not enough to define a field. If this work is going to mature into a useful benchmark, it needs to grow into thousands of task instances across multiple disciplines, with enough breadth to distinguish narrow task skill from more general domain competence.

It also needs multimodality much earlier than many evals do. Design and engineering work are not purely text workflows. Drawings, schedules, details, markups, diagrams, and spatial context are central to the job. A serious benchmark for this domain will need multimodal inputs and multimodal tool use as part of the core design, not as an optional extension added later.

Then there is the harder evaluation problem: tasks where there is no single clean quantitative answer. A lot of real engineering work is about adequacy, judgement, prioritisation, and review quality rather than one exact number. That is where expert-authored rubrics, and eventually rubric-driven reward systems, become crucial. Recent work such as Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains points in that direction. If we want to benchmark useful domain work rather than only easily scored work, we will need much better machinery for structured qualitative evaluation.

And beyond single tasks, there is process. Many real workflows are long-horizon and compositional: they are made of many smaller tasks chained together across time, artefacts, and decisions. That is part of why starting with sharply scoped task instances still makes sense. They are the building blocks. Over time, the harder benchmark will be the composition problem: whether agents can string those capabilities together reliably across longer processes without losing quality, context, or control.

The Current Bottom Line

If you force a one-line conclusion, it is this:

Current agent capability on real engineering tasks is still highly conditional on the operating environment: the best systems verify better, finish more reliably, and look much weaker once the scaffold is removed.

That is encouraging, but it is also a warning.

You can learn the wrong lesson from a bad eval.

And you can misunderstand both strength and weakness if you are only looking at model names instead of the full system around them.

This work is still in its early phases and still narrow. But it is already telling us something useful: the next layer of progress will not come from bigger scoreboards alone. It will come from better tasks, better harnesses, and a clearer view of how much of domain capability is native to the model and how much is being supplied by the environment around it.

Where Capability Actually Lives in Agentic Engineering

Theodoros Galanos — Tue, 10 Mar 2026 00:00:00 GMT

import ConceptGrid from '../../components/charts/ConceptGrid.astro'; import { harnessLayersData, openQuestionsData, taskProcessData, } from '../../data/where-capability-charts';

Estimated reading time: 16 minutes

Engineering is a useful stress test for agentic systems because it exposes a kind of weakness that general demos often hide. The problem is not that the domain is technical in the abstract. It is that the work is instance-bound, constraint-heavy, and intolerant of plausibly generic answers. To be useful here, an agent has to do more than sound competent. It has to stay attached to the artifact, preserve the method, survive verification, and finish in a form that another system or person can trust.

To be useful here, an agent has to do more than sound competent. It has to stay attached to the artifact, preserve the method, survive verification, and finish in a form that another system or person can trust.

That changes the way we should think about progress. In domains like engineering, capability cannot be understood as a property of the base model alone. What matters is whether the full system can remain faithful to the specific document, drawing, assumptions, standards, and constraints that define the task in front of it. The question is not just whether a model can reason in the neighborhood of the work. It is whether the operating environment lets reliable work happen at all.

This broader shift is already visible across AI development. State-of-the-art behaviour increasingly comes from compound systems (BAIR, 2024) rather than single model calls, and actual work-related use still occupies a much narrower space than theoretical capability would suggest, which is part of why deployment questions matter so much Anthropic, 2026. Engineering sharpens that pattern. It forces a harder question than many benchmark settings do: where does reliable capability actually live? How much is native to the model, how much is supplied by tools and verifiers, and how much emerges only when the environment is designed correctly, especially in a landscape where benchmark coverage is still skewed toward more convenient task domains.

That is the question this essay is really about.

TL;DR

In engineering, useful agent performance depends less on generic fluency than on staying attached to the specific artifact, assumptions, constraints, and output format of the task at hand.
That means capability does not live in the base model alone. It is distributed across the full system: tools, verifiers, control flow, output contracts, interfaces, and human review structure.
The real design problem is therefore not just model selection. It is harness design: deciding what the agent should infer, what should be externalised into tools, what should be checked, and how the work should remain controllable.
Many important engineering workflows are better understood as processes rather than isolated tasks, which makes orchestration, UX, visibility, and intervention part of the capability story too.
If we want meaningful progress in engineering and AEC, we need better experimental environments and benchmarks that reflect real artifacts, real failure modes, and the actual conditions required for reliable work.

Why AEC Engineering Breaks Most AI Agent Approaches

It is tempting to treat engineering as one more expert domain that large models will gradually absorb as they become smarter and better trained. There is some truth in that. Engineering work is full of technical language, quantitative reasoning, standards, and procedural knowledge, all of which are at least partially representable in text, math, and code. But that description misses the thing that makes the domain difficult in practice.

Engineering is not hard only because it contains specialized knowledge. It is hard because the work is tied to particular artifacts and particular consequences. A calculation is not just a calculation. It belongs to a drawing set, a schedule, a climate zone, a standard, a collection of assumptions, and a downstream chain of decisions that may depend on it being right for this case and not some nearby case. In that environment, generic competence is not enough. A system can sound perfectly fluent and still fail the job by drifting away from the assigned instance.

That is why instance fidelity matters so much. In many tasks, being approximately in the right conceptual neighborhood is enough to be useful. In engineering, that can be the beginning of failure rather than the end of it. A model that substitutes a different design scenario, silently shifts a governing assumption, applies the wrong interpretation of a standard, or answers in a way that cannot be checked by another party has not produced a near miss. It has broken attachment to the artifact. That kind of failure is especially dangerous because it can still look competent on a casual read.

This is also why engineering is a more interesting challenge than a simple test of mathematical or coding ability. The hard part is often not deriving a formula. It is staying inside the bounds of the real task while carrying the correct assumptions all the way through to a legible output. That starts to look less like raw problem solving and more like a problem of task control: can the system monitor what object it is operating on, track which assumptions still govern the case, notice when more checking is needed, and converge on a form of completion that another party can actually use? General-purpose capability does not automatically guarantee that kind of discipline, and one of the open questions is how much of it is native to the model versus supplied by the surrounding harness.

Seen that way, the question is not whether models can do engineering in some broad, promotional sense. The question is what conditions make engineering work stay controllable. That is a narrower question, but it is also the one that matters if the goal is not spectacle but dependable use.

The Harness Is Where Engineering Capability Gets Made

Once engineering is framed as artifact-bound and verification-sensitive work, the role of the harness looks different. It is no longer reasonable to think of the harness as a thin wrapper around model capability. The harness determines what the system can see, what method it can invoke, what constraints are explicit, what gets checked, what counts as a recoverable error, and what form the final answer must take. In other words, it helps determine not just how the system runs, but what kind of cognition the system is able to express.

That matters because many of the hardest parts of engineering workflows live in precisely those layers. Tools can carry method that would otherwise need to be reconstructed unreliably from prompt context. More than that, many tools are artifacts of accumulated domain expertise. They are places where a field has already embedded procedures, assumptions, checks, tolerances, and accepted ways of doing work. In some cases that expertise is explicit and executable, as in calculations, lookups, and verification routines. In other cases it is more qualitative, showing up as workflows, review habits, and best-practice sequences. Verifiers can enforce habits of checking that a model may not apply consistently on its own. Output contracts can force the system to conclude in a format that is inspectable, comparable, and operationally usable. Turn budgets and control flow can decide whether an agent has enough room to complete a careful review or whether it will collapse into partial work and malformed output.

In that sense, the harness is not just infrastructure. It is part of the cognitive system. It allocates where reasoning happens, where discipline comes from, and how failure is surfaced. Recent work from OpenAI on harness engineering makes the same point from another angle: once agents are doing real work, progress depends heavily on the legibility of the environment, the structure of the feedback loops, and the extent to which knowledge has been made accessible and enforceable inside the system. A calculator tool is not merely a convenience. It is a decision about which parts of the method should be made stable and externalised. A verifier is not just a quality filter. It is a decision to make certain forms of checking structurally available to the agent. Even the shape of the prompt matters less as isolated wording than as one part of a larger control architecture.

This is why questions about orchestration in engineering cannot be reduced to prompt engineering. The meaningful design problem is architectural. What should the agent infer versus look up? What should be encoded in tools versus policy? Which checks should happen during reasoning and which should happen after? When should the system stop and ask for clarification instead of filling gaps with plausible narration? These are not implementation details to be cleaned up after capability arrives. They are part of how capability is built.

Once you see that clearly, a lot of standard debates start to look underspecified. Asking which model is best without asking what environment it is operating in is often the wrong question. In engineering, the more useful question is where the capability actually lives. Some of it lives in the model. Some of it lives in the tools, verifiers, and control policies around it. And some of it may only emerge when those pieces are composed in the right way. If that is true, then progress in this domain will depend not only on better models, but on better harness engineering for engineering.

If that is true, then progress in this domain will depend not only on better models, but on better harness engineering for engineering.

Open Questions for AI Agents in Engineering and AEC

This is where the real research agenda begins. If we are serious about agentic engineering, there are several questions that still look underexplored and experimentally open. They are not small implementation questions. They are structural questions about where competence comes from, how reliability is made, and what exactly we are trying to optimise for.

The first question is where domain knowledge should live. Some engineering method can be carried in the prompt, some in retrieved standards, some in dedicated tools, and some in verifier logic or decomposition policy. Those choices are not equivalent. A method embedded in prose instructions is available in a very different way from a method embedded in a tool or a structured reference. One of the central design problems in this domain is deciding which knowledge should remain internal to the model's reasoning and which should be stabilised outside it.

The second question is the unit of work. We still do not know the right granularity for engineering agents. Some tasks may be small enough to support reliable execution but too narrow to create useful leverage. Others may be large enough to matter operationally but so broad that the agent starts drifting, skipping checks, or losing attachment to the governing artifact. The choice is not between tiny tasks and ambitious autonomy as abstract ideals. It is about finding the span of work within which an agent can still remain controllable. And once the relevant unit turns out not to be a task but a process, questions of decomposition, review structure, and interaction design move from the margins to the centre.

The third question is what actually predicts reliability. Final scores matter, but they are not enough. A system that arrives at a good answer through a fragile process may be much less useful than a system that exposes when it is uncertain, revisits questionable steps, compresses toward clear findings, and behaves differently when it is in trouble. We do not yet know which trace-level signals are robust enough to support runtime monitoring or intervention, but that is exactly the kind of question that becomes important once you care about real workflows instead of static evaluation alone. Recent work on reasoning structure and agent behaviour is starting to point in this direction, treating intermediate behaviour as something analysable rather than just an opaque path to the final score, as in Du et al.'s Agentic Bonds framework.

The fourth question is how failure should be described. It is too coarse to talk about engineering failures as if they were all reasoning failures. In practice, the failure may be loss of task attachment, substitution of the instance, unsupported use of standards, malformed completion, or verbose but weakly grounded narration. Those are different breakdowns, and they imply different remedies. If we do not distinguish them, we will end up measuring systems in ways that hide the mechanisms we most need to improve.

The fifth question is how much capability a well-designed environment can supply. This is one of the most important and most delicate unknowns. A good harness can clearly unlock behaviour that would not reliably appear in a weak setup. But that raises a harder interpretive problem. When does scaffolding help a model express real competence, and when does it effectively perform too much of the task on the model's behalf? We need to know that boundary, because it affects how we read both benchmark results and deployment claims.

Each question changes what we should build, what we should measure, and how we should interpret success. Taken together, they suggest that the real frontier in agentic engineering is not just model improvement. It is experimental clarity about how capability is distributed across the whole system.

From Tasks to Processes: Why Engineering Workflows Need More Than Single-Step Agents

One reason this design space is easy to misunderstand is that we often talk about work as if it arrives in neatly bounded tasks. Sometimes it does. But many of the workflows that matter most in engineering are not really tasks in that sense. They are processes: longer-horizon structures with multiple artifacts, multiple experts, natural gates of review and control, and repeated moments where uncertainty has to be managed rather than ignored.

That distinction matters because a process creates a different design problem from a task. A task invites a question like: can the system complete this unit of work correctly? A process invites harder questions. How should work be decomposed across time? Where should evidence accumulate? When should one expert intervene directly, and when should the system continue on its own? What kind of intermediate state has to be visible for redirection, challenge, or sign-off to be meaningful? The moment you move from tasks to processes, orchestration stops being a thin implementation layer and becomes part of the substance of the work.

This is especially visible in workflows that look simple when described at a distance. A due diligence engagement can sound like one task: review the available material and produce a judgment. In practice it is nothing like a single bounded action. It is a process of evidence gathering, interpretation, cross-checking, escalation, synthesis, and review, often with several experts working in parallel and stepping in at different moments for different reasons. The important problem is not just whether an agent can perform one inference. It is whether the overall system can support the sequence of inferences, checks, and interventions that make the result trustworthy.

That is also why LLM UX becomes more important, not less, once agents enter the picture. If the real unit of work is a process, then the interaction pattern between humans and the system becomes part of the capability. We need to know what it means for an expert to drive the loop rather than merely appear in the loop at pre-specified checkpoints. We need to know how a system should request judgment, how it should expose uncertainty, and how it should allow redirection without forcing the human to reconstruct the entire state of the process from scratch.

This is not just a matter of convenience. It is a cognitive issue. These systems can already accumulate more intermediate state, more branching hypotheses, and more raw evidence than any single human can comfortably keep in working memory. So the question becomes: what is the control surface for a process like that? What is the communication layer that abstracts complexity without hiding the very evidence a reviewer may need to inspect? What kinds of summaries, provenance views, exception queues, and escalation mechanisms let a human meaningfully steer a process they cannot fully replay in their head?

Seen this way, agentic engineering is not only about building systems that can do tasks. It is about building systems that can participate in processes without dissolving the human capacity to understand, direct, and verify what is happening. That may be one of the deepest reasons harness design matters so much in engineering. The harness is not just coordinating tools around a model. It is helping define the structure through which work, judgment, and control move over time.

AEC Needs Better Benchmarks, Not Better Demos

If capability is environmentally expressed, then environment design has to become an empirical science rather than a collection of intuitions. In engineering, and especially in AEC, that point still has not been fully absorbed. Too much of the current conversation remains stuck at the level of demos, generic prompting claims, or isolated examples of model fluency. What the field needs instead is controlled experimentation on engineering task environments. Instead of asking whether a model can solve a cherry-picked problem, we need task families that let us vary conditions deliberately, observe what changes, and learn which combinations of tools, constraints, interfaces, and verification loops actually produce dependable work.

That starts with grounding. The tasks have to come from artifacts people actually use, and they have to reflect the problem spaces and value spaces that matter in practice. Every benchmark task is not just an isolated prompt. It is an instance of a broader problem class, and part of the job is identifying those classes deliberately rather than sampling whatever happens to be easy to score. From there, the environment itself has to become legible to experimentation: harness variants, tool access, verifier behaviour, output constraints, turn budgets, and other control variables need to be adjustable so we can see what is actually driving performance. And the measurements cannot stop at end scores. We need outcome metrics, but also trace-level analysis and explicit failure taxonomies that tell us what kind of process the system used and what kind of breakdown occurred when it failed.

The evaluation target also has to expand beyond isolated tasks. In many real settings, the important object is a process: a longer-horizon workflow with multiple artifacts, multiple actors, parallel workstreams, and repeated gates of review and redirection. The benchmark surface therefore cannot stop at correctness on a single step. It has to ask how control is handed off, how intermediate state is exposed, how uncertainty is surfaced, and how the system behaves when a human expert needs to redirect the work without redoing it from scratch.

That pushes evaluation toward a more realistic style. Engineering work is not purely textual, rarely single-step, and often only partly verifiable by one exact answer. Drawings, schedules, specifications, details, markups, diagrams, and spatial context are part of the task, not decorative extras. A serious research program in this space will need multimodal inputs much earlier than many current evals do. It will also need better machinery for tasks where adequacy, prioritisation, and review quality matter more than one final number.

It also raises a cognitive problem as well as a technical one. Once these systems are processing more state, more evidence, and more branching intermediate work than any human can comfortably hold in mind, the design challenge shifts again. What is the communication layer between the system and the expert? How should complexity be abstracted without hiding the very evidence a reviewer may need to see? What summaries, state representations, provenance trails, and escalation patterns let a human steer a process they cannot fully replay in their head? Those questions sit at the boundary between interface design, cognition, and control, and they are likely to matter just as much as the underlying model or toolchain.

The point is not to make benchmarks larger for their own sake. The point is to make them shaped enough like the work that they can teach us something real. If the benchmark is too convenient, it will mainly reward convenience. If the environment is too unlike practice, it will tell us very little about what systems can actually be trusted to do.

Reliability Before Autonomy: What AEC Firms Actually Need from AI Agents

The most useful near-term goal is probably not full engineering autonomy. It is reliable leverage on bounded tasks. Systems that can assist with scoped review, structured checking, discrepancy detection, and well-instrumented analysis may already create value long before end-to-end automation becomes plausible.

That is also a more useful way to think about autonomy itself. In practice, autonomy is not just a property of the model. It is an achievement of the system. A workflow becomes more autonomous when the surrounding environment makes the task legible, encodes the relevant constraints, exposes the right tools, and provides enough feedback for the agent to act with bounded independence rather than uncontrolled freedom.

In practice, autonomy is not just a property of the model. It is an achievement of the system.

That framing matters because it keeps the work tied to real workflow value instead of speculative theatre. Engineering organisations do not need a general claim that an agent is intelligent. They need to know whether a system can reduce time on a defined task while preserving reviewability and limiting failure cost. In many cases, the right near-term design is not autonomy but disciplined assistance inside a human-controlled process.

That is also why reliability has to come before ambition. A narrower system that stays attached to the artifact, exposes its work clearly, and fails in legible ways is often more valuable than a broader one that produces impressive but weakly controllable outputs. In engineering, trust is not a cosmetic feature added after the fact. It is part of the product. If a system cannot show why it acted, preserve the assumptions it relied on, and support meaningful intervention when things go wrong, then the path to autonomy runs through fragility rather than leverage.

This is especially important in domains like AEC, where the work is high consequence, low volume, and deeply review-shaped. The near-term question is not whether a system can replace expertise wholesale. It is whether it can make expert time more powerful without dissolving accountability. The prize is not a machine that replaces engineering judgment in the abstract. It is a system that can participate in engineering workflows in ways that are inspectable, recoverable, and worth trusting on defined slices of the job.

Why Harness Engineering for AEC Matters Now

This is the kind of work that matters if we want progress in engineering AI to be meaningful rather than theatrical. The glamorous parts of this field are easy to spot. The boring middle is not. But the boring middle is where much of the real leverage lives: task selection, workflow design, harness construction, interface design, verification, and evaluation. If we skip that layer, we do not get reliable systems. We get better demos.

If we skip that layer, we do not get reliable systems. We get better demos.

The interesting frontier is not another round of vague claims that models are getting smarter. It is the slower and more consequential work of identifying valuable problem spaces, building task environments around them, designing the right control surfaces, and learning how capability is distributed across models, tools, interfaces, and human judgment.

That is also what makes this an unusually exciting moment to work on these problems. The design space is still open. The operational patterns are not settled. The important domains are still underexplored. That means there is real room to shape the field: not just by building better models, but by building better harnesses, better evaluations, better interfaces, and better ways for experts and autonomous systems to work together on real tasks and real processes.

If engineering and AEC are going to benefit meaningfully from these systems, that work cannot be left to benchmark convenience or generic product abstractions. It will require people who understand the domains, the artifacts, the review practices, and the economics of real workflows to help define what good looks like. Before this becomes a field of grand claims, it should become a field of good experiments, good environments, and good judgment about where these systems can actually be trusted.