Beyond the coding agent: building a research harness from first principles

Codex, Claude Code, Pi, and the other harnesses in daily use are coding harnesses. Almost every feature in them exists to make one agent better at changing one repository: diff-based file editing, language-server lookups for definitions and references, lint and test loops that run after each edit, a git worktree per pull request, permission prompts around shell commands, and context compaction tuned to keep the current file in view. All of it assumes that the unit of work is an edit and that the verifier is a test suite that finishes in seconds.

The research workflow is different. It consists of hypotheses with experiments and validation attached, and the metrics usually come from backtests or inference on a validation dataset that can take hours. While the agent explores, dozens of hypotheses and experiments can be in flight at once, and each has to be logged, compared, and verified. Along the way, hypotheses branch and get promoted or dropped. Evidence accumulates and must stay retrievable later. An artifact means nothing without the code, data, and environment that produced it. To this end, we built Oh My Quant, a research-native harness, for our own quant workflow.

The result

To benchmark the harness we used OpenAI's MLE-bench Lite, a set of Kaggle competitions and the closest public benchmark we know of to machine learning research work. For each competition the agent receives a task description, a dataset, a time budget, and a compute budget, and has to produce a submission for grading. We ran Oh My Quant on it with Claude Opus 4.6, one agent per competition, ten hours of wall clock (the benchmark's canonical budget is 24 hours), and one GPU each. We first reported the result on X.

It medalled in 80.95% of the competitions. At the time of writing that is above every entry on the public leaderboard, including multi-agent systems that spend their run on tree search over candidate solutions, on evolutionary populations, or on a team of specialised agents. We chose Opus 4.6 over newer frontier models because it already appears on the leaderboard. That holds the model fixed and leaves the harness as the variable, which makes the comparison fairer.

Pass rate (any medal) on the MLE-bench Lite split. Leaderboard entries, models, and budgets are as published. Oh My Quant had 10 hours per competition against the others' 24.

Comparison with existing agentic research systems

The MLE-bench leaderboard has many other agentic research systems on it. Most of them put the model inside a search procedure written by the system's authors, which Oh My Quant deliberately avoids. Famou-Agent 2.0 runs an evolutionary search over islands of candidate solutions. MARS+ and ML-Master 2.0 plan with Monte Carlo tree search. Kapso runs a tree search over experiments. InternAgent is a team of specialised agents, and Operand is an ensemble of models. In each case the harness decides which roles run in what order, how many candidates are evaluated, how the budget is split between exploring and exploiting, and when the search stops.

Oh My Quant fixes none of that. It runs only one agent, one model, and one generic system prompt, with no fixed roles, workflows, or delegation. The agent chooses where to look, what to try, and in what order.

Design principles

A harness sits between the model and the world. It decides what the model sees, what it may do, what runs where, and what gets written down. Every design decision in Oh My Quant follows from four principles:

Principle 1: Let the model own judgment; let the harness enforce invariants. Anything that needs intelligence goes to the model: what to investigate, what a result means, what to try next. Anything deterministic, strict, or invariant goes to the runtime. A model that misreads a log should not be able to corrupt the record, and a runtime should never decide mechanically whether a hypothesis is worth pursuing.

Principle 2: Let progress in model intelligence be our tailwind. The research agent gets the most freedom we can give it: no fixed workflow, no fixed roles, no fixed acceptance criteria written by the harness. Every piece of procedure hard-coded into a harness is a bet that the model cannot be trusted with that decision, and that bet loses value with each model release. We would rather the harness get better for free when the model does.

Principle 3: Make exploration and exploitation cheap and recoverable. Running, monitoring, and retrieving experiments in parallel should cost the research agent as little as possible, and auditing or reproducing a result later should cost almost nothing. Anything slow or tedious corrupts the research agent's context and interrupts the research, which means fewer experiments, less evidence, and lower research quality.

Principle 4: Make evidence durable and conclusions traceable. Every hypothesis, decision, and experiment must be recoverable months later, together with what it rested on and what it changed. Research objects relate to each other semantically: this experiment tested that hypothesis, and that code change produced this artifact. A plain log cannot hold those relations, so the tracking layer needs graph semantics.

Implementation

The components below are how these principles are implemented in Oh My Quant. Each begins with the principles it serves.

Overview of the harness. The research memory on the left holds the graph and the evidence history, the experiment system on the right produces sealed evidence and baseline decisions, and the agents in the middle read from one and drive the other.

Forkable research workers with retained experiments

Principles 1, 3, 4

The harness lets the research agent spawn children that run concurrently, message each other, return reports in a predefined format, and persist across sessions in their own isolated checkouts. Every agent and subagent doing parallel research gets its own workspace, spawned and tracked by the harness runtime: it snapshots the working tree, gives each worker a private checkout of it, and spawns a sandbox if needed. Persistent subagents can finish an assignment, park, and continue later when the main research agent sends them a new one. The runtime also handles the handoffs between agents, so whoever picks up the research thread sees exactly what was run and what the outputs were. That makes parallel research comparable and recoverable.

Every run the runtime launches also gets its own TensorBoard directory. The plots are visible to the agent that owns the run, to reviewers and critics, and to the human in a live viewer, so anyone can inspect a training process while it runs.

Retrievable record of every step and decision

Principles 1, 3, 4

Everything the harness executes, from a bash command to a cloud training job, ends in a receipt: what ran, where, against which commit, with what exit status, for how long, and where its output artifacts are stored. The runtime writes and seals the receipt, then hands the agent a receipt id.

Receipts are queryable by both agents and humans. Once it has the id, the agent can refer to any executed command or tool call by its receipt id. When it records an experiment, it names receipts, and the runtime resolves them into the code changes, the runs, and the archived artifacts. Because the harness owns the receipt system, a model that misreads a log or a number cannot corrupt the record.

A semantic research tracking graph

Principles 1, 2, 4

Alongside the receipts, the harness keeps a research tracking graph whose nodes are research objects: hypotheses, acceptance criteria, experiments, code changes, artifacts, and references. Relationships between objects are named edges. Research objects are often the artifacts of executed commands that already have a receipt id, and in those cases the two tracking systems are linked, so each can be queried from the other.

The graph fixes no sequence of agent actions. It can record that a hypothesis now has two experiments against it and one for it, and that one of them ran on a code change that has since been deprecated. That lets hypotheses branch, accumulate contradictory evidence, and be revised later by adding new records and relationships. The agent supplies the semantic judgments, deciding what counts as support and what counts as a narrowing of the question, and the tooling handles the records and the links. The agent deliberately has no tool to delete or overwrite a node, so every failed experiment and wrong conclusion stays visible and traceable.

Subagents as function calls: one-shot agents for semantic retrieval

Principles 1, 2, 3, 4

A research graph that spans months is too large to hand to the research agent as context. Keyword or syntax-level search is easy for the harness to provide, but semantic search over subtle research questions needs some intelligence and an understanding of the research itself.

As models get stronger, it becomes easier to use a subagent as a function call for well-defined tasks that still need intelligence. In our case, semantic retrieval over the research graph is such a call. The research agent states an information need and the decision it is trying to make; the runtime spawns a one-shot retrieval agent with read-only tools, the relevant context, and a fixed output schema. That retrieval agent chooses the query formulations, decides which subgraph to traverse, opens artifacts when the summary is not enough, ranks what it found by relevance to the stated decision, and returns the answer like a function result. The runtime validates the output against the schema and asks for a rework when it does not conform.

A harness-owned critic agent

Principle 1

The critic compensates for a weakness in today's models. Over a long enough run, even a frontier model tends to get stuck in a loop: it fixates on a small detail, keeps adjusting it, and loses the ability to see the bigger picture or to pull itself out.

Agents are poor judges of their own trajectories. Nothing in a long transcript prompts the question of whether the last hour moved the target, and asking the agent to step back does not work once its context window is polluted: it steps back into the same framing it was already in.

So the harness runtime steers from time to time. On a fixed schedule it spawns a critic: a fresh agent with no memory of the session, given only the context it needs for the job. The critic's sole job is to read the recent research records and session traces and decide whether the research agent is stuck. It writes a memo on what it sees, and the runtime drops that memo into the research agent's inbox before the next cycle. The critic makes one strategic call, without prescribing code, parameters, or schedules, and stops.

Two cases from our runs show what this looks like in practice. In one, the agent kept circling between several architecture baselines, unable to pick one. The critic pointed out that the CNN baseline was undertrained, so the comparison the agent was stuck on was not a fair one. In another, the agent had spent cycle after cycle adjusting the learning rate of a boosting algorithm. The critic told it to leave the learning rate alone and try a different model family, which could add more value later through ensembling.