Overview

  • Autoresearch is the practice of turning experimental research into an agent-run optimization loop: an AI coding agent proposes a change, edits an executable research artifact, runs a bounded experiment, reads the metric, keeps or reverts the change, and repeats until the search budget is exhausted. In the concrete single-GPU setting, karpathy/autoresearch gives the agent a compact LLM training setup, lets it edit train.py, runs fixed 5-minute training experiments, evaluates val_bpb, and treats lower validation bits per byte as the target metric.

  • The following figure (source) shows an autoresearch progress curve from the repository, where sequential fixed-budget experiments are evaluated by validation bits per byte and the running best result improves over time.

What autoresearch automates

  • Autoresearch automates the outer loop of empirical ML work rather than only the inner loop of gradient-based training. The inner loop still optimizes model weights using ordinary training, typically next-token cross-entropy for language modeling:
\[\mathcal{L}_{\mathrm{CE}}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \log p_{\theta}(x_t \mid x_{<t})\]

  • The outer loop optimizes the research program around that training run: architecture choices, optimizer settings, batch sizes, attention patterns, model depth, learning-rate schedules, tokenizer and sequence-length tradeoffs, and other code-level decisions. In the single-GPU implementation, the repository is intentionally small: prepare.py handles fixed constants, data preparation, tokenizer training, dataloading, and evaluation; train.py contains the GPT model, optimizer, and training loop that the agent edits; and program.md acts as the lightweight instruction layer that defines how the agent should behave.

  • The metric used in this setup is validation bits per byte, which is useful because it is less tied to a particular vocabulary size than raw token loss. A typical conversion from average token negative log-likelihood in nats to bits per byte is:

    \[\mathrm{BPB} = \frac{\mathcal{L}_{\mathrm{nats}}}{\ln 2} \cdot \frac{N_{\mathrm{tokens}}}{N_{\mathrm{bytes}}}\]
    • where lower values indicate better compression of held-out text. This makes it suitable for comparing architecture and tokenizer changes inside a bounded training budget, as long as all candidates are evaluated on the same validation data and hardware regime.

The core loop

  • At a high level, autoresearch can be written as a propose, run, evaluate, decide loop:

    \[c_{t+1} \sim P(c \mid \mathcal{H}_t, I)\] \[m_{t+1} = \mathrm{Evaluate}(c_{t+1}; B)\] \[\mathcal{H}_{t+1} = \mathcal{H}_{t} \cup \{(c_{t+1}, m_{t+1}, \ell_{t+1})\}\]
    • where, \(c_t\) is a code candidate, \(P\) is the coding-agent proposer, \(I\) is the instruction layer, \(B\) is the fixed experimental budget, \(m_t\) is the measured score, \(\ell_t\) is the log of what happened, and \(\mathcal{H}_t\) is the accumulated history. The important design choice is that the search unit is executable code rather than a prompt string alone. This puts autoresearch closer to program search and harness engineering than to ordinary prompt optimization.

  • This is why Automatic Prompt Optimization with “Gradient Descent” and Beam Search by Pryzant et al. (2023) is relevant as background because it frames prompt revision as text-space optimization, but autoresearch generalizes the optimized artifact from text instructions to runnable ML code. TextGrad by Yuksekgonul et al. (2024) is also relevant because it treats natural-language feedback as an optimization signal for compound AI systems, but autoresearch uses full code execution and validation metrics as the main feedback channel.

Why fixed budgets matter

  • A fixed wall-clock training budget is central to autoresearch because it prevents the agent from “improving” results merely by training longer. In the single-GPU setup, every candidate gets the same 5-minute training window excluding startup and compilation, which makes changes to depth, batch size, model width, optimizer, attention pattern, and data throughput comparable under the same resource constraint.

  • This turns the objective from “find the best model eventually” into “find the best model under this compute envelope”:

    \[c^{*} = \arg\min_{c \in \mathcal{C}} \mathrm{BPB} \left( \mathrm{Train}(c, B) \right)\]
    • where \(\mathcal{C}\) is the space of valid code candidates and \(B\) is the bounded training budget. The resulting search naturally favors changes that improve learning speed, data efficiency, numerical stability, and hardware utilization, not just asymptotic quality.

How Meta-Harness fits into autoresearch

  • Meta-Harness is the natural generalization of autoresearch from “optimize a training file” to “optimize the harness around an LLM system.” In Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026), a harness is the code that decides what information to store, retrieve, and present to a fixed language model; the search procedure uses a coding-agent proposer that inspects prior source code, scores, and execution traces through a filesystem before proposing new harness code.

  • The following figure (source) shows the Meta-Harness search loop: an agent reads a filesystem containing prior candidates’ source code, execution traces, and scores; proposes a new harness; evaluates it on tasks; stores the proposed code, reasoning traces, and evaluation score back into the filesystem; and repeats.

  • The formal harness objective is:

    \[H^{*} = \arg\max_{H} \mathbb{E}_{x \sim X,\ \tau \sim p_M(H,x)} \left[ r(\tau, x) \right]\]
    • where \(M\) is the frozen base model, \(X\) is the task distribution, \(H\) is the harness, \(\tau\) is the rollout trajectory induced by running the model inside the harness, and \(r(\tau, x)\) is the task reward. This differs from ordinary model training because the weights of \(M\) may remain fixed while the outer-loop system searches over code that controls memory, retrieval, prompt construction, tool use, and state updates.

The key conceptual shift

  • The key shift is from optimizing parameters to optimizing research process. Traditional ML optimization changes \(\theta\), the model weights. Autoresearch changes \(c\), the code that defines the experiment. Meta-Harness changes \(H\), the code that defines the model’s operating environment. These layers can be viewed as nested optimization problems:
\[\theta^{*}(c) = \arg\min_{\theta} \mathcal{L}_{\mathrm{train}}(\theta; c)\] \[c^{*} = \arg\min_{c} \mathcal{L}_{\mathrm{val}}(\theta^{*}(c); c, B)\] \[H^{*} = \arg\max_{H} \mathbb{E} \left[ r(\tau, x) \right]\]

Why logs, traces, and filesystem access are first-class

  • Autoresearch becomes more powerful when the agent can inspect not just the final scalar metric, but also the code diff, console output, training curves, validation breakdowns, failure logs, and previous decisions. Meta-Harness makes this explicit by storing each candidate’s source code, scores, prompts, tool calls, model outputs, and state updates in a filesystem that the proposer can query with ordinary tools such as grep and cat, instead of compressing all feedback into a short summary.

  • This matters because code-level failures are often nonlocal. A candidate may degrade validation loss because a batch-size change destabilized the optimizer, because an attention-window change reduced throughput, because a data-loading change shifted token statistics, or because an apparently harmless prompt or harness edit changed the model’s downstream behavior many steps later. Scalar scores tell the agent what happened; traces help it infer why it happened.

Working definition

  • For the rest of this primer, autoresearch means an autonomous, metric-driven, code-editing research loop with five components: a bounded experimental sandbox, an editable artifact, a proposer agent, an evaluator, and a durable memory of attempts. The single-GPU version uses train.py as the editable artifact and val_bpb as the metric; Meta-Harness uses task-specific harness code as the editable artifact and task reward, accuracy, pass rate, or Pareto tradeoffs as the metric. Both are instances of the same deeper pattern: use agents to search over the process that produces model behavior, not only over the model output itself.

System architecture

  • Autoresearch systems are best understood as experimental operating systems for AI research: they define who can change what, how changes are evaluated, how results are logged, and how future proposals condition on past attempts. The minimal version has one editable research artifact, one evaluator, one metric, one instruction file, and one persistent experiment log; richer versions generalize the editable artifact from model-training code to full harness code that controls prompting, retrieval, memory, tools, and state.

Editable artifact

  • The editable artifact is the unit of search. In a compact LLM-training setup, the editable artifact is usually a single file containing the model definition, optimizer, and training loop, while data preparation and evaluation utilities remain fixed to keep the search space bounded and diffs reviewable. This design makes the agent’s changes easy to inspect and makes regression analysis practical because each candidate can be understood as a concrete code diff rather than a hidden policy update.

  • A useful rule is to separate code into three strata:

    • Frozen substrate: data download, tokenizer construction, dataloader correctness, validation-set construction, deterministic metric computation, hardware setup, and safety checks.
    • Editable research code: architecture, optimizer, batch sizing, schedules, attention patterns, normalization, loss variants, numerical precision choices, and training-loop structure.
    • Instruction layer: the agent’s research policy, constraints, experiment protocol, logging requirements, and decision rules.

  • This separation resembles the distinction between an LM program and its optimizer in DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines by Khattab et al. (2023), where the developer specifies a pipeline and the system compiles or tunes parts of it rather than treating every prompt as a one-off string.

Evaluator

  • The evaluator is the part of the system that turns an edited artifact into a comparable score. In the basic training setup, the evaluator runs a bounded training job and reports validation bits per byte, with lower values being better. A fixed time budget, such as 5 minutes per experiment, makes candidates comparable because each proposal must improve learning speed, hardware utilization, or modeling efficiency under the same wall-clock constraint.

  • A typical evaluator should return a structured record:

candidate_id
parent_candidate_id
git_diff_or_full_source
start_time
end_time
hardware_metadata
training_tokens
validation_loss
validation_bpb
throughput_tokens_per_second
peak_memory
nan_or_crash_flag
stderr_stdout_excerpt
full_log_path
decision
  • For language modeling, the main inner-loop objective is still next-token prediction:
\[\mathcal{L}_{\mathrm{LM}}(\theta) = -\frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T_i} \log p_{\theta}(x_{i,t} \mid x_{i,<t})\]

  • The autoresearch evaluator does not optimize this loss directly; it measures the result of a bounded training run induced by a code candidate. The outer-loop score can be written as:

    \[s(c) = \mathrm{BPB} \left( \mathrm{Train}(\theta_0, c, B) \right)\]
    • where \(c\) is the code candidate, \(\theta_0\) is the initialization procedure, and \(B\) is the fixed compute budget.

Proposer agent

  • The proposer is a coding agent that reads the current codebase, inspects past results, forms a hypothesis, edits code, runs tests or experiments, and records what happened. This is more powerful than a raw LLM prompt because the proposer can use shell tools, inspect files, execute code, revert changes, and diagnose failures through logs.

  • The proposer can be treated as a stochastic policy over code edits:

    \[c_{t+1} \sim \pi_{\phi} \left( c \mid I, \mathcal{D}_t, \mathcal{R}_t \right)\]
    • where \(I\) is the instruction layer, \(\mathcal{D}_t\) is the codebase and experiment history, and \(\mathcal{R}_t\) is the current research objective. This connects autoresearch to Evolution through Large Models by Lehman et al. (2022), which shows that code-generating language models can act as intelligent mutation operators over programs, and to AlphaEvolve by Novikov et al. (2025), which uses coding agents and automated evaluators to evolve algorithms through direct code changes.

Instruction layer

  • The instruction layer is the “research org code.” It tells the agent what success means, which files are editable, how long each experiment should run, how to log hypotheses, how to handle crashes, and when to keep or discard a change. In a small system, this can be a Markdown file that acts like a lightweight skill; in larger systems, it can include role-specific instructions, experiment queues, coding standards, ablation policies, and escalation rules.

  • A strong instruction layer should include:

    • Objective: optimize validation BPB, pass rate, accuracy, reward, latency-adjusted quality, or a Pareto frontier.
    • Search constraints: editable files, forbidden files, maximum experiment duration, maximum memory, and allowed dependencies.
    • Scientific hygiene: one primary hypothesis per experiment, record the diff, record why the change was tried, and record why it was kept or reverted.
    • Failure handling: immediately revert NaN-producing candidates, distinguish crash from metric regression, and preserve crash logs.
    • Exploration policy: alternate local improvements with occasional larger rewrites, but avoid stacking many untested changes at once.
    • Reporting policy: append each experiment to a durable table, including candidate ID, score, parent, diff summary, and notes.

  • The idea that the system can improve by editing the agent design itself is closely related to Automated Design of Agentic Systems by Hu et al. (2025), which frames agent design as a search problem over executable agentic systems rather than a purely manual engineering task.

Durable memory and experiment history

  • A durable memory turns a sequence of isolated attempts into cumulative research. At minimum, the system should store source code, diffs, scalar metrics, logs, and human-readable notes for every candidate. In a more capable setup, the proposer can query the full filesystem of past attempts using tools such as grep, cat, notebooks, plots, and structured result files.

  • This is the central architectural insight behind Meta-Harness: the proposer should not only see compressed summaries or scalar scores; it should be able to selectively inspect raw prior code, scores, prompts, tool calls, model outputs, state updates, and execution traces. The filesystem can be much larger than the proposer’s context window, so the agent retrieves only what it needs at each iteration rather than packing all history into one prompt.

  • A useful directory layout is:

runs/
  000_seed/
    source/
    metrics.json
    stdout.log
    stderr.log
    trace.jsonl
    notes.md
  001_depth6_lr3e-4/
    source/
    diff.patch
    metrics.json
    stdout.log
    stderr.log
    trace.jsonl
    notes.md
leaderboard.tsv
frontier.json
current_best/
  • This memory design makes the system closer to an empirical scientist than a blind optimizer. It can notice that several high-depth candidates improved early loss but crashed late, that a throughput improvement came from reduced sequence length rather than better modeling, or that a prompt-harness change improved easy cases but hurt rare classes.

Search controller

  • The search controller decides how candidates are generated and evaluated. The simplest controller is greedy hill climbing: accept a candidate only if it improves the metric. A more robust controller maintains a population of candidates and a Pareto frontier over multiple objectives such as quality, context length, runtime, and memory.

  • For a single scalar metric where lower is better:

\[c_{\mathrm{best},t} = \arg\min_{c_i \in \{c_1,\dots,c_t\}} s(c_i)\]

  • For multiple metrics, candidate \(a\) Pareto-dominates candidate \(b\) if:

    \[\forall j,\ f_j(a) \le f_j(b) \quad \text{and} \quad \exists k,\ f_k(a) < f_k(b)\]
    • where each \(f_j\) is a cost-like metric such as validation BPB, latency, memory, or context tokens. Pareto tracking is especially useful when optimizing harnesses because a slightly less accurate harness may be preferable if it uses far fewer tokens, fewer model calls, or less wall-clock time. Meta-Harness explicitly maintains a population and Pareto frontier while leaving parent selection flexible: the proposer can inspect any prior harness and its traces rather than being constrained to a fixed evolutionary parent rule.

Validation gates

  • Validation gates protect the search from wasting compute on invalid candidates. A basic autoresearch system should perform static and dynamic checks before running the full experiment:
format check
import check
unit smoke test
short forward/backward pass
short validation pass
NaN/Inf guard
memory estimate
full bounded experiment
  • For harness search, the analogous gates are interface validation, tool-call validation, prompt-shape validation, output-parser validation, and budget validation. In Meta-Harness, proposed harnesses are evaluated only after passing interface validation, and each evaluated candidate contributes code, scores, and traces back into the filesystem.

Implementation skeleton

  • A practical minimal loop looks like this:
def autoresearch_loop(seed_code, proposer, evaluator, budget, max_iters):
    history = []
    best = seed_code

    for t in range(max_iters):
        context = build_research_context(history, best)
        candidate = proposer.propose(context)

        valid, validation_report = validate_candidate(candidate)
        if not valid:
            history.append({
                "candidate": candidate,
                "status": "invalid",
                "report": validation_report,
            })
            continue

        result = evaluator.run(candidate, budget=budget)
        history.append({
            "candidate": candidate,
            "status": "evaluated",
            "metrics": result.metrics,
            "logs": result.logs,
            "diff": result.diff,
        })

        if result.metrics["val_bpb"] < score(best):
            best = candidate

        write_history_to_disk(history)

    return best, history
  • A Meta-Harness-style version changes the optimized artifact from training code to a model harness:
def meta_harness_loop(seed_harnesses, proposer, task_set, model, max_iters):
    filesystem = ExperimentFilesystem()
    population = list(seed_harnesses)

    for harness in population:
        result = evaluate_harness(harness, model, task_set)
        filesystem.store(harness=harness, result=result)

    for t in range(max_iters):
        proposal_context = filesystem.path
        new_harnesses = proposer.propose_harnesses(proposal_context)

        for harness in new_harnesses:
            if not passes_interface_validation(harness):
                filesystem.store_invalid(harness)
                continue

            result = evaluate_harness(harness, model, task_set)
            filesystem.store(harness=harness, result=result)
            population.append(harness)

    return compute_pareto_frontier(filesystem.results)
  • The key implementation detail is that proposal_context should be a navigable filesystem, not a giant serialized prompt. This lets the proposer decide whether to inspect the best runs, the worst regressions, the most recent diffs, specific failure traces, or raw code from older candidates.

Metrics table as the shared interface

  • A durable metrics table is the shared interface between the evaluator, proposer, and human reviewer. It should be append-only and machine-readable. For LLM training, useful columns include:
run_id
parent_id
status
val_bpb
train_loss
tokens_per_second
num_parameters
depth
max_seq_len
device_batch_size
total_batch_size
optimizer
learning_rate
peak_memory_gb
wall_time_seconds
notes
  • For harness optimization, useful columns include:
run_id
parent_id
status
accuracy
pass_rate
reward
context_tokens
num_model_calls
latency_seconds
tool_calls
crash_rate
parse_error_rate
trace_dir
notes
  • The system should never rely only on the final metric. A candidate with slightly better BPB but much worse throughput may not be a true improvement under longer budgets. A harness with higher accuracy but twice the context cost may be worse at deployment scale. This is why Pareto tracking is often a better default than a single leaderboard.

Failure modes designed into the architecture

  • Autoresearch systems need architecture-level defenses because agents will otherwise exploit ambiguity in the objective. Common failure modes include metric hacking, accidental test leakage, overfitting to a tiny validation set, increasing runtime while appearing to improve quality, silently changing evaluation code, and producing changes that work only on the current hardware. Keeping data preparation and evaluation frozen reduces these risks, while logging full diffs and traces makes suspicious improvements inspectable.

  • For harnesses, the analogous risks are prompt overfitting, hard-coded labels, benchmark-specific if-statements, brittle parsers, leakage from previous test results, and inflated context use. Code-space search has one practical advantage here: brittle shortcuts are often visible in the source code, unlike weight-space overfitting, where the failure may be hidden inside parameters.

Running the research loop

  • The research loop is where autoresearch stops being a collection of scripts and becomes an autonomous experimental process. A good loop is not just “let an agent edit code.” It is a disciplined cycle of hypothesis formation, constrained editing, validation, measurement, memory update, and selection.

The experiment lifecycle

  • Each experiment should begin with a concrete hypothesis, not just a code change. A hypothesis can be local, such as “reducing depth while increasing batch size may improve validation BPB within the fixed wall-clock budget,” or structural, such as “changing the attention pattern may improve throughput enough to outweigh a small modeling-quality loss.” The agent should then make the smallest coherent edit that tests the hypothesis, run validation gates, launch the bounded experiment, and write a postmortem.

  • A practical lifecycle is:

    • Plan: inspect the leaderboard, current best code, and recent regressions.
    • Hypothesize: state the mechanism expected to improve the metric.
    • Edit: modify only the allowed artifact.
    • Validate: run static checks, smoke tests, and NaN guards.
    • Evaluate: run the full bounded experiment.
    • Record: save source, diff, logs, metrics, and notes.
    • Select: keep, revert, branch, or mark for follow-up.

  • This differs from Self-Refine by Madaan et al. (2023), which iteratively improves model outputs using self-feedback, because autoresearch applies the refinement loop to executable research code and evaluates the result through external experiments rather than only through textual critique.

Candidate generation

  • Candidate generation should balance exploitative local edits with occasional exploratory changes. Purely local search often gets stuck refining the current best configuration, while unconstrained exploration wastes compute on invalid or noisy candidates. A good proposer should alternate among several edit families:

    • Hyperparameter edits: learning rate, warmup, weight decay, batch size, dropout, gradient clipping, optimizer betas, and scheduler shape.
    • Architecture edits: depth, width, MLP ratio, normalization, attention pattern, residual scaling, positional encoding, and parameter tying.
    • Training-loop edits: mixed precision, compilation, gradient accumulation, evaluation cadence, data packing, loss scaling, and optimizer step ordering.
    • Efficiency edits: fused operations, memory layout, reduced overhead, activation checkpointing, and batch-shape tuning.
    • Robustness edits: NaN checks, safer initialization, fallback paths, and clearer logging.

  • The outer-loop proposal distribution can be written as a mixture:

    \[\pi(c_{t+1} \mid \mathcal{H}_t) = \sum_{k=1}^{K} \alpha_k \pi_k(c_{t+1} \mid \mathcal{H}_t)\]
    • where each \(\pi_k\) is an edit family and \(\alpha_k\) is the current probability of selecting that family. In practice, the agent can adapt \(\alpha_k\) from history: if optimizer edits keep causing instability, reduce their frequency; if attention-pattern edits repeatedly improve throughput without hurting BPB, inspect them more deeply.

  • This view is close to Evolution through Large Models by Lehman et al. (2022), which treats language models as mutation operators for program search, but autoresearch typically uses a stronger experimental memory and a tighter evaluation budget.

Evaluation under a fixed budget

  • A fixed budget turns every proposal into a resource-constrained optimization problem. The score is not “best validation loss eventually,” but “best validation loss after the allowed compute.” For language modeling, the evaluator should measure at least:
\[\mathrm{val\_bpb}\] \[\mathrm{tokens/sec} = \frac{N_{\mathrm{train\ tokens}}}{T_{\mathrm{wall}}}\] \[\mathrm{effective\ improvement} = \mathrm{BPB}_{\mathrm{parent}} - \mathrm{BPB}_{\mathrm{candidate}}\]

  • A candidate that improves BPB by making the model much larger may still be undesirable if it only wins because it accidentally receives more effective training tokens, changes the validation path, or makes evaluation inconsistent. The fixed time budget protects against some of this, but the system should still log throughput, train tokens, parameter count, memory usage, and crash status.

  • A useful scalarized score for early triage is:

    \[J(c) = \mathrm{BPB}(c) + \lambda_1 \cdot \max(0, M(c) - M_{\max}) + \lambda_2 \cdot \mathbb{1}[\mathrm{crash}(c)] + \lambda_3 \cdot \mathbb{1}[\mathrm{nan}(c)]\]
    • where \(M(c)\) is peak memory, \(M_{\max}\) is the allowed memory threshold, and the indicator penalties prevent invalid runs from being ranked as promising.

Selection and acceptance

  • The simplest acceptance rule is greedy improvement:

    \[\mathrm{accept}(c_{t+1}) = \mathbb{1} \left[ s(c_{t+1}) < s(c_{\mathrm{best}}) \right]\]
    • where lower \(s\) is better. Greedy acceptance is easy to audit, but it can be too conservative. Some candidates may be worse overall while containing a useful sub-change, such as faster data loading or a more stable initialization. A better system stores all candidates, keeps the current best as the deployment baseline, and allows future proposals to branch from any prior run.

  • For noisy metrics, use a margin:

    \[\mathrm{accept}(c_{t+1}) = \mathbb{1} \left[ s(c_{t+1}) < s(c_{\mathrm{best}}) - \epsilon \right]\]
    • where \(\epsilon\) is a practical significance threshold estimated from repeated baseline runs. Without this threshold, the agent may overfit to random measurement variation.

Branching and population management

  • Autoresearch should not be a single linear chain unless the experiment cost is extremely high. A population gives the agent multiple promising lineages to revisit. Each candidate can be tagged by its parent, edit family, and outcome:
run_id    parent    family          val_bpb    status
000       none      seed            0.9978     kept
001       000       optimizer       0.9905     kept
002       001       depth           0.9874     kept
003       002       attention       0.9846     kept
004       003       batch-size      crash      rejected
005       002       scheduler       0.9839     kept

  • A population-based controller can sample parents according to a softmax over score:

    \[P(\mathrm{parent}=i) = \frac{ \exp(-s_i / \tau) }{ \sum_j \exp(-s_j / \tau) }\]
    • where \(\tau\) controls exploration. Smaller \(\tau\) focuses on the current best; larger \(\tau\) lets the agent revisit diverse candidates.

  • This is related to GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning by Agrawal et al. (2025), which maintains a Pareto-aware reflective optimization process over prompts, but autoresearch extends the same evolutionary pressure to executable code and experimental systems.

Trace-driven diagnosis

  • Scalar metrics are too compressed for serious autonomous research. A useful loop gives the agent access to full traces: code diffs, stdout, stderr, validation curves, throughput, memory, seed, hardware metadata, and notes. The agent should be able to answer questions such as:

    • Did the candidate improve because it trained more tokens per second?
    • Did it improve early but plateau worse?
    • Did it reduce BPB by changing tokenizer or validation behavior?
    • Did it crash only after evaluation, during compilation, or during backward pass?
    • Did a worse candidate contain a subcomponent worth transplanting?

  • The Meta-Harness pattern makes this trace-driven workflow explicit: the proposer reads prior code, scores, and execution traces from a filesystem, chooses what to inspect, and uses that evidence to propose the next harness rather than relying on a fixed summary. The same design should be used for autoresearch on training code.

Handling crashes and invalid candidates

  • Crashes are data. They should not disappear from the history because they teach the agent which regions of the search space are unstable. Each failed candidate should be classified:
syntax_error
import_error
shape_error
out_of_memory
nan_loss
timeout
metric_missing
evaluation_changed
external_dependency_failure

  • For each class, the loop should have a default policy. Syntax and import errors should trigger immediate repair or rejection. Out-of-memory candidates should be logged with peak memory and likely cause. NaN-producing candidates should be rejected unless the purpose of the next run is explicitly to stabilize them. Timeout candidates should record partial progress but should not be compared directly against full-budget runs.

  • A strong NaN guard looks like:

if not torch.isfinite(loss):
    raise RuntimeError(f"Non-finite loss at step {step}: {loss.item()}")
  • A strong metric guard checks that the result file exists, has the expected schema, and was produced by the current run ID. This prevents the agent from accidentally reusing stale scores.

Preventing evaluation drift

  • Evaluation drift is one of the most dangerous failure modes. If the agent can edit the validation data, metric computation, tokenizer accounting, or result parser, it may accidentally or deliberately make the score easier. The frozen substrate should therefore include:

    • validation split construction
    • evaluation-token selection
    • BPB computation
    • result serialization
    • run-time enforcement
    • comparison script
    • data download and preprocessing

  • For extra safety, hash the evaluator files before each experiment:

\[h_t = \mathrm{SHA256} \left( \mathrm{prepare.py} \parallel \mathrm{eval.py} \parallel \mathrm{validation\_manifest} \right)\]
  • Then reject candidates where \(h_t \ne h_0\) unless a human explicitly approved an evaluator migration.

Multi-objective research

  • Autoresearch becomes more useful when it optimizes multiple metrics instead of a single leaderboard number. In LLM training, relevant objectives include BPB, throughput, memory, parameter count, and code complexity. In harness optimization, relevant objectives include task score, context tokens, number of model calls, latency, parse-error rate, and tool-call cost.

  • A candidate belongs on the Pareto frontier if no other candidate is at least as good on every objective and strictly better on one:

\[c_i \in \mathcal{P} \iff \nexists c_j: \left( \forall k,\ f_k(c_j) \le f_k(c_i) \right) \land \left( \exists k,\ f_k(c_j) < f_k(c_i) \right)\]
  • This is especially important for harnesses because a high-accuracy system that uses enormous context may be less valuable than a slightly lower-accuracy system that is cheaper, faster, and more robust. Meta-Harness uses this kind of Pareto framing when multiple objectives such as accuracy and context cost matter.

Research notes as training data for the next loop

  • Every experiment should produce a short note in a consistent format:
Hypothesis:
Changing X should improve Y because Z.

Change:
Modified A, B, and C.

Result:
val_bpb changed from 0.9846 to 0.9839.
Throughput changed from 410k tok/s to 398k tok/s.
No NaNs or crashes.

Interpretation:
The improvement is real but small. The slower throughput suggests the modeling change helped more than the efficiency cost hurt.

Next:
Try the same architecture with the previous faster batch schedule.
  • These notes matter because they give the proposer a compact human-readable layer on top of raw logs. They should not replace raw traces, but they make the history easier to navigate. TextGrad by Yuksekgonul et al. (2024) is relevant because it shows how textual feedback can serve as an optimization signal for AI systems, but in autoresearch that feedback should be paired with executable traces and external metrics.

A robust loop template

  • A production-grade autoresearch loop should look like this:
def run_one_experiment(candidate, parent, budget):
    run_id = allocate_run_id()

    save_source_snapshot(run_id, candidate)
    save_diff(run_id, parent, candidate)

    validation = validate_candidate(candidate)
    save_json(run_id, "validation.json", validation)

    if not validation["ok"]:
        save_status(run_id, "invalid")
        return {"run_id": run_id, "status": "invalid"}

    result = launch_training(
        source=candidate,
        wall_clock_budget=budget,
        run_id=run_id,
    )

    metrics = parse_metrics(result)
    guards = check_metric_integrity(metrics)

    save_json(run_id, "metrics.json", metrics)
    save_text(run_id, "stdout.log", result.stdout)
    save_text(run_id, "stderr.log", result.stderr)
    save_json(run_id, "guards.json", guards)

    if not guards["ok"]:
        save_status(run_id, "rejected_metric_integrity")
    elif metrics["nan_or_inf"]:
        save_status(run_id, "rejected_nan")
    elif metrics["crashed"]:
        save_status(run_id, "rejected_crash")
    else:
        save_status(run_id, "evaluated")

    append_leaderboard(run_id, metrics)
    return {"run_id": run_id, "status": "done", "metrics": metrics}
  • The loop’s most important invariant is that every candidate, including invalid ones, leaves behind enough evidence for the next proposer to learn from it. This is what turns autonomous tinkering into cumulative research.

Harness optimization with Meta-Harness

  • Autoresearch becomes most powerful when the optimized object is not only a model-training script, but the full harness around a model: the code that decides what the model sees, what it remembers, what tools it can use, how outputs are parsed, how intermediate state is updated, and how future calls are conditioned on prior events. Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026) formalizes this as an outer-loop search over executable harness code, using a coding-agent proposer that can inspect previous code, scores, and execution traces through a filesystem.

What a harness is

  • A harness is the executable environment wrapped around a fixed model. In a simple classifier, the harness may only build a prompt. In a retrieval-augmented math solver, it may select solved examples, format them, call the model, parse the answer, and retry on malformed outputs. In an agentic coding system, it may maintain task state, decide when to call the shell, summarize files, route tool outputs, recover from errors, and decide when to submit.

  • The harness is therefore a policy over context and control flow:

    \[a_t, p_t, z_t = H(s_t, x, \mathcal{M}_t)\]
    • where \(s_t\) is the current task state, \(x\) is the task instance, \(\mathcal{M}_t\) is external memory, \(p_t\) is the prompt or model input, \(a_t\) is the next action or tool call, and \(z_t\) is the updated internal state. The model itself remains fixed, but the harness can change the distribution of trajectories the model produces.

  • A rollout can be written as:

    \[\tau = \left( s_0, p_0, y_0, a_0, s_1, p_1, y_1, a_1, \dots, s_T \right)\]

    • where \(y_t \sim M(\cdot \mid p_t)\) is the model output at step \(t\). Harness optimization searches for the harness \(H\) that maximizes expected reward:

      \[H^{*} = \arg\max_{H} \mathbb{E}_{x \sim X,\ \tau \sim p_M(H,x)} \left[ r(\tau, x) \right]\]

  • This objective is useful because it makes clear that harness engineering is not “prompt tweaking”; it is program search over the system that induces model behavior. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines by Khattab et al. (2023) is relevant because it treats LM pipelines as optimizable programs rather than fixed prompt strings, which is the same systems-level direction taken by harness optimization.

Why harnesses are high-leverage

  • Harnesses are high-leverage because many model failures are not caused by missing parametric knowledge alone. They are caused by missing context, poorly selected examples, lossy summaries, brittle parsers, premature stopping, unhelpful tool routing, or state that is stored in the wrong form. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis et al. (2020) is relevant because it shows how external non-parametric memory can improve generation when the right information is retrieved and conditioned on.

  • The practical implication is that a fixed model \(M\) can produce very different outcomes under different harnesses:

    \[p_M(\tau \mid H_1, x) \ne p_M(\tau \mid H_2, x)\]
    • even though the underlying model weights are identical. A better harness changes the task distribution seen by the model at inference time: it can expose better examples, compress history more faithfully, recover from tool errors, ask for structured outputs, or decompose tasks into more solvable subproblems.

The search loop

  • The Meta-Harness loop has three core actions: propose, evaluate, and log. A coding-agent proposer reads prior experience from disk, proposes a new harness, the evaluator runs the harness on search tasks, and the resulting code, metrics, prompts, tool calls, model outputs, and traces are written back to disk for future iterations.

  • A minimal implementation looks like:

def search_harnesses(seed_harnesses, proposer, model, search_tasks, n_iters):
    archive = FilesystemArchive()
    population = []

    for harness in seed_harnesses:
        result = evaluate(harness, model, search_tasks)
        archive.write_run(harness=harness, result=result)
        population.append(harness)

    for step in range(n_iters):
        proposal = proposer.propose(
            archive_path=archive.root,
            instructions="Inspect prior code, metrics, and traces before editing.",
        )

        if not validate_interface(proposal):
            archive.write_invalid(proposal)
            continue

        result = evaluate(proposal, model, search_tasks)
        archive.write_run(harness=proposal, result=result)
        population.append(proposal)

    return pareto_frontier(archive.results)
  • The outer loop is deliberately simple. The proposer is not limited to a hand-written mutation operator, a fixed parent-selection rule, or a compressed prompt of previous scores. It can inspect anything in the archive and decide whether to make a local edit, revert a failed idea, combine two prior candidates, or rewrite the harness structure.

Filesystem as experience memory

  • The filesystem is the main implementation trick. Instead of forcing the proposer to consume all prior experience in one context window, the archive exposes experience as searchable files. A good run directory should contain:
runs/
  042/
    harness.py
    diff.patch
    metrics.json
    score.txt
    prompts.jsonl
    model_outputs.jsonl
    tool_calls.jsonl
    state_updates.jsonl
    failures.jsonl
    notes.md

  • This design lets the proposer perform selective credit assignment. It can search for repeated parse failures, compare the best and worst candidates, inspect only runs that improved one metric, or trace why a specific task failed. The key difference from ordinary prompt optimization is that the feedback channel is not a single scalar or short summary; it is a complete empirical record.

  • This matters because harness behavior often has long-range dependencies. A memory-update change at step \(2\) may only affect the final answer at step \(15\). A retrieval format change may help one task family and hurt another. A parser change may increase measured accuracy by reducing invalid outputs, even if reasoning quality is unchanged. These effects are difficult to diagnose from aggregate scores alone.

What gets optimized

  • Harness code can optimize many system components at once, as delineated below:

    • Prompt construction:

      • Prompt construction decides which instructions, examples, retrieved passages, intermediate summaries, and output schemas are shown to the model. A candidate prompt builder might choose different templates by task type:

      python id="vxu44o" def build_prompt(task, memory, retrieved, mode): if mode == "classification": return render_classification_prompt(task, memory.examples) if mode == "math": return render_math_prompt(task, retrieved.proofs) if mode == "coding": return render_agent_prompt(task, memory.repo_summary, memory.errors) raise ValueError(mode)

      • Useful search dimensions include instruction order, schema strictness, example count, chain-of-thought visibility policy, error-recovery text, and whether to include successful or failed prior attempts.

    • Retrieval policy:

      • Retrieval policy decides what external information to fetch and how to format it. For a query \(q\) and document set \(D\), a sparse retrieval harness might rank candidates by BM25:
      \[\mathrm{score}(q,d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{ f(t,d)(k_1 + 1) }{ f(t,d) + k_1(1 - b + b \cdot |d|/\mathrm{avgdl}) }\]
      • The harness can search over query rewriting, top-\(k\), diversity filters, deduplication, answer-aware reranking, proof-pattern retrieval, and context formatting. In retrieval-augmented reasoning, the important question is not whether retrieval exists, but whether the harness retrieves examples that are structurally useful for the target problem.

    • Memory update:

      • Memory update decides what persists across steps:
      \[\mathcal{M}_{t+1} = U(\mathcal{M}_t, s_t, y_t, a_t, o_t)\]
      • where \(o_t\) is a tool observation. Search can modify whether memory stores raw transcripts, compressed summaries, symbolic state, failed attempts, tool errors, retrieved examples, or confidence estimates. In long-horizon agents, memory design is often as important as the initial prompt because later decisions depend on what was preserved.

    • Tool orchestration:

      • Tool orchestration decides which tools are available, when to call them, and how observations are fed back into the model. For coding agents, this includes shell commands, file reads, test execution, patch application, and submission logic. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces by Merrill et al. (2026) is relevant because it evaluates agents on long-horizon terminal tasks where harness decisions around tools, state, and recovery strongly affect pass rate.

    • Output parsing and repair:

      • Parsing is often underappreciated. A model can reason correctly but fail the benchmark because the answer is not emitted in the expected format. A harness can search over strict JSON schemas, regex extraction, fallback parsers, self-repair calls, and validation loops:
      def parse_with_repair(raw_output, schema, model):
    parsed = try_parse(raw_output, schema)
    if parsed.ok:
        return parsed.value

    repair_prompt = build_repair_prompt(raw_output, schema, parsed.error)
    repaired = model(repair_prompt)
    return try_parse(repaired, schema).value
      • The evaluator should log parse errors separately from reasoning errors, because these imply different edits.

Evaluation design

  • Harness evaluation should produce both aggregate metrics and task-level traces. A classification harness might report accuracy and context tokens. A retrieval reasoning harness might report pass@1, retrieved-document overlap, parse-error rate, and model-call count. An agentic coding harness might report pass rate, test failures, timeouts, number of tool calls, wall-clock time, and final submission status.

  • For a task set \(X_{\mathrm{search}} = \{x_1,\dots,x_n\}\), a simple score is:

\[\hat{R}(H) = \frac{1}{n} \sum_{i=1}^{n} r(\tau_i, x_i)\]
  • For stochastic models, use repeated samples:
\[\hat{R}_K(H) = \frac{1}{nK} \sum_{i=1}^{n} \sum_{k=1}^{K} r(\tau_{i,k}, x_i)\]
  • For cost-aware harnesses, use a constrained or scalarized objective:
\[J(H) = -\hat{R}(H) + \lambda_c \cdot \mathrm{ContextTokens}(H) + \lambda_m \cdot \mathrm{ModelCalls}(H) + \lambda_t \cdot \mathrm{Latency}(H)\]
  • or maintain a Pareto frontier instead of collapsing everything into one scalar. Pareto tracking is usually safer because the desired tradeoff between quality and cost may change later.

Search-set and test-set separation

  • The search set is the set of tasks used during harness evolution. The test set is held out until final evaluation. The proposer should never see test results during search. Otherwise, harness search can overfit just like hyperparameter tuning can overfit a validation set.

  • A clean protocol is:

train/search tasks:
  used repeatedly during outer-loop search

development audits:
  used occasionally by humans to inspect failure modes

held-out test tasks:
  used once for final reporting

contamination checks:
  used to detect leakage, hard-coded labels, or task-specific shortcuts
  • For public discovery benchmarks, where repeated benchmark-specific iteration is part of the setting, the system should still audit for explicit task leakage, brittle string matching, and hard-coded answer paths.

Trace schema

  • A useful trace schema for harness optimization is:
{
  "run_id": "057",
  "task_id": "math_0183",
  "harness_hash": "abc123",
  "model": "fixed-base-model",
  "steps": [
    {
      "t": 0,
      "state_summary": "...",
      "prompt_path": "prompts/000.txt",
      "model_output_path": "outputs/000.txt",
      "tool_call": null,
      "observation_path": null,
      "parser_status": "ok"
    }
  ],
  "final_answer": "...",
  "reward": 1,
  "context_tokens": 18422,
  "model_calls": 3,
  "latency_seconds": 41.2,
  "failure_type": null
}
  • This schema supports both automated analysis and agent inspection. The proposer can search for parser_status: failed, tasks with high context and low reward, or tool calls that repeatedly precede failures.

Credit assignment over harness edits

  • Credit assignment is hard because one harness edit may affect many downstream behaviors. Suppose a new harness improves reward from \(0.42\) to \(0.48\). The improvement might come from better retrieval, stricter parsing, more examples, or a subtle interaction between memory and prompt format. The proposer should therefore compare candidate traces, not only candidate scores.

  • A practical approach is to decompose each result by failure type:

\[\mathrm{ErrorRate}(H) = \mathrm{ReasoningError}(H) + \mathrm{RetrievalError}(H) + \mathrm{ParseError}(H) + \mathrm{ToolError}(H) + \mathrm{TimeoutError}(H)\]
  • This decomposition tells the next proposal where to intervene. If parse errors dominate, do not rewrite retrieval. If retrieval errors dominate, do not tune the final answer parser. If timeouts dominate, reduce tool loops or context size.

Harness search patterns

  • Local repair:

    • Local repair modifies a single component, such as output parsing or retry logic. It is low risk and often useful after a regression.
    Observation:
Many failed tasks contain the correct answer but invalid formatting.

Edit:
Add strict extraction and one repair call.

Expected effect:
Reduce parse-error rate without changing reasoning behavior.

  • Component swap:

    • Component swap replaces one subsystem while preserving the rest of the harness.
    Observation:
Dense retrieval hurts some reasoning tasks, while sparse retrieval is more stable.

Edit:
Keep sparse retrieval, but rewrite query generation and top-k filtering.

Expected effect:
Improve retrieved-example relevance without changing model-call budget.

  • Additive fallback:

    • Additive fallback preserves the current best behavior and adds a recovery path only when the default path fails.
    answer = primary_solver(task)
if not verifier.accepts(answer):
    answer = fallback_solver(task, previous_answer=answer)
return answer
    • This is often safer than rewriting the primary path.

  • Full rewrite:

    • Full rewrites are useful when traces show the current architecture is structurally misaligned with the task. They are risky and should be isolated into separate branches so that the system can recover if they regress.

Meta-Harness-style proposer instructions

  • A strong proposer instruction for harness search should emphasize evidence before editing:
Before proposing a new harness:
1. Inspect the current Pareto frontier.
2. Compare at least one improved run and one regression.
3. Read task-level traces for representative failures.
4. Identify the dominant failure mode.
5. Make one coherent edit that targets that failure mode.
6. Preserve evaluator interfaces and logging.
7. Record the hypothesis, diff summary, expected metric movement, and risks.
  • The point is not to make the agent verbose. The point is to force the search loop to behave like experimental science: observe, hypothesize, intervene, measure, and update.

Why full-history access beats summaries

  • Summaries are useful, but they are lossy. A summary might say “retrieval was noisy,” while the raw traces reveal that retrieval was good but the formatting caused the model to ignore the examples. A summary might say “tool use increased,” while raw logs reveal that one specific command pattern caused most timeouts. Full-history access lets the proposer form sharper causal hypotheses.

  • This is the same reason autoresearch logs full training traces rather than only the best BPB. A 0.002 BPB improvement means little without knowing whether it came from faster throughput, a true modeling gain, a changed validation path, or noise.

Implementation checklist

  • A practical harness-optimization setup should include:

    • Harness interface: a stable function signature such as run_task(task, model, tools, config) -> Result.
    • Evaluator: deterministic task loading, metric computation, budget enforcement, and result serialization.
    • Trace logger: prompts, outputs, tool calls, state updates, parser status, and final reward.
    • Archive: one immutable directory per candidate.
    • Proposer sandbox: read access to all prior runs, write access only to new candidate files.
    • Validation gates: import checks, interface checks, budget checks, parser checks, and small smoke tasks.
    • Selection logic: leaderboard plus Pareto frontier.
    • Leakage audit: scan candidate code for task IDs, answer strings, benchmark-specific shortcuts, and forbidden file access.
    • Final evaluation: run only frontier candidates on held-out tasks.

  • This turns harness engineering into a repeatable autoresearch loop: the agent searches over executable model scaffolding, the evaluator supplies grounded feedback, and the filesystem preserves enough experience for the next proposal to be better than a blind mutation.

Implementing the single-GPU research substrate

  • The simplest useful autoresearch substrate is a bounded, single-GPU language-model training environment where the agent is allowed to edit the model and training loop, but not the data-preparation or evaluation contract. The goal is to make experiments cheap, comparable, and inspectable, so the agent can run many trials and accumulate evidence instead of making one large, hard-to-debug change. karpathy/autoresearch is a compact implementation of this pattern: an agent edits the training file, runs a fixed-budget experiment, checks validation bits per byte, and repeats.

Repository boundary design

  • A good substrate begins by drawing a hard boundary between stable infrastructure and editable research code. The stable layer should own data download, tokenizer construction, validation split creation, dataloading, evaluation, metric calculation, and result serialization. The editable layer should own the model architecture, optimizer, schedule, precision choices, batch sizing, and training-loop mechanics. This boundary matters because an autonomous agent should be able to improve training, but should not be able to accidentally improve the score by changing what “validation” means.

  • A minimal layout looks like:

project/
  prepare.py        # fixed data prep, tokenizer, dataloaders, validation metric
  train.py          # editable model, optimizer, training loop
  program.md        # agent instructions and research protocol
  results.tsv       # append-only experiment ledger
  runs/
    000_seed/
    001_candidate/
    002_candidate/
  • The single editable file is not just a convenience. It is a search-space regularizer. If the agent can edit every file, the experiment becomes harder to audit and easier to corrupt. If the agent can edit only one coherent training file, every candidate can be reviewed as a single diff. The small-codebase design is similar in spirit to nanochat, which is designed as a minimal single-node LLM training harness covering tokenization, pretraining, finetuning, evaluation, inference, and chat UI.

Fixed budget as the experimental contract

  • The fixed budget is the main comparability device. If every candidate trains for exactly \(B\) wall-clock seconds, then the outer loop rewards changes that improve quality under the same resource limit:

    \[c^{*} = \arg\min_{c \in \mathcal{C}} \mathrm{BPB} \left( \mathrm{Train}(c; B) \right)\]
    • where \(c\) is the code candidate and \(B\) is the fixed training budget. In practice, this budget should exclude one-time startup or compilation when those are not part of the research question, but include normal training overhead once the run begins. The single-GPU setup uses a fixed 5-minute training run so the agent can run many comparable experiments overnight, roughly turning a workstation into a small autonomous research lab.

  • This fixed-budget framing changes the meaning of “better.” A larger model may have better asymptotic performance but worse fixed-budget BPB because it sees fewer updates. A smaller model may win because it trains more tokens per second. The system is therefore optimizing the joint interaction of architecture, optimizer, batch size, sequence length, compiler behavior, and hardware utilization, not only model expressivity.

Validation bits per byte

  • Validation bits per byte is a useful autoresearch metric because it is closer to a compression-normalized score than raw token-level loss. If the model reports average negative log-likelihood in nats over validation tokens, the conversion is:
\[\mathrm{bits} =\frac{\mathcal{L}_{\mathrm{nats}}}{\ln 2}\]
  • If validation contains \(N_{\mathrm{bytes}}\) bytes and \(N_{\mathrm{tokens}}\) tokens, a practical BPB estimate is:
\[\mathrm{BPB} =\frac{ \sum_{i=1}^{N_{\mathrm{tokens}}} -\log_2 p_{\theta}(x_i \mid x_{<i}) }{ N_{\mathrm{bytes}} }\]
  • Lower BPB is better. The reason BPB is preferable to raw token loss in a setting where the tokenizer may change is that token loss depends strongly on how many tokens the tokenizer emits. BPB anchors the score to bytes, making comparisons more meaningful across vocabulary-size and tokenization changes.

Training objective

  • The inner-loop training objective is standard autoregressive language modeling. For a sequence \(x_1,\dots,x_T\), the model minimizes:
\[\mathcal{L}_{\mathrm{LM}}(\theta) =-\frac{1}{T} \sum_{t=1}^{T} \log p_{\theta}(x_t \mid x_{<t})\]
  • The outer loop does not replace gradient descent. It wraps around it. The model weights \(\theta\) are optimized by the usual trainer, while the agent searches over code candidates \(c\) that determine the architecture, optimizer, data flow, schedule, precision, and other training details:
\[\theta_T(c) = \mathrm{Train}(\theta_0, c, B)\] \[s(c) = \mathrm{BPB} \left( \theta_T(c) \right)\]
  • This separation is important. Autoresearch is not a new optimizer for model weights; it is an optimizer over the experimental program that produces trained weights.

Model architecture search knobs

  • The editable training file should expose a small number of high-leverage knobs, but the agent should still be free to rewrite their relationships. For a compact GPT-like model, the most important knobs are usually:

    • Depth: number of transformer blocks.
    • Width: embedding dimension and hidden dimension.
    • Attention heads: number of heads and head dimension.
    • Sequence length: maximum context length.
    • MLP ratio: expansion factor in the feed-forward network.
    • Normalization: pre-norm, post-norm, RMSNorm, LayerNorm, or variants.
    • Attention pattern: full attention, local attention, alternating local/global patterns, or sliding windows.
    • Parameter tying: tied input/output embeddings or separate matrices.

  • A practical agent instruction is to prefer changes that preserve tensor-shape clarity and throughput instrumentation. Many architecture edits look promising in isolation but silently reduce tokens per second enough to lose under the fixed budget.

  • The outer-loop comparison should therefore log both quality and speed:

    \[\Delta \mathrm{BPB} = \mathrm{BPB}_{\mathrm{candidate}} - \mathrm{BPB}_{\mathrm{parent}}\] \[\Delta \mathrm{TPS} = \mathrm{TPS}_{\mathrm{candidate}} - \mathrm{TPS}_{\mathrm{parent}}\]
    • where \(\mathrm{TPS}\) is tokens per second. A candidate with \(\Delta \mathrm{BPB} < 0\) and \(\Delta \mathrm{TPS} > 0\) is a clean win. A candidate with better BPB and worse throughput may still be valuable, but it should be marked as a tradeoff rather than blindly accepted.

Optimizer design

  • The optimizer is one of the highest-leverage editable components because the fixed-budget score rewards fast early learning. A typical baseline can combine AdamW for embeddings, normalization, and scalar parameters with a specialized optimizer for matrix-shaped hidden-layer weights. Decoupled Weight Decay Regularization by Loshchilov and Hutter (2017) is relevant because AdamW separates weight decay from the adaptive gradient step, making the weight-decay coefficient less entangled with the learning rate than ordinary \(L_2\) regularization in Adam.

  • AdamW can be written as:

    \[g_t = \nabla_{\theta} \mathcal{L}_t(\theta_t)\] \[m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t\] \[v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2\] \[\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}\] \[\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon} - \eta \lambda \theta_t\]
    • where \(\eta\) is the learning rate and \(\lambda\) is the decoupled weight-decay coefficient.

  • Muon-style optimizers are also relevant in this setting because they are designed for hidden-layer matrices and have been used in small-model training speedrun contexts. Muon is Scalable for LLM Training by Liu et al. (2025) studies techniques for scaling Muon to larger LLM training, and Muon: An optimizer for hidden layers in neural networks is a practical optimizer writeup describing Muon as an optimizer for hidden layers.

  • A simplified Muon-like update applies momentum and then orthogonalizes a matrix update direction:

\[M_t = \mu M_{t-1} + G_t\] \[O_t \approx \mathrm{Orthogonalize}(M_t)\] \[W_{t+1} = W_t - \eta O_t\]
  • The practical design pattern is to let the agent search optimizer partitioning rules:
def parameter_groups(model):
    matrix_params = []
    scalar_or_embedding_params = []

    for name, p in model.named_parameters():
        if p.ndim == 2 and "embed" not in name and "lm_head" not in name:
            matrix_params.append(p)
        else:
            scalar_or_embedding_params.append(p)

    return [
        {"params": matrix_params, "optimizer": "muon"},
        {"params": scalar_or_embedding_params, "optimizer": "adamw"},
    ]
  • The evaluator should log optimizer grouping, learning rates, weight decay, gradient norms, NaN events, and whether any parameter group received no gradients.

Batch-size and sequence-length tradeoffs

  • Batch size and sequence length define how much data each forward/backward pass processes. If \(B_d\) is device batch size and \(L\) is sequence length, then tokens per step are approximately:
\[N_{\mathrm{tokens/step}} = B_d \cdot L\]
  • With gradient accumulation \(A\), the total batch tokens per optimizer update are:
\[N_{\mathrm{tokens/update}} = A \cdot B_d \cdot L\]

  • Under a fixed wall-clock budget, increasing \(L\) can improve long-context modeling but reduce throughput. Increasing \(B_d\) can improve hardware utilization but may reduce update frequency or trigger out-of-memory failures. Increasing gradient accumulation can stabilize optimization but may reduce the number of optimizer updates within the budget.

  • A good autoresearch agent should not tune these independently. It should reason about the product \(A \cdot B_d \cdot L\), memory pressure, and update count:

\[N_{\mathrm{updates}} \approx \frac{ T_{\mathrm{budget}} \cdot \mathrm{tokens/sec} }{ A \cdot B_d \cdot L }\]
  • The fixed-budget objective often rewards configurations that maintain enough updates for fast learning while keeping throughput high.

Learning-rate schedule

  • A learning-rate schedule is another high-impact edit. A standard warmup plus cosine decay schedule is:
\[\eta_t = \begin{cases} \eta_{\max}\frac{t}{T_{\mathrm{warmup}}}, & t < T_{\mathrm{warmup}} \\ \eta_{\min} + \frac{1}{2} (\eta_{\max}-\eta_{\min}) \left( 1 + \cos \left( \pi \frac{t-T_{\mathrm{warmup}}}{T_{\mathrm{total}}-T_{\mathrm{warmup}}} \right) \right), & t \ge T_{\mathrm{warmup}} \end{cases}\]

  • For autoresearch, the schedule should be parameterized by time budget or estimated total steps, not by a fixed epoch count. Otherwise, architecture or throughput changes can accidentally make the schedule too short or too long.

  • A robust schedule API is:

```python id=”olpaid” def get_lr(step, total_steps, warmup_frac, max_lr, min_lr): warmup_steps = int(warmup_frac * total_steps)

if step < warmup_steps:
    return max_lr * step / max(1, warmup_steps)

progress = (step - warmup_steps) / max(1, total_steps - warmup_steps)
cosine = 0.5 * (1.0 + math.cos(math.pi * progress))
return min_lr + cosine * (max_lr - min_lr) ```
  • The agent can then search over warmup_frac, max_lr, min_lr, and schedule shape without breaking comparability.

Precision and numerical stability

  • A single-GPU autoresearch loop should include explicit numerical-stability instrumentation because agents will try aggressive learning rates, optimizer variants, and batch-size changes. At minimum, every run should detect non-finite loss, non-finite gradients, and exploding norms:
if not torch.isfinite(loss):
    raise RuntimeError(f"non-finite loss at step {step}: {loss.item()}")

total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
if not torch.isfinite(total_norm):
    raise RuntimeError(f"non-finite grad norm at step {step}: {total_norm}")
  • The results table should distinguish instability from ordinary underperformance:
status:
  evaluated
  nan_loss
  nan_grad
  out_of_memory
  timeout
  import_error
  syntax_error
  metric_missing
  • This classification helps the next proposal. A NaN run suggests reducing learning rate, adding gradient clipping, changing initialization, or adjusting precision. A slow but stable run suggests optimizing throughput. A high-throughput but poor-BPB run suggests insufficient capacity or too few effective updates.

Result ledger

  • The result ledger should be append-only. It is both the memory of the research loop and the audit trail for humans. A useful results.tsv schema is:
run_id
parent_id
timestamp
status
val_bpb
train_loss
val_loss
tokens_per_sec
train_tokens
steps
params
depth
seq_len
device_batch_size
grad_accum
total_batch_tokens
optimizer_summary
max_lr
weight_decay
peak_mem_gb
diff_summary
notes_path
  • A candidate should never overwrite a previous result. Even failed runs are valuable because they define the boundary of the stable search space.

Candidate snapshotting

  • Every run should preserve the exact code that produced it. The easiest pattern is to copy the editable file into the run directory and store a patch against the parent:
runs/017/
  train.py
  diff.patch
  metrics.json
  stdout.log
  stderr.log
  notes.md
  • The agent should be instructed to inspect both successful and failed snapshots. Improvements often come from transplanting a subcomponent from a worse overall candidate, such as a faster data layout or safer initialization.

Agent prompt for single-GPU research

  • A useful program.md style instruction should be operational, not philosophical:
You are optimizing validation BPB under a fixed 5-minute training budget.

Allowed:
- Edit train.py.
- Change architecture, optimizer, schedule, batch sizing, precision, and training loop.
- Run smoke tests and full experiments.
- Inspect all previous runs.

Forbidden:
- Do not edit data preparation, validation data, tokenizer accounting, metric computation, or result parser.
- Do not compare partial runs to full runs.
- Do not keep NaN-producing candidates.
- Do not delete old run logs.

Before each edit:
- Inspect the current best run.
- Inspect at least one recent regression.
- State a hypothesis.
- Make one coherent change.

After each run:
- Record the metric, throughput, memory, status, diff summary, and interpretation.
  • This instruction layer is the closest thing to “research management.” It defines the norms of the autonomous lab.

Guardrails against false progress

  • The system should reject or flag any run that violates the experimental contract. Common false-progress modes include changing validation data, reducing evaluation tokens, changing byte accounting, accidentally reading stale metrics, skipping hard batches, silently shortening training, or comparing a partial run to a full run.

  • A simple metric-integrity check can hash the frozen files:

\[h_{\mathrm{eval}} = \mathrm{SHA256} ( \texttt{prepare.py} \parallel \texttt{validation\_manifest} \parallel \texttt{metric\_code} )\]
  • Then every run records:
{
  "eval_hash": "abc123",
  "expected_eval_hash": "abc123",
  "metric_integrity_ok": true
}
  • If the hashes differ, the run should be excluded from the leaderboard.

Human review interface

  • Even in autonomous mode, the system should make human review easy. The best interface is a compact progress plot plus a sortable leaderboard. Each improvement should link to its run directory, diff, logs, and notes. The progress plot should distinguish kept candidates from discarded candidates so a reviewer can see whether the agent is actually climbing or just sampling randomly.

  • A minimal review checklist is:

    • Does the best candidate preserve the evaluator?
    • Is the improvement larger than run-to-run noise?
    • Did throughput or train-token count change substantially?
    • Did parameter count change?
    • Did the model actually train for the full budget?
    • Are there hidden warnings in stderr?
    • Does the diff look like a general method rather than a metric hack?

  • The goal is not to remove humans. The goal is to move humans from manually editing every experiment to supervising an autonomous experimental process.

Designing the autonomous research agent

  • The agent is the part of an autoresearch system that turns stored evidence into the next experiment. It is not merely a code generator. It is a research operator that reads prior runs, diagnoses failures, proposes hypotheses, edits the allowed artifact, runs checks, interprets metrics, and leaves behind evidence for the next iteration.

Agent responsibilities

  • A useful autonomous research agent has five responsibilities: inspect, hypothesize, edit, evaluate, and explain. During inspection, it should examine the current best run, recent regressions, failure logs, and diffs. During hypothesis formation, it should identify a mechanism rather than only a desired outcome. During editing, it should make a change narrow enough to support credit assignment. During evaluation, it should obey the fixed budget and preserve the metric contract. During explanation, it should write enough context for later agents or humans to understand why the run happened.

  • This is the operational distinction between an agent and a sampler. A sampler proposes candidates from a prompt. A research agent maintains a working memory of the experiment history and uses tools to interrogate that history. Reflexion: Language Agents with Verbal Reinforcement Learning by Shinn et al. (2023) is relevant because it shows how agents can improve across trials by storing language feedback in memory instead of updating model weights.

  • A practical agent loop should include the following steps:

    • Read the leaderboard before editing: The agent should identify the current best candidate, the best score, the most recent candidates, and the main performance trend before deciding what to change.
    • Inspect the current best code: The agent should read the exact source snapshot that produced the current best run so that it does not accidentally regress a useful implementation detail.
    • Inspect recent regressions: The agent should examine at least one or two failed or underperforming candidates to understand which edit families recently caused crashes, slowdowns, NaNs, or worse validation metrics.
    • Identify the dominant opportunity or failure mode: The next edit should target a specific observed pattern, such as unstable optimization, poor throughput, parser failures, excessive context use, or repeated timeouts.
    • Write a hypothesis before changing code: The agent should record what mechanism it expects to improve and what evidence would falsify that expectation.
    • Make one coherent edit: The candidate should test a single interpretable idea, or a tightly coupled group of changes, so the result can be attributed to the intervention.
    • Run validation gates before the full experiment: The agent should run syntax checks, import checks, smoke tests, and metric-path checks before spending the full budget.
    • Run the bounded experiment: The candidate should be evaluated under the same time, data, and metric constraints as previous candidates.
    • Write metrics and notes: The agent should append structured metrics and a human-readable postmortem to the archive.
    • Decide whether to keep, revert, branch, or revisit: The agent should preserve the best candidate, reject invalid candidates, branch from promising but incomplete ideas, and mark useful regressions for future transplanting.

Research memory

  • The agent’s memory should have two layers: structured metrics and raw evidence. Structured metrics let the agent sort, filter, and compare runs quickly. Raw evidence lets it diagnose why a run behaved the way it did.

  • For single-GPU training, the memory should include the following artifacts:

    • An append-only results ledger: The ledger should store run IDs, parent IDs, status, validation BPB, throughput, memory use, parameter count, edit family, and notes path so the agent can compare experiments quickly.
    • A source snapshot for each run: Each run should preserve the exact train.py or editable artifact that produced the result, making it possible to reproduce and inspect candidates later.
    • A patch against the parent run: Each candidate should store a diff that shows exactly what changed relative to its parent, which supports credit assignment and human audit.
    • Structured metrics: Each run should save metrics such as validation loss, validation BPB, train loss, tokens per second, number of steps, peak memory, and crash status in a machine-readable file.
    • Raw stdout and stderr logs: Console logs should be retained because they often contain warnings, compilation behavior, CUDA errors, NaN reports, dataloader issues, or other signals that are not captured in scalar metrics.
    • Human-readable notes: Each run should include a short postmortem describing the hypothesis, change, result, interpretation, and next step.
    • Training curves or per-step traces: Curves are useful for distinguishing candidates that improve early learning from candidates that merely happen to finish with a slightly better final metric.

  • For harness optimization, the memory should also include prompts, model outputs, tool calls, parser failures, state updates, per-task rewards, and context-token usage. The key pattern is that the archive must be larger than any single prompt and still be navigable by the agent. This is why filesystem access is so useful: it lets the proposer selectively inspect the parts of history that matter for the next proposal rather than relying on a compressed summary.

  • The memory update after each run can be modeled as:

    \[\mathcal{D}_{t+1} = \mathcal{D}_{t} \cup \{c_t, \Delta_t, m_t, \ell_t, n_t\}\]
    • where \(c_t\) is the candidate code, \(\Delta_t\) is the diff, \(m_t\) is the metric record, \(\ell_t\) is the raw log bundle, and \(n_t\) is the note written after the experiment.

Hypothesis quality

  • A good hypothesis is mechanistic and falsifiable. “Try a smaller model” is weak. “Reducing depth from \(8\) to \(6\) may improve fixed-budget BPB because throughput and update count will increase more than per-token modeling capacity decreases” is stronger.

  • A useful hypothesis should contain the following parts:

    • Observation: The agent should state the specific metric, trace pattern, or failure mode that motivated the experiment.
    • Proposed cause: The agent should explain the mechanism it believes is responsible for the observed pattern.
    • Intervention: The agent should describe the single coherent change it will make to test that mechanism.
    • Expected movement: The agent should state which metrics should improve or worsen if the hypothesis is correct.
    • Risk: The agent should record the most likely regression mode, such as underfitting, slower throughput, instability, memory pressure, or parser fragility.

  • For example, a strong hypothesis would say that recent deeper models improved early train loss but lost validation BPB under the fixed 5-minute budget; the likely cause is that they are update-limited and see fewer optimizer steps; the intervention is to reduce depth while modestly increasing batch size; the expected result is higher throughput and lower validation BPB; and the main risk is that the smaller model may become capacity-limited and underfit.

  • This format makes later credit assignment easier. If the candidate improves throughput but worsens BPB, the hypothesis was only partly right. If it improves BPB without improving throughput, the mechanism was probably wrong even though the result was good.

Tool use policy

  • An autoresearch agent should use tools in a predictable order. Before editing, it should read. Before a full experiment, it should run a cheap check. Before accepting a result, it should verify metric integrity.

  • A robust tool policy should include the following behaviors:

    • Before editing, read the empirical state of the search: The agent should inspect the results ledger, the current best source snapshot, recent diffs, and logs from recent regressions before proposing a new candidate.
    • Before editing, search for repeated failures: The agent should scan prior logs for crashes, NaNs, out-of-memory errors, metric warnings, parser failures, timeouts, and stale-result risks.
    • Before a full run, execute cheap validation gates: The agent should run syntax checks, import checks, interface checks, and a short smoke-training or smoke-evaluation pass before launching the full experiment.
    • Before accepting metrics, verify freshness: The agent should ensure that the metric file was produced by the current run ID and not inherited from a stale or partially failed run.
    • After a full run, parse and validate the metrics: The agent should check the metric schema, compare evaluator hashes, classify the run status, and detect NaN, crash, timeout, or integrity failures.
    • After a full run, preserve the full evidence bundle: The agent should append the ledger, copy the source snapshot, save raw logs, write structured metrics, and record a postmortem.

  • This differs from pure reasoning-search methods such as Tree of Thoughts: Deliberate Problem Solving with Large Language Models by Yao et al. (2023), which expands and evaluates intermediate reasoning paths; autoresearch expands executable research states and evaluates them through real experiments.

Agent roles

  • A single agent can run the whole loop, but larger autoresearch systems benefit from role separation. Roles are not personalities; they are permissions and evaluation contracts.

    • Researcher: The researcher proposes code changes. It has read access to all prior runs and write access to candidate code. It should be optimized for hypothesis generation, implementation, and interpretation.
    • Evaluator: The evaluator runs experiments and computes metrics. It should be deterministic and conservative. Ideally, it should not be able to edit the candidate except through a clean checkout mechanism.
    • Auditor: The auditor checks for metric drift, leakage, suspicious diffs, stale result files, and hidden failures. It should compare evaluator hashes, inspect changed files, and reject invalid candidates.
    • Curator: The curator maintains the frontier, summarizes families of experiments, and decides which branches deserve more budget.

  • This role split prevents one agent from unconsciously optimizing the score by weakening the evaluator. It also makes the research organization scalable: many researchers can propose candidates, one evaluator can score them, and one curator can maintain a clean frontier.

Permission boundaries

  • Permissions are part of the algorithm. The agent should have enough freedom to discover non-obvious improvements, but not enough freedom to invalidate the experiment.

  • A practical permission map should use the following boundaries:

    • Editable files: The agent may edit the candidate artifact, such as train.py, harness.py, or candidate-local configuration files that affect only the proposed run.
    • Read-only files: The agent may inspect but not modify data preparation, validation data, metric computation, tokenizer accounting, benchmark task files, and previous run directories.
    • Append-only files: The agent may append to results ledgers, run notes, and leaderboard history, but should not rewrite prior records.
    • Forbidden actions: The agent should not delete old runs, edit held-out test results, alter metric parsers, weaken time-budget enforcement, or change validation data.

  • In the compact training setup, the intended pattern is that data preparation and runtime utilities remain fixed while the agent edits the single training file and the human iterates on the instruction file.

Planning depth

  • The agent should plan enough to avoid random edits, but not so much that it spends the whole budget reasoning instead of running experiments. A useful default is shallow planning for routine edits and deeper planning for structural rewrites.

  • For local edits, the agent should inspect a small number of highly relevant runs, identify one hypothesis, change one component, and run one experiment. For structural rewrites, the agent should inspect the frontier, compare several regressions, write a short design note, run stronger smoke tests, and only then launch a full experiment.

  • Planning can be framed as a value-of-information problem. The agent should spend more time inspecting history when the expected cost of a bad experiment is high:

\[\mathrm{VOI} = \mathbb{E} [ U(a \mid \mathrm{extra\ evidence}) - U(a \mid \mathrm{current\ evidence}) ] - C_{\mathrm{inspection}}\]
  • If inspection is cheap and full runs are expensive, deeper diagnosis is worthwhile. If runs are cheap and logs are simple, faster experimentation may dominate.

Exploration and exploitation

  • Autoresearch needs both exploitation and exploration. Exploitation refines the current best candidate. Exploration tries different mechanisms that may initially underperform but reveal useful ideas.

  • A simple controller can allocate experiments as:

    \[P(\mathrm{explore}) = \max \left( p_{\min}, p_0 \cdot e^{-t / \tau} \right)\]
    • where \(p_0\) is the initial exploration rate, \(p_{\min}\) is the minimum long-run exploration rate, and \(\tau\) controls annealing.

  • A more agentic version assigns each candidate a novelty score and a quality score:

\[S(c) = -\mathrm{BPB}(c) + \lambda_n \cdot \mathrm{Novelty}(c) - \lambda_r \cdot \mathrm{Risk}(c)\]
  • Novelty can be estimated from edit family, architecture family, optimizer family, or distance from previous diffs. Risk can be estimated from crash history, memory pressure, number of changed lines, or whether the edit touches fragile code.

Skill libraries

  • As the agent discovers useful procedures, it should turn them into reusable skills. A skill is not necessarily code used by the final model; it can be a research maneuver. Examples include “bisect a NaN regression,” “compare throughput-normalized candidates,” “audit evaluator integrity,” “transplant only the scheduler from a candidate,” or “cluster failures by parser status.”

  • Voyager: An Open-Ended Embodied Agent with Large Language Models by Wang et al. (2023) is relevant because it uses an ever-growing library of executable skills to support lifelong exploration and reuse across tasks.

  • A research skill should specify when to use it, what evidence it needs, and how to verify that it worked. For example, a component-transplant skill should be used when a candidate is worse overall but contains one useful subsystem. The agent should identify the subsystem and its local dependencies, copy only that subsystem into the current best candidate, preserve all unrelated behavior, run smoke tests, run the full budget, and compare the new result against both the current best and the donor candidate.

  • This prevents useful ideas from being discarded just because they appeared inside a bad candidate.

Communication between agents

  • When multiple agents run in parallel, they need a shared protocol. Without one, they will duplicate work, overwrite each other, or interpret metrics inconsistently.

  • A simple shared protocol should include the following stages:

    • Claiming work: An agent should reserve a candidate ID, identify the intended parent run, and write the edit family it plans to explore.
    • Running evaluation: The evaluator should mark the candidate as running, launch the experiment, and prevent other agents from reusing the same run ID.
    • Finishing evaluation: The evaluator should write the final status, structured metrics, raw logs, and source snapshot.
    • Interpreting results: The researcher should write a postmortem explaining whether the hypothesis was supported and what should happen next.
    • Curating the frontier: The curator should update the leaderboard or Pareto frontier and mark promising branches for further work.

  • The ledger should include the run ID, claiming agent, parent ID, status, edit family, start time, finish time, metric, and notes path. For parallel search, the candidate proposal distribution should penalize duplicated edit families already in flight:

    \[P(c) \propto \exp \left( \frac{ Q(c) - \lambda D(c, \mathcal{I}) }{ \tau } \right)\]
    • where \(Q(c)\) is expected quality, \(\mathcal{I}\) is the set of in-flight experiments, and \(D(c,\mathcal{I})\) measures similarity to active proposals.

Auditor checks

  • The auditor should run automatically after every candidate. Its job is not to judge scientific taste; its job is to enforce the contract.

  • For training-code autoresearch, auditor checks should verify that editable-file boundaries held, evaluator hashes matched, the result file was produced during the current run, the candidate trained for the full budget, validation token and byte counts did not change, stderr did not contain hidden warnings, and the run compared against the correct parent.

  • For harness optimization, auditor checks should additionally verify that the harness did not access held-out labels, hard-code benchmark task IDs, change parser behavior without logging parse-error rate, increase context beyond the allowed budget, or silently retry more times than permitted.

  • Auditing is especially important in code-space search because the agent is powerful enough to change the system around the metric. The advantage is that suspicious shortcuts are visible as code diffs or file accesses, so they can often be detected automatically.

Research organization code

  • The instruction file is the research organization encoded as text. It should evolve more slowly than candidate code. Humans should edit it when they observe systematic problems in the agent’s behavior, such as too many risky rewrites, insufficient logging, repeated NaN runs, or failure to inspect regressions.

  • A strong instruction file should define the mission, scientific method, permissions, experiment protocol, memory protocol, selection protocol, and safety protocol. The mission should specify the target metric and budget. The scientific method should require every run to have a hypothesis, a coherent change, and an interpretation. The permissions should define editable, read-only, append-only, and forbidden files. The experiment protocol should define validation gates, the full-run command, the metric parser, and result location. The memory protocol should define how to name runs, write notes, and inspect history before editing. The selection protocol should define how to compare candidates, branch, and update the frontier. The safety protocol should define how to handle crashes, NaNs, out-of-memory failures, suspicious improvements, and evaluator drift.

  • This aligns with the broader lesson that scalable search procedures tend to outperform brittle hand-coded solutions as compute and agents improve. The Bitter Lesson by Sutton (2019) is relevant because it argues that general methods that scale with computation, especially search and learning, tend to dominate hand-built knowledge over time.

Agent evaluation

  • The research agent itself should be evaluated, not only the candidates it produces. Useful agent-level metrics include:
\[\mathrm{hit\ rate} = \frac{ \# \mathrm{improving\ candidates} }{ \# \mathrm{evaluated\ candidates} }\] \[\mathrm{invalid\ rate} = \frac{ \# \mathrm{invalid\ candidates} }{ \# \mathrm{proposed\ candidates} }\] \[\mathrm{time\ to\ best} = \min \{t : s(c_t) = \min_i s(c_i)\}\] \[\mathrm{useful\ regression\ rate} = \frac{ \# \mathrm{regressions\ later\ mined\ for\ useful\ components} }{ \# \mathrm{regressions} }\]
  • For harness search, also track trace-inspection behavior, such as how many prior candidates the proposer inspected and whether it looked at raw traces before editing. A system that improves only by luck may still find good candidates, but it will be less reliable than one that consistently diagnoses failure modes before proposing changes.

Practical default policy

  • A good default policy for one autonomous agent should require the agent to read the leaderboard, read the current best candidate, inspect two recent failures, choose one edit family, write a hypothesis, make a small coherent change, run smoke checks, run the fixed-budget experiment, append structured metrics, write a postmortem, and decide whether to keep, revert, branch, or revisit.

  • Every tenth iteration, the system can allow a larger exploration. In that case, the agent should review all runs by edit family, identify underexplored regions, propose one structural change, run stronger validation before the full budget, and preserve the previous best unchanged.

  • This policy keeps the loop scientific without making it rigid. It gives the agent enough structure to accumulate knowledge and enough freedom to discover improvements humans did not pre-specify.

Evaluation

  • Autoresearch needs evaluation machinery that is stricter than ordinary ad hoc experimentation because the optimizer is allowed to edit code. The evaluator defines the scientific contract: what counts as progress, what is forbidden, which metrics are comparable, and which candidates must be rejected even if they appear to improve the headline score.

Metric choice

  • The primary metric should match the research substrate. In single-GPU language-model training, validation bits per byte is a good default because it measures compression quality on held-out text and is less tied to vocabulary size than raw token loss. In harness optimization, the primary metric is usually task accuracy, pass rate, reward, or another task-specific outcome. In long-horizon terminal agents, pass rate is often the central metric because a candidate either completes the task under the benchmark’s tests or fails. Terminal-Bench by Merrill et al. (2026) is relevant because it evaluates agents on hard terminal tasks with task-specific environments, human-written solutions, and verification tests, making pass rate a natural benchmark-level metric.

  • For language modeling, the core validation metric can be written as:

\[\mathrm{BPB} = \frac{ \sum_{i=1}^{N_{\mathrm{tokens}}} -\log_2 p_{\theta}(x_i \mid x_{<i}) }{ N_{\mathrm{bytes}} }\]

  • For task-solving harnesses, the empirical reward estimate is:

    \[\hat{R}(H) = \frac{1}{n} \sum_{i=1}^{n} r(\tau_i, x_i)\]
    • where \(H\) is the harness, \(x_i\) is a task instance, \(\tau_i\) is the rollout produced by the model inside the harness, and \(r(\tau_i, x_i)\) is the task reward.

Secondary metrics

  • The headline metric is not enough. A candidate that improves BPB while doubling memory use, slowing training dramatically, or relying on more context may be a poor research outcome. Every evaluation should log secondary metrics that explain the cost of the improvement.

    • Throughput: The evaluator should record tokens per second, examples per second, or tasks per hour so that quality gains can be separated from efficiency regressions.
    • Training exposure: The evaluator should record the number of training tokens, optimizer steps, and completed validation tokens so candidates are compared under equivalent work.
    • Memory use: The evaluator should record peak accelerator memory because some candidates only work by moving close to an out-of-memory boundary.
    • Parameter count: The evaluator should record model size because fixed-budget improvements can come from better capacity, better speed, or both.
    • Context cost: Harness evaluations should record prompt tokens, retrieved tokens, number of model calls, and tool-call count because a higher score may come from using substantially more inference budget.
    • Failure rate: The evaluator should separately track crashes, NaNs, parser failures, timeouts, invalid outputs, and budget violations so that failure modes do not get collapsed into one vague “bad run” label.

  • A cost-aware scalar objective can be useful for triage:

    \[J(c) = s(c) + \lambda_{\mathrm{mem}} \cdot \max(0, M(c)-M_{\max}) + \lambda_{\mathrm{time}} \cdot T(c) + \lambda_{\mathrm{fail}} \cdot \mathbb{1}[\mathrm{invalid}(c)]\]
    • where \(s(c)\) is the main score, \(M(c)\) is memory use, \(T(c)\) is runtime cost, and invalid candidates receive an explicit penalty. For many research settings, however, a Pareto frontier is cleaner than scalarization because the desired tradeoff between quality, speed, memory, and context cost may change later.

Pareto evaluation

  • A candidate is Pareto-optimal when no other candidate is at least as good on every tracked objective and strictly better on at least one. This is especially important in harness search, where a candidate may improve accuracy but use more tokens, more model calls, or more tool steps.
\[c_i \in \mathcal{P} \iff \nexists c_j: \left( \forall k,\ f_k(c_j) \le f_k(c_i) \right) \land \left( \exists k,\ f_k(c_j) < f_k(c_i) \right)\]
  • A good evaluation report should therefore present both the single best candidate under the chosen deployment preference and the frontier of alternatives. One candidate might be the best high-accuracy configuration, another might be the best low-cost configuration, and another might be the best robust configuration with few invalid outputs.

Search and test splits

  • Autoresearch needs a clean split between search-time feedback and final evaluation. The search set is allowed to influence proposals. The test set is not. This distinction is essential because an outer-loop agent can overfit just as easily as a human hyperparameter tuner.

    • Search set: The agent can repeatedly evaluate candidates here, inspect failures, and use traces to guide future edits.
    • Validation or selection set: The system can use this for candidate selection or frontier pruning when the search set is noisy or too small.
    • Held-out test set: The final report should evaluate only selected candidates here, and the proposer should not see the results during search.
    • Audit set: A small set of tasks can be reserved for checking leakage, parser shortcuts, memorized task IDs, or suspicious benchmark-specific behavior.

  • In public discovery settings, repeated benchmark iteration may be part of the competition or research objective, but the system should still audit candidates for hard-coded task identifiers, answer strings, and benchmark-specific shortcuts. This is especially important when optimizing executable harness code rather than only model weights.

Noise control

  • Autoresearch often makes decisions from short, noisy runs. A small BPB improvement may be real, or it may be seed noise, hardware jitter, dataloader variation, or stochastic sampling. The evaluator should estimate noise before treating small differences as discoveries.

  • For repeated runs of the same candidate, the mean and standard error are:

\[\bar{s} = \frac{1}{K} \sum_{k=1}^{K} s_k\] \[\mathrm{SE} = \frac{ \sqrt{\frac{1}{K-1}\sum_{k=1}^{K}(s_k-\bar{s})^2} }{ \sqrt{K} }\]

  • A simple acceptance rule with a margin is:

    \[\mathrm{accept}(c) = \mathbb{1} [ s(c) < s(c_{\mathrm{best}})-\epsilon ]\]
    • where \(\epsilon\) should be chosen based on repeated baseline runs. When experiments are expensive, the system can use single runs for exploration and repeated runs only for frontier candidates.

Budget integrity

  • Budget integrity means every candidate receives the same allowed resources. Without this, the agent may discover ways to appear better by training longer, evaluating less data, retrying more often, or using a larger context budget.

    • Wall-clock budget: Training candidates should run for the same measured duration, with clear rules for whether compilation and startup are included.
    • Token budget: Training candidates should report how many tokens they actually processed, and harness candidates should report how many input and output tokens they used.
    • Tool budget: Agentic harnesses should have explicit limits on shell calls, retries, web calls, file reads, or subprocesses when those costs matter.
    • Evaluation budget: Every candidate should evaluate on the same task set, validation tokens, or benchmark split unless the run is explicitly marked as a partial diagnostic run.
    • Retry budget: Harnesses should not silently improve pass rate by adding unbounded retries; retries should be counted and capped.

  • A candidate should be excluded from the leaderboard if it violates budget assumptions, even if the headline metric improves.

Failure taxonomy

  • Failures should be classified, not merely discarded. Failed runs are valuable because they define unstable regions of the search space and help future agents avoid repeated mistakes.

    • Syntax or import failure: The candidate cannot load, so the problem is implementation correctness rather than research quality.
    • Shape failure: Tensor dimensions, parser schema, or tool-output formats are inconsistent, which usually means the edit was not integrated carefully.
    • Out-of-memory failure: The candidate exceeded hardware limits, which may indicate that the idea needs a smaller batch, shorter sequence length, checkpointing, or a narrower model.
    • NaN or Inf failure: The candidate became numerically unstable, which often points to learning rate, precision, normalization, initialization, gradient clipping, or optimizer partitioning.
    • Timeout failure: The candidate exceeded runtime limits, which can happen because of inefficient code, excessive tool loops, slow retrieval, or too many retries.
    • Metric-integrity failure: The run did not produce a trustworthy metric, reused a stale result, changed evaluator files, or produced incomplete outputs.
    • Leakage failure: The candidate accessed forbidden labels, task IDs, held-out answers, or changed the data path in a way that invalidates comparison.

  • This taxonomy gives the next proposal a useful map. A candidate with a good idea but an out-of-memory failure may deserve a scaled-down retry. A metric-integrity failure should be rejected regardless of apparent performance.

Trace evaluation

  • Trace evaluation explains why a candidate succeeded or failed. In harness search, aggregate score is often too compressed because a harness can fail through retrieval, reasoning, parsing, tool use, memory update, or timeout. The evaluator should therefore produce task-level traces.

  • Each trace should answer several questions:

    • What did the model see: The trace should preserve the prompt, retrieved context, memory state, and system instructions used for the model call.
    • What did the model do: The trace should preserve model outputs, tool calls, intermediate answers, retries, and state updates.
    • What did the harness do: The trace should preserve parser decisions, verifier decisions, routing decisions, retrieval rankings, and stopping conditions.
    • What failed: The trace should label the dominant failure type when possible, such as retrieval miss, invalid format, wrong reasoning, timeout, tool error, or verifier rejection.
    • What did it cost: The trace should record context tokens, model calls, tool calls, latency, and any retry count.

  • This is the evaluation-side reason that full-history access matters. A proposer can only perform meaningful credit assignment if the evaluator leaves behind enough evidence to distinguish superficially similar failures.

Benchmark selection

  • The right benchmark depends on the capability being optimized. A single-GPU language-model substrate needs a held-out text validation set and a stable BPB computation. A coding-agent substrate needs tasks with executable tests. A terminal-agent substrate needs full environments, setup scripts, expected outcomes, and verification. SWE-agent by Yang et al. (2024) is relevant because it shows that agent-computer interface design can materially affect software-engineering agent performance, so evaluation should include the interface and tools rather than only the base model’s text output.

  • Benchmark choice should follow these principles:

    • Use tasks that expose the bottleneck: If the research question is retrieval, use tasks where retrieved evidence actually matters; if the question is tool use, use tasks requiring real tool interaction.
    • Use executable verification when possible: Unit tests, formal checks, exact-match answer checkers, and deterministic graders reduce ambiguity and make autonomous evaluation easier.
    • Include easy and hard tasks: Easy tasks catch regressions in basic functionality, while hard tasks provide gradient for frontier improvement.
    • Track per-family results: Aggregate metrics can hide regressions on minority task families, so benchmarks should record subgroup performance when possible.
    • Avoid tiny search sets: A small repeated search set invites overfitting, especially when the proposer can inspect detailed traces.

Evolutionary evaluation

  • Autoresearch is closely related to evolutionary program search because candidates are proposed, evaluated, selected, and mutated. AlphaEvolve by Novikov et al. (2025) is relevant because it describes an autonomous coding-agent pipeline that improves algorithms by directly editing code and receiving evaluator feedback. OpenEvolve is relevant as an open-source evolutionary coding-agent framework for generating, mutating, evaluating, and selecting code candidates.

  • The evaluator in such systems should support three comparison modes:

    • Parent comparison: The candidate should be compared against the exact parent it modified, which helps identify whether the local edit helped.
    • Best-so-far comparison: The candidate should be compared against the global best, which determines whether it should become the new default.
    • Frontier comparison: The candidate should be compared against the Pareto frontier, which determines whether it offers a new tradeoff even if it is not the best on the headline metric.

Reporting

  • A useful autoresearch report should make the improvement auditable. It should not only say that the best candidate improved the metric; it should show the search trajectory, the frontier, the winning diff summary, and the failure analysis.

  • A strong report should include:

    • Progress curve: The report should show best-so-far score over evaluated candidates, with invalid runs and partial runs visually distinguished from valid full evaluations.
    • Leaderboard: The report should list the top candidates, their parents, metrics, costs, failure rates, and links to source snapshots.
    • Pareto frontier: The report should show the tradeoff between quality and cost, such as BPB versus throughput or accuracy versus context tokens.
    • Ablations: The report should isolate which component of the winning candidate mattered, especially when the winning diff combined multiple changes.
    • Robustness checks: The report should repeat the best candidate under additional seeds, task subsets, or held-out tasks when feasible.
    • Leakage audit: The report should document that evaluator files, validation data, task labels, and held-out answers were not modified or accessed improperly.
    • Trace examples: The report should include representative successes and failures so humans can verify that the system improved for the intended reason.

  • The evaluator is therefore not just a scoring script. It is the foundation that makes autonomous research cumulative, comparable, and trustworthy.

Memory

  • Autoresearch depends on memory because the value of each experiment is not only its final score. Each run contributes evidence about what works, what fails, what is unstable, and which ideas deserve to be revisited. Without durable memory, an agent is just sampling code edits. With durable memory, it becomes a cumulative research process.

Archive

  • The archive is the permanent record of the search. It should store every valid, invalid, crashed, and partial candidate in a way that future agents can inspect. The archive should be append-only by default because overwriting old runs destroys the causal trail needed for debugging.

  • A useful archive should include:

    • Candidate source: Each run should preserve the exact editable artifact that produced the result, such as the full training file, harness file, prompt builder, retrieval policy, or tool-orchestration code.
    • Parent diff: Each run should include a patch against its parent so that the agent can identify what changed without re-reading entire files.
    • Metrics: Each run should include structured metrics such as validation BPB, task accuracy, pass rate, context tokens, throughput, latency, memory use, and failure status.
    • Logs: Each run should retain stdout, stderr, warnings, stack traces, model-call logs, tool-call logs, and evaluator messages.
    • Traces: Harness runs should store prompts, retrieved context, model outputs, parser decisions, tool observations, state updates, retries, and final answers.
    • Notes: Each run should include a concise postmortem with the hypothesis, intervention, result, interpretation, and suggested next step.
    • Integrity metadata: Each run should record evaluator hashes, data hashes, run start time, run end time, hardware metadata, package versions, and random seeds when applicable.

  • This archive layout supports both automated inspection and human review. It also mirrors the key Meta-Harness idea: the proposer should access a large filesystem of prior code, scores, and execution traces selectively rather than receiving only compressed summaries.

Ledgers

  • The ledger is the compact index over the archive. It should be small enough for the agent to read frequently and structured enough for automated sorting. The archive holds the full evidence; the ledger holds the searchable map.

  • A good ledger row should include:

    • Run identity: The row should record a unique run ID, parent ID, candidate family, timestamp, and agent identity when multiple agents are involved.
    • Primary result: The row should record the main score, such as validation BPB, accuracy, pass rate, or reward.
    • Cost profile: The row should record wall-clock time, tokens processed, context tokens, model calls, tool calls, peak memory, and throughput.
    • Status: The row should distinguish evaluated runs from syntax errors, import errors, NaN failures, out-of-memory failures, timeouts, parser failures, leakage failures, and metric-integrity failures.
    • Change summary: The row should include a short description of the edit, such as “reduced depth and increased batch size,” “added parser repair,” or “changed retrieval query construction.”
    • Pointers: The row should link to the source snapshot, diff, metrics file, trace directory, logs, and notes.

  • The ledger can be treated as the agent’s high-level memory:

    \[L_t = \{(i, p_i, s_i, q_i, \kappa_i, \rho_i)\}_{i=1}^{t}\]
    • where \(i\) is the run ID, \(p_i\) is the parent ID, \(s_i\) is the main score, \(q_i\) is the cost vector, \(\kappa_i\) is the edit family, and \(\rho_i\) points to the full record.

Traces

  • Traces are the most important memory object for harness optimization. A scalar score can say that a harness failed, but the trace explains whether it failed because retrieval missed the right example, the model reasoned incorrectly, the parser rejected a correct answer, a tool call timed out, or the harness stopped too early.

  • A trace should preserve the causal chain:

    \[x \rightarrow H \rightarrow p_0 \rightarrow y_0 \rightarrow a_0 \rightarrow o_0 \rightarrow s_1 \rightarrow \dots \rightarrow r\]
    • where \(x\) is the task, \(H\) is the harness, \(p_t\) is the model input at step \(t\), \(y_t\) is the model output, \(a_t\) is an action or tool call, \(o_t\) is the observation, \(s_t\) is harness state, and \(r\) is the reward.

  • Trace fields should include:

    • Task metadata: The trace should identify the task family, difficulty, input length, expected output format, and evaluation rubric without exposing held-out labels to the proposer when that would create leakage.
    • Context construction: The trace should store the system prompt, developer instructions, examples, retrieved passages, memory summaries, and any schema shown to the model.
    • Model interaction: The trace should preserve model outputs, sampling parameters, retry attempts, stop reasons, and token counts for each model call.
    • Tool interaction: The trace should preserve commands, arguments, outputs, exit codes, timeouts, and files touched by tool calls.
    • State updates: The trace should log how memory, scratch state, retrieved context, intermediate answers, and verifier state changed after each step.
    • Parsing and verification: The trace should record parser decisions, repair attempts, verifier calls, extracted answers, and final grading status.
    • Costs: The trace should record latency, context tokens, output tokens, number of calls, and any budget violations.

  • MemGPT: Towards LLMs as Operating Systems by Packer et al. (2023) is relevant because it frames memory management as an explicit systems problem for LLM agents, which is the same kind of design pressure that trace-rich autoresearch systems expose when optimizing harness behavior.

Summaries

  • Summaries are useful for navigation, but they should never replace raw evidence. A summary can help the agent decide which run to inspect, while the raw trace lets it verify what actually happened.

  • A good memory system should maintain three summary levels:

    • Run summary: Each candidate should have a short note describing the hypothesis, edit, result, and interpretation.
    • Family summary: Each edit family should have a rolling summary of patterns, such as which optimizer changes improved BPB, which retrieval changes caused regressions, or which parser changes reduced invalid outputs.
    • Frontier summary: The best candidates should have a concise comparison that explains why each candidate remains on the frontier and what tradeoff it represents.

  • The risk is that summaries compress away the exact diagnostic signal needed for credit assignment. Meta-Harness explicitly argues that short feedback templates, scalar scores, and compressed summaries are poorly matched to harness engineering because they remove information needed to connect downstream failures to earlier harness decisions.

Retrieval

  • As the archive grows, the agent cannot read everything. It needs retrieval over prior experiments. Retrieval should combine structured queries over the ledger with text search over logs, notes, diffs, and traces.

  • Useful retrieval modes include:

    • Best-run retrieval: The agent should quickly retrieve the current best candidate, the best candidate by cost-adjusted score, and the best candidate in each edit family.
    • Regression retrieval: The agent should retrieve candidates that worsened the metric after similar edits, because regressions often explain what not to repeat.
    • Failure retrieval: The agent should search for repeated failure types such as NaNs, out-of-memory errors, parser failures, invalid JSON, stale metrics, or timeouts.
    • Similarity retrieval: The agent should retrieve prior candidates with similar diffs, architectures, optimizer settings, retrieval policies, or prompt templates.
    • Task-level retrieval: For harness search, the agent should retrieve all traces for a task family or all examples where a candidate failed but a baseline succeeded.
    • Anomaly retrieval: The agent should retrieve runs with suspicious improvements, unusual throughput, unexpectedly low context use, or mismatched evaluator hashes.

  • Retrieval can be formalized as selecting a subset of evidence \(E_t\) from the full archive \(\mathcal{D}_t\):

    \[E_t = \mathrm{Retrieve} \left( q_t,\mathcal{D}_t,k \right)\]
    • where \(q_t\) is the agent’s current diagnostic query and \(k\) controls the amount of evidence retrieved. The important design choice is that the agent can issue new queries interactively rather than being forced to accept a fixed summary chosen by the outer loop.

Credit

  • Memory exists to support credit assignment. The agent needs to infer which changes caused an improvement or regression. This is difficult because a single candidate may modify several interacting components.

  • A practical credit-assignment workflow should include:

    • Parent comparison: The agent should compare each candidate against its direct parent so that the local effect of the edit can be estimated.
    • Sibling comparison: The agent should compare candidates that share the same parent but differ in one component, which is useful for isolating effects.
    • Ablation comparison: When a winning candidate changes multiple components, the system should run follow-up candidates that remove or isolate each component.
    • Trace comparison: For harnesses, the agent should compare per-task traces before and after the edit, not only aggregate scores.
    • Cost comparison: The agent should check whether a quality gain came from more computation, more context, more retries, or genuinely better behavior.
    • Failure-mode comparison: The agent should compare how the distribution of errors changed, such as fewer parse failures but more reasoning failures.

  • If a candidate changes components \(a\), \(b\), and \(c\), the improvement cannot be safely attributed to the whole bundle. A minimal ablation plan evaluates \(H_{abc},\quad H_{ab},\quad H_{ac},\quad H_{bc},\quad H_a,\quad H_b,\quad H_c\) when the budget allows. In practice, autoresearch systems can use cheaper partial ablations first and reserve full evaluations for candidates that remain promising.

Curves

  • Curves are more informative than final scores. A final validation BPB might hide that a candidate learned faster early but plateaued worse, or learned slowly but was still improving at the time limit.

  • For training-code autoresearch, useful curves include:

    • Training loss over time: This curve shows whether a candidate is learning smoothly, diverging, or underfitting.
    • Validation BPB over time: This curve reveals whether the final score is stable or whether the candidate was lucky at the last checkpoint.
    • Tokens per second over time: This curve catches warmup effects, compiler effects, dataloader stalls, and memory-related slowdowns.
    • Gradient norm over time: This curve helps diagnose instability, optimizer misconfiguration, or overly aggressive learning rates.
    • Learning rate over time: This curve verifies that schedule changes behave as intended under different step counts.

  • For harness autoresearch, useful curves and series include:

    • Reward by task order: This series reveals whether failures cluster by task family, difficulty, or context length.
    • Context tokens by task: This series catches candidates that improve quality by spending much more context on hard examples.
    • Model calls by task: This series shows whether retries or decomposition loops are driving gains.
    • Failure type by iteration: This series reveals whether the harness is trading one failure mode for another.
    • Latency by task: This series identifies long-tail tasks where the harness gets stuck.

Compression

  • As memory grows, the system needs compression without losing auditability. The right pattern is layered compression: raw evidence remains on disk, while derived summaries, embeddings, and indexes make it easier to navigate.

  • A good compression strategy should include:

    • Immutable raw logs: Raw evidence should remain available so that summaries can be checked.
    • Structured extraction: The system should extract metrics, statuses, failure types, token counts, and costs into machine-readable tables.
    • Text summaries: The system should create concise run summaries and family summaries for quick review.
    • Search indexes: The system should index diffs, notes, logs, prompts, and traces so the agent can search by concept or keyword.
    • Frontier views: The system should maintain a compact view of Pareto-optimal candidates and why they matter.
    • Staleness markers: Summaries should record which raw evidence they were derived from so the system knows when they are outdated.

  • Compression should preserve reversibility at the decision level. That means every summary claim should point back to the run IDs, trace IDs, or metrics that support it.

Sharing

  • In multi-agent autoresearch, memory is shared infrastructure. The system should prevent agents from duplicating work, overwriting results, or drawing conclusions from incomplete runs.

  • Shared memory should support:

    • Reservations: Agents should claim run IDs and parent candidates before editing so that concurrent work does not collide.
    • Run states: Each run should have a state such as planned, running, evaluated, invalid, rejected, or frontier.
    • Locks: The evaluator should lock active run directories while writing metrics and logs.
    • Conflict detection: The system should detect when two agents propose near-identical edits or branch from stale parents.
    • Frontier updates: A curator or automated process should update the frontier only after metric integrity checks pass.
    • Notifications: Agents should be able to see when a new best candidate appears so they can rebase future work.

  • A shared archive turns independent agents into a research organization. The shared memory is the medium through which agents coordinate, avoid repeated mistakes, and compound discoveries.

Retention

  • Not all memory has the same long-term value, but deletion should be conservative. Failed runs often become useful later when a new agent wants to understand instability boundaries.

  • A practical retention policy should be:

    • Keep all frontier candidates permanently: These are the best known tradeoffs and should remain reproducible.
    • Keep all suspicious improvements permanently: These are needed for audit, even if later rejected.
    • Keep representative failures: The archive should preserve examples of each failure type so agents can learn from them.
    • Downsample redundant failures: If hundreds of candidates fail with the same syntax error or identical out-of-memory trace, the system can compress them into a family summary while retaining representative raw logs.
    • Preserve parent chains: Any candidate that influenced a frontier candidate should be kept because it supports provenance.
    • Separate cold storage from active memory: Older raw logs can move to cold storage, while summaries and indexes stay active.

  • Retention is part of scientific integrity. A system that keeps only the winners is easier to fool because it loses the negative evidence needed to understand why the winners worked.

  • Autoresearch is a search process over executable research artifacts. The search space may contain model architectures, optimizer configurations, training loops, retrieval policies, prompt builders, parser logic, tool-use policies, memory-update rules, and complete agent harnesses. The central design question is how to move through that space efficiently while preserving scientific interpretability.

Search unit

  • The search unit should be a coherent candidate that can be evaluated end to end. In single-GPU training, the candidate is usually the editable training file. In harness optimization, the candidate is the harness code that controls model calls, context construction, retrieval, parsing, and state updates. In agentic coding, the candidate may include tool policies, terminal interaction rules, retry logic, and task-submission behavior.

  • A candidate can be represented as:

    \[c_t = (a_t, h_t, o_t, e_t)\]
    • where \(a_t\) is the architecture or algorithmic structure, \(h_t\) is the hyperparameter configuration, \(o_t\) is the orchestration logic, and \(e_t\) is the evaluator-facing interface. The evaluator should only accept candidates that preserve \(e_t\), because changing the interface or metric contract makes candidates incomparable.

  • A good search unit should satisfy the following properties:

    • Executable: The candidate should be runnable by the evaluator without manual intervention, and invalid candidates should fail early through validation gates rather than during expensive evaluation.
    • Comparable: The candidate should be evaluated under the same metric, budget, task set, and interface contract as other candidates in its comparison group.
    • Inspectable: The candidate should be stored as source code plus a diff against its parent so future agents can understand exactly what changed.
    • Attributable: The candidate should test one coherent hypothesis or a tightly coupled bundle of changes, which makes it easier to infer why the result changed.
    • Reusable: Useful subcomponents, such as a better schedule, safer parser, or faster retrieval filter, should be easy to transplant into future candidates.

Search modes

  • Autoresearch should mix several search modes because different stages of the research process require different kinds of exploration, as indicated below:

    • Local refinement: The agent makes a small edit to the current best candidate, such as adjusting a learning rate, changing a retry threshold, reducing parser strictness, or modifying the prompt format. This mode is efficient when the current design is already strong and failures are localized.
    • Branching exploration: The agent starts from a non-best candidate that contains an interesting idea, such as a faster architecture that underperformed because of an unstable schedule. This mode prevents promising mechanisms from being discarded too early.
    • Component transplant: The agent extracts one useful subsystem from a candidate and inserts it into a stronger parent. This is useful when a candidate regressed overall but improved one measurable submetric.
    • Ablation: The agent removes or isolates a component from a winning candidate to test whether the component actually caused the improvement. This mode turns a discovered improvement into a more trustworthy result.
    • Structural rewrite: The agent replaces a whole subsystem, such as the retrieval policy, memory representation, optimizer partitioning, or tool loop. This mode is risky but necessary when traces show that the current structure is misaligned with the task.
    • Frontier expansion: The agent searches for candidates that are not best on the headline score but offer a useful tradeoff, such as lower context cost, fewer model calls, better latency, or lower memory use.

Proposal policy

  • The proposal policy decides what candidate to try next. A simple policy samples changes from the current best. A stronger policy conditions on the full archive, including failures, traces, diffs, and frontier candidates.

  • The proposal distribution can be written as:

    \[c_{t+1} \sim \pi_{\phi} \left( c \mid \mathcal{D}_t, \mathcal{P}_t, g \right)\]
    • where \(\mathcal{D}_t\) is the archive of prior runs, \(\mathcal{P}_t\) is the current Pareto frontier, and \(g\) is the current research goal. The policy can be implemented by a coding agent that reads the archive, forms a hypothesis, edits the candidate artifact, and records a rationale.

  • Large Language Models as Optimizers by Yang et al. (2023) is relevant because OPRO shows that language models can propose improved solutions from a history of candidate values, but autoresearch generalizes this idea from natural-language prompt candidates to executable code candidates.

Parent choice

  • Parent choice determines which prior candidate the next edit starts from. Greedy search always edits the current best candidate, but this can prematurely discard diverse mechanisms. Population-based search maintains multiple possible parents.

  • A softmax parent-selection rule is:

    \[P(i) = \frac{ \exp(-s_i/\tau) }{ \sum_j \exp(-s_j/\tau) }\]
    • where \(s_i\) is the score of candidate \(i\) and lower is better. The temperature \(\tau\) controls exploration: small \(\tau\) favors the best candidate, while larger \(\tau\) gives more probability to diverse candidates.

  • Parent choice should consider more than the headline score:

    • Score quality: Strong candidates should be sampled often because they represent the best known designs.
    • Cost profile: A candidate with slightly worse quality but much lower memory, context, or latency may be a valuable parent.
    • Novelty: A candidate from an underexplored edit family may deserve further search even if its first result was not best.
    • Failure potential: A failed candidate can be a useful parent if the failure is clearly repairable, such as out-of-memory caused by a batch-size choice rather than a flawed idea.
    • Trace evidence: A candidate that improved a specific failure class, such as parser failures or retrieval misses, may be worth extending even if aggregate accuracy did not improve.

Mutation design

  • A mutation is an edit to a candidate. In autoresearch, mutations should be semantically meaningful rather than random text changes. The best mutations are small enough to evaluate cleanly but large enough to express a real research idea.

  • Useful mutation families include:

    • Parameter mutation: The agent changes numeric values such as learning rate, warmup fraction, weight decay, batch size, sequence length, retrieval top-\(k\), parser thresholds, or retry limits.
    • Structural mutation: The agent changes model depth, attention pattern, optimizer grouping, retrieval stages, memory representation, prompt sections, tool-routing logic, or verification loops.
    • Schedule mutation: The agent changes learning-rate schedules, evaluation cadence, retry schedules, curriculum order, or context-expansion policies.
    • Interface mutation: The agent changes how information is presented to the model, such as example formatting, schema strictness, chain decomposition, retrieved-context ordering, or final-answer extraction.
    • Safety mutation: The agent adds guards such as NaN checks, parser validation, timeout handling, budget enforcement, fallback behavior, or evaluator-integrity checks.
    • Efficiency mutation: The agent changes computation layout, batching, caching, retrieval precomputation, prompt compression, or tool-call minimization.

  • AlphaEvolve by Novikov et al. (2025) is relevant because it frames LLM-based coding agents as part of an evolutionary loop that proposes direct code changes, receives evaluator feedback, and iteratively improves algorithms.

Crossover

  • Crossover combines useful components from multiple candidates. In code-space autoresearch, crossover should usually be semantic rather than line-based. The agent should identify which subsystem to transfer, preserve its dependencies, and keep unrelated behavior unchanged.

  • A good crossover workflow should include:

    • Identify donor value: The agent should explain what the donor candidate contributed, such as faster throughput, fewer parse errors, better retrieval precision, lower memory, or improved stability.
    • Identify recipient strength: The agent should explain why the recipient candidate is the stronger base, such as better overall BPB, higher pass rate, or a cleaner cost profile.
    • Transfer only the target subsystem: The agent should avoid copying unrelated prompt text, hyperparameters, or logging changes that would confound the result.
    • Preserve evaluator interfaces: The combined candidate should still satisfy the same metric contract and validation gates.
    • Compare against both parents: The result should be evaluated against the donor and the recipient so the system can tell whether the combination was additive.

  • Crossover is most useful when the archive stores enough detail to identify subsystem-level wins. This is another reason to record secondary metrics and traces rather than only final scores.

Selection

  • Selection decides which candidates remain active. Greedy selection only keeps the best candidate, but autoresearch usually benefits from retaining multiple candidates with different strengths.

  • The active set can be defined as:

\[A_t = \mathcal{P}_t \cup B_t \cup N_t\]

  • where \(\mathcal{P}_t\) is the Pareto frontier, \(B_t\) is a small set of high-quality backups, and \(N_t\) is a small set of novel or underexplored candidates. This prevents the search from collapsing too early.

  • Selection should preserve:

    • The current best: The strongest candidate under the primary metric should always remain available as a safe parent and deployment baseline.
    • Frontier candidates: Candidates that represent different quality-cost tradeoffs should remain active even when they are not best on the headline metric.
    • Promising failures: Candidates with repairable failures should remain indexed, especially if they introduced a valuable component.
    • Diverse edit families: The active set should include different mechanisms so the search can recover from local optima.
    • Recent probes: A few recent candidates should remain easy to inspect because they carry fresh evidence about the current search region.

  • GEPA by Agrawal et al. (2025) is relevant because it combines genetic prompt evolution, natural-language reflection, and Pareto-based selection, which maps naturally onto autoresearch when the optimized artifact is executable code rather than only a prompt.

Exploration control

  • Search should start broad and become more exploitative as evidence accumulates, while still reserving some budget for surprises. A simple annealed exploration probability is:

    \[P(\mathrm{explore}) = p_{\min} + (p_0 - p_{\min}) e^{-t/\tau}\]
    • where \(p_0\) is the initial exploration rate, \(p_{\min}\) is the long-run exploration floor, and \(\tau\) controls how quickly exploration decays.

  • Exploration should increase when the search stagnates. For example, if no candidate improves the frontier for \(K\) iterations, the system can increase novelty pressure:

    \[S(c) = Q(c) + \lambda_n \mathrm{Novelty}(c) - \lambda_r \mathrm{Risk}(c)\]
    • where \(Q(c)\) is expected quality, \(\mathrm{Novelty}(c)\) measures distance from prior candidates, and \(\mathrm{Risk}(c)\) estimates invalid-run probability.

  • A practical exploration policy should include:

    • Early broad search: The first phase should test several edit families to identify which mechanisms matter most.
    • Middle focused search: The system should allocate more budget to edit families with repeated evidence of improvement.
    • Stagnation recovery: If the best score does not move for several iterations, the agent should inspect failures and try a more structural change.
    • Frontier search: The agent should occasionally target lower cost, higher speed, or better robustness rather than only headline quality.
    • Periodic ablation: The system should spend some budget verifying which components of the best candidate actually matter.

Bandit framing

  • Autoresearch can be framed as a bandit over edit families. Each edit family is an arm, each evaluated candidate produces a reward, and the system gradually shifts budget toward families with better returns.

  • Let \(k\) index edit families and \(\hat{\mu}_k\) be the observed mean improvement from family \(k\). An upper-confidence rule is:

    \[\mathrm{UCB}_k = \hat{\mu}_k + \alpha \sqrt{ \frac{\ln t}{n_k} }\]
    • where \(n_k\) is the number of attempts from family \(k\) and \(\alpha\) controls exploration. The agent can use this as a planning prior, while still making semantically rich edits within the chosen family.

  • This framing is useful for managing budget, but it should not replace trace-driven reasoning. A low-performing family may contain one repairable idea, and a high-performing family may stop working once its easy gains are exhausted.

Search traces

  • Search itself should be logged, not just candidate outcomes. The system should record what the agent inspected before proposing an edit, which parent it chose, which hypothesis it wrote, and why it selected that edit family.

  • A search trace should include:

    • Inspection record: The trace should record which leaderboard rows, source snapshots, diffs, logs, and task traces the agent examined before editing.
    • Parent rationale: The trace should explain why the selected parent was the right starting point for the next candidate.
    • Hypothesis: The trace should state the expected mechanism and the metric movement predicted by the agent.
    • Edit family: The trace should classify the proposal as a hyperparameter edit, structural edit, parser repair, retrieval change, prompt change, efficiency change, or another family.
    • Risk prediction: The trace should record the most likely failure modes before evaluation, such as NaNs, underfitting, timeout, context bloat, or parser brittleness.
    • Postmortem: The trace should compare the predicted outcome with the actual outcome and update what the agent should believe next.

  • These search traces make the optimizer itself debuggable. A system that repeatedly proposes risky edits without inspecting failures can be corrected by changing the instruction layer or parent-selection policy.

Baselines

  • Autoresearch should be compared against simple baselines. Otherwise, it is hard to know whether the agent is performing intelligent search or merely sampling enough candidates to get lucky.

  • Useful baselines include:

    • Best-of-\(N\): The system samples \(N\) independent candidates from the same starting point without using history, which tests whether iterative feedback matters.
    • Random local edits: The system applies simple random perturbations to hyperparameters or prompt sections, which tests whether the coding agent’s semantic edits add value.
    • Greedy hill climb: The system only edits the current best candidate and accepts improvements, which tests whether population diversity matters.
    • Scores-only optimizer: The proposer sees candidate code and scalar scores but not raw traces, which tests whether traces are useful.
    • Summary-only optimizer: The proposer sees code, scores, and compressed summaries but not raw logs, which tests whether summaries preserve enough diagnostic information.
    • Human baseline: A human-designed harness or training configuration provides a practical reference point for whether the autonomous loop is competitive.

  • OpenEvolve is relevant because it provides an open-source evolutionary coding-agent framework for generating, mutating, evaluating, and selecting program candidates, making it a useful point of comparison for code-space autoresearch systems.

Stopping

  • The search loop should stop for clear reasons rather than drift indefinitely. Stopping criteria can be budget-based, performance-based, or reliability-based.

  • A good stopping policy should include:

    • Evaluation budget: The search should stop after a fixed number of valid evaluations, wall-clock hours, GPU-hours, or model-call budget.
    • Stagnation: The search should stop or change mode when the frontier has not improved for a specified number of iterations.
    • Noise floor: The search should stop treating improvements as meaningful when changes are smaller than the estimated metric noise.
    • Failure rate: The search should pause for instruction or validation redesign if invalid candidates exceed a threshold.
    • Audit concerns: The search should stop if suspicious improvements, evaluator drift, leakage, or budget violations appear.
    • Deployment readiness: The search can stop when the frontier includes a candidate that satisfies the required quality, cost, latency, memory, and robustness constraints.

  • The final output of search should not be only the best candidate. It should include the best candidate, the frontier, the ablations, the failure analysis, the audit report, and the reusable lessons discovered during the run.

Safety

  • Autoresearch gives an agent the ability to change executable research code, so safety is not an optional add-on. The core safety question is whether the system can improve the target metric without invalidating the experiment, leaking answers, wasting compute, corrupting state, or producing changes that humans cannot audit.

Invariants

  • The safest autoresearch systems define a small set of invariants that every candidate must preserve. These invariants should be enforced by code, not only by instructions.

    • Metric invariance: The candidate should not change the metric definition, validation split, benchmark grader, result parser, byte accounting, or success criteria unless the run is explicitly marked as an evaluator migration rather than a candidate comparison.
    • Budget invariance: The candidate should not train longer, evaluate fewer examples, use unbounded retries, exceed the model-call budget, or silently skip expensive cases.
    • Data invariance: The candidate should not alter training data, validation data, held-out tasks, task labels, answer files, or contamination filters.
    • Interface invariance: The candidate should preserve the evaluator-facing function signatures, result schema, required output files, and expected error behavior.
    • Archive invariance: The candidate should not delete, rewrite, or obscure previous results because the search history is the evidence base for later conclusions.
    • Permission invariance: The candidate should only modify editable artifacts and should treat frozen infrastructure, previous runs, and held-out data as read-only or forbidden according to the experiment contract.

  • These invariants are the difference between autonomous research and unconstrained self-modifying code. They let the agent explore aggressively while keeping comparisons meaningful.

Sandboxing

  • The evaluator should run candidates in a sandboxed environment. The sandbox does not need to be elaborate at first, but it should prevent candidate code from modifying the wrong files or accessing forbidden resources.

  • A practical sandbox should provide the following constraints:

    • Fresh working copy: Each candidate should run in a clean copy of the repository so failed edits, generated files, and local caches do not contaminate later runs.
    • Read-only frozen files: Data preparation, validation manifests, metric code, and benchmark inputs should be mounted read-only when possible.
    • Scoped write access: Candidate code should write only to its assigned run directory, temporary directory, and expected metric output path.
    • Network policy: Network access should be disabled unless it is part of the task, because online access can introduce leakage, nondeterminism, or dependency drift.
    • Resource limits: The sandbox should enforce GPU memory expectations, wall-clock limits, process limits, file-size limits, and retry limits.
    • Environment capture: Each run should record package versions, hardware type, driver version, environment variables, and commit hashes so later reviewers can reproduce or explain behavior.

  • Sandboxing is especially important for harness optimization because a harness may call tools, read files, spawn subprocesses, or invoke other models. The search system should treat tool access as part of the candidate’s budget and permissions, not as an invisible implementation detail.

Integrity checks

  • Integrity checks make false progress visible. The evaluator should check that the candidate improved the intended system rather than weakening the measurement apparatus.

  • A strong integrity layer should include:

    • File hashes: The evaluator should hash frozen files such as data preparation, validation data manifests, task files, metric code, and grading scripts before each run and reject candidates when hashes differ unexpectedly.
    • Metric freshness: The evaluator should verify that the result file was created during the current run, contains the current run ID, and was not copied from a previous run.
    • Task count checks: The evaluator should verify that the candidate evaluated the expected number of validation examples, benchmark tasks, or validation tokens.
    • Budget checks: The evaluator should verify that wall-clock time, token budget, context budget, retry budget, and tool-call budget are within allowed limits.
    • Output schema checks: The evaluator should verify that metrics, traces, and final answers match the expected schema so downstream comparisons do not silently parse malformed outputs.
    • Diff checks: The evaluator should inspect changed files and reject candidates that modified forbidden paths or smuggled evaluation logic into editable code.
    • Leakage checks: The evaluator should scan candidate code, prompts, retrieval indexes, and generated traces for held-out answer strings, benchmark task IDs, forbidden labels, or explicit test-set shortcuts.

  • A useful hash-based guard is:

\[h_{\mathrm{frozen}} = \mathrm{SHA256} \left( F_{\mathrm{data}} \parallel F_{\mathrm{metric}} \parallel F_{\mathrm{tasks}} \parallel F_{\mathrm{parser}} \right)\]

  • A run is leaderboard-eligible only if:

    \[h_{\mathrm{frozen}}^{(t)} = h_{\mathrm{frozen}}^{(0)}\]
    • unless a human explicitly declares a new evaluation version and resets the comparison group.

Leakage

  • Leakage is any path by which the search process uses information that should not be available at proposal time. It can be obvious, such as reading held-out labels, or subtle, such as repeatedly tuning on a public benchmark until the harness encodes benchmark-specific quirks.

  • Leakage prevention should cover several surfaces:

    • Data leakage: The agent should not access held-out labels, test solutions, hidden graders, or answer files while proposing candidates.
    • Trace leakage: The proposer should not see test-set traces during search, because traces reveal failure-specific information that can guide overfitting even when labels are hidden.
    • Retrieval leakage: Retrieval corpora should be deduplicated against evaluation tasks, especially for math, coding, and question-answering settings where near-duplicate solutions can make retrieval look artificially strong.
    • Prompt leakage: Prompts and harness code should be scanned for task IDs, exact expected answers, hidden-label strings, or benchmark-specific conditional branches.
    • Leaderboard leakage: Public benchmark iteration should be reported honestly as benchmark-specific discovery rather than clean held-out generalization.

  • A clean protocol keeps search traces and final-test traces separate. The proposer can inspect search-set failures as much as needed, but final test results should be used only after candidate selection.

Reproducibility

  • An autoresearch result should be reproducible enough that a human can rerun the winning candidate and understand why it was selected. Full determinism is not always possible on GPU workloads or stochastic model calls, but the system should record enough state to make reruns meaningful.

  • A reproducibility record should include:

    • Source snapshot: The exact candidate code, parent code, and patch should be stored permanently.
    • Environment: The system should record package versions, Python version, CUDA version, GPU type, driver version, model version, tokenizer version, and relevant environment variables.
    • Randomness: The evaluator should record random seeds, sampling settings, data-ordering settings, and whether deterministic kernels were enabled.
    • Budget: The run should record wall-clock budget, actual elapsed time, tokens processed, optimizer steps, context tokens, model calls, and retries.
    • Data identity: The run should record dataset hashes, validation manifest hashes, retrieval-corpus hashes, benchmark task versions, and contamination-filter versions.
    • Metric identity: The run should record metric-code hashes and result-schema versions.
    • Trace identity: Harness runs should preserve the prompts, outputs, tool calls, parser decisions, and final grading records needed to reproduce task-level outcomes.

  • For stochastic systems, the final report should include repeated evaluations of frontier candidates when feasible. A candidate that wins once by a tiny margin may not be reliable enough to deploy or claim as a discovery.

Human audit

  • Autoresearch should move humans from manual iteration to oversight, not remove them from the scientific loop. Human review is especially valuable when a candidate improves the metric in an unexpected way, changes a large amount of code, or introduces a new mechanism that the evaluator was not designed to police.

  • A human audit should ask:

    • Was the metric contract preserved: The reviewer should confirm that the candidate did not change validation data, metric code, task selection, byte accounting, result parsing, or budget enforcement.
    • Is the improvement larger than noise: The reviewer should compare the gain to repeated baseline variation and rerun the candidate if the margin is small.
    • Did the candidate improve for the intended reason: The reviewer should inspect traces, curves, and ablations to distinguish true capability improvements from increased compute, extra context, more retries, or parser shortcuts.
    • Is the code general: The reviewer should look for hard-coded task IDs, brittle if-statements, answer strings, benchmark-specific heuristics, or changes that only work on one machine.
    • Is the candidate maintainable: The reviewer should check whether the winning code is readable, modular enough to reuse, and compatible with future experiments.
    • Are failures understood: The reviewer should inspect both successes and representative failures so the system’s limits are not hidden by an aggregate score.

  • Human audit should be triggered automatically for unusually large improvements, candidates that modify many lines, candidates that touch near-forbidden paths, and candidates that change the cost-quality tradeoff substantially.

Security

  • An autoresearch system runs model-written code, so it should be treated as an untrusted-code execution environment. Even benign agents can generate harmful behavior accidentally, such as deleting logs, exhausting disk, spawning runaway processes, or leaking credentials through tool outputs.

  • A secure deployment should include:

    • Credential isolation: API keys, cloud credentials, private datasets, and personal files should not be exposed to candidate code unless strictly required.
    • Process limits: Candidate runs should have limits on subprocess count, runtime, CPU, memory, GPU memory, disk writes, and open files.
    • Filesystem isolation: Candidate code should not have write access outside the run directory and should not be able to modify the archive or evaluator.
    • Network isolation: Network access should be disabled by default and granted only for tasks that explicitly require it.
    • Dependency control: Candidate code should not freely install arbitrary packages during evaluation unless the sandbox captures and approves dependency changes.
    • Log redaction: Logs should avoid storing secrets, credentials, personal data, or private file paths that future agents could read.
    • Kill switches: The orchestration layer should be able to stop runaway jobs, disable an agent, freeze the archive, and mark the current search as tainted.

  • The more capable the proposer, the more important these controls become. Capability increases the chance of useful discoveries, but it also increases the need for hard boundaries.

Robustness

  • A winning candidate should be tested beyond the exact setting that produced it. Robustness checks distinguish real improvements from overfit artifacts.

  • Useful robustness checks include:

    • Seed robustness: Rerun the candidate under multiple seeds or sampling settings to estimate variance.
    • Budget robustness: Evaluate nearby budgets, such as shorter and longer runs, to see whether the improvement is a fixed-budget artifact or a generally better method.
    • Task robustness: Test on task families not used during search when possible.
    • Model robustness: For harnesses, evaluate whether the same harness helps different base models or only the model used during search.
    • Cost robustness: Check whether the candidate remains useful under stricter memory, latency, context, or tool-call budgets.
    • Ablation robustness: Remove or isolate components of the winning candidate to verify which parts are necessary.

  • A candidate is more trustworthy when it improves the main metric, survives integrity checks, generalizes beyond the search set, and has an interpretable mechanism supported by traces or ablations.

Governance

  • Autoresearch systems need governance because they can accumulate many small automated decisions into a large research result. Governance defines when the system is allowed to continue, when humans must intervene, and what evidence is required before a result is accepted.

  • A practical governance policy should specify:

    • Promotion rules: A candidate should become the new default only after passing metric integrity checks, budget checks, and minimum improvement thresholds.
    • Frontier rules: A candidate should enter the frontier when it offers a non-dominated quality-cost tradeoff and passes audit.
    • Escalation rules: Human review should be required for suspicious improvements, evaluator-adjacent diffs, leakage warnings, high-cost runs, or large structural rewrites.
    • Deprecation rules: Candidates should be removed from active consideration when they are dominated, non-reproducible, invalid, or dependent on a tainted evaluator version.
    • Reporting rules: Final results should disclose search budget, number of evaluated candidates, invalid-run rate, candidate-selection procedure, held-out evaluation protocol, and known limitations.
    • Pause rules: The search should stop automatically when leakage is detected, evaluator hashes change unexpectedly, invalid-run rates become too high, or resource use exceeds limits.

  • Safety is therefore not a separate stage after search. It is part of the search algorithm. The evaluator, sandbox, archive, auditor, and governance rules together define which discoveries count as valid.

Scaling

  • Autoresearch can begin with one agent, one GPU, one editable file, and one metric, but the long-term pattern is a distributed research organization: many agents propose candidates, shared evaluators score them, an archive stores evidence, and frontier candidates are promoted only after audit. Scaling is therefore not only a compute problem. It is a coordination, memory, evaluation, and governance problem.

Parallelism

  • The simplest scaling move is to run multiple candidates in parallel. Parallelism increases search throughput, but it also introduces coordination problems: agents may duplicate edits, branch from stale parents, overwrite results, or interpret incomplete runs as final evidence.

  • A parallel system should enforce the following rules:

    • Run reservation: Each agent should reserve a unique run ID, parent candidate, and edit family before modifying code so that concurrent proposals do not collide.
    • Immutable parent snapshots: Each candidate should branch from a fixed source snapshot rather than a moving working directory, which prevents one agent’s changes from silently affecting another agent’s run.
    • Evaluator queue: Candidate evaluation should go through a central queue that enforces budget, resource limits, and metric integrity checks.
    • In-flight visibility: Agents should be able to see which edit families are currently running so they can avoid duplicating work.
    • Delayed promotion: A candidate should not become the new default until its metrics are finalized, validated, and written to the archive.
    • Rebase policy: Agents should periodically rebase future proposals on the current frontier when a new best candidate appears.

  • The main objective is to preserve the scientific meaning of each run. More parallelism is only helpful if the system still knows exactly which code, parent, budget, and metric produced each result.

Scheduling

  • As the number of candidates grows, scheduling becomes a research decision. The system must decide which candidates deserve scarce GPU time, which candidates should receive cheap smoke tests only, and which frontier candidates deserve expensive confirmation.

  • A useful scheduler should support multiple queues:

    • Smoke-test queue: This queue runs syntax checks, import checks, interface checks, tiny training runs, parser checks, and budget validation before expensive evaluation.
    • Exploration queue: This queue evaluates novel candidates that test underexplored mechanisms or branch from non-best parents.
    • Exploitation queue: This queue evaluates local refinements of frontier candidates.
    • Ablation queue: This queue isolates components of high-performing candidates to verify causality.
    • Confirmation queue: This queue reruns frontier candidates under additional seeds, task subsets, or stricter budgets.
    • Audit queue: This queue runs leakage checks, evaluator-hash checks, suspicious-diff scans, and reproducibility checks.

  • A simple scheduling priority can combine expected value and cost:

    \[\mathrm{Priority}(c) = \frac{ \mathbb{E}[\Delta(c)] \cdot P(\mathrm{valid}(c)) }{ \mathrm{Cost}(c) }\]
    • where \(\mathbb{E}[\Delta(c)]\) is the expected improvement, \(P(\mathrm{valid}(c))\) is the estimated probability that the candidate passes validation, and \(\mathrm{Cost}(c)\) is expected runtime or model-call cost.

Compute budgets

  • Autoresearch should make compute budgets explicit at every level. A single candidate has a run budget. A search campaign has a total budget. A confirmation phase has a separate robustness budget. Without this separation, the agent may spend all resources on exploration and leave no budget for validation.

  • A practical budget plan should include:

    • Per-candidate budget: Each training run or harness evaluation should have a fixed wall-clock, token, task, retry, and tool-call budget.
    • Campaign budget: The full search should have a maximum number of valid evaluations, invalid evaluations, GPU-hours, model calls, or wall-clock hours.
    • Confirmation budget: The system should reserve resources to rerun top candidates, estimate variance, and run held-out evaluations.
    • Ablation budget: The system should reserve resources to test whether the winning components actually caused the improvement.
    • Audit budget: The system should reserve resources for integrity checks, leakage scans, and reproducibility tests.
    • Fallback budget: The system should reserve a small amount of compute for recovery when a promising candidate fails because of an implementation bug rather than a bad idea.

  • A useful budget allocation is:

\[B_{\mathrm{total}} = B_{\mathrm{search}} + B_{\mathrm{confirm}} + B_{\mathrm{ablate}} + B_{\mathrm{audit}}\]
  • The exact split depends on the cost of evaluation and the noise level of the metric. Cheap noisy experiments need more confirmation. Expensive deterministic experiments need stronger candidate filtering before full evaluation.

Multi-agent roles

  • Scaling works better when agents specialize. Specialization reduces conflicts and makes permissions easier to reason about.

  • A multi-agent autoresearch organization should include:

    • Proposer agents: These agents inspect the archive, write hypotheses, edit candidate artifacts, and submit candidates to the evaluator queue.
    • Debugging agents: These agents repair invalid candidates, diagnose crashes, and convert promising failed runs into valid variants.
    • Ablation agents: These agents isolate components of high-performing candidates and test whether each component is necessary.
    • Auditor agents: These agents enforce file boundaries, check metric integrity, scan for leakage, and flag suspicious improvements.
    • Curator agents: These agents maintain the leaderboard, Pareto frontier, run-family summaries, and active research agenda.
    • Documentation agents: These agents turn raw experiments into readable summaries, reproducibility records, and final reports.

  • This role separation resembles a human research group, but the coordination medium is the archive rather than meetings. Each agent leaves behind structured evidence, and the next agent acts on that evidence.

Shared context

  • The archive becomes the shared context for the research organization. As scale increases, the archive must support fast retrieval, reliable indexing, and provenance tracking. Otherwise, agents will make decisions from stale or incomplete memory.

  • Shared context should include:

    • Global leaderboard: The system should maintain a current view of best candidates, frontier candidates, invalid-run rates, and recent improvements.
    • Run-family summaries: The archive should summarize which edit families are helping, which are failing, and which remain underexplored.
    • Failure database: The archive should index NaNs, out-of-memory failures, parser failures, timeouts, leakage warnings, and metric-integrity failures.
    • Component registry: Useful subcomponents, such as a stable optimizer group, parser repair routine, retrieval filter, or prompt template, should be recorded as reusable modules.
    • Open questions: The curator should maintain a list of unresolved hypotheses, such as whether a gain came from throughput, capacity, retrieval relevance, parser repair, or extra context.
    • Tainted runs: The archive should clearly mark runs that are invalid, non-comparable, leakage-suspect, or produced under a deprecated evaluator.

  • The shared context should always distinguish facts from interpretations. A metric is a fact produced by an evaluator. A postmortem is an interpretation. A family summary is a compressed interpretation over many runs. Agents should be able to trace summaries back to raw evidence.

Model diversity

  • A scaled autoresearch system can use different models for different roles. Stronger models may be better at structural rewrites and trace diagnosis. Smaller models may be sufficient for formatting, log parsing, smoke-test repair, or summarization.

  • A model-diverse setup should use:

    • Frontier proposer models: Strong models should be reserved for high-leverage design decisions, structural rewrites, difficult debugging, and final synthesis.
    • Cheap maintenance models: Smaller models can update ledgers, summarize logs, classify failures, extract metrics, and prepare candidate reports.
    • Independent auditor models: Auditors should ideally differ from proposers so that the same model’s blind spots do not affect both proposal and validation.
    • Specialized retrievers: Retrieval over the archive may use lexical search, embeddings, structured filters, or hybrid retrieval depending on the evidence type.
    • Human review: Humans should remain in the loop for suspicious improvements, high-impact claims, and changes to the evaluation contract.

  • The important principle is to spend reasoning budget where it changes decisions. A frontier model does not need to parse every log line if a cheaper model or script can extract the same failure status reliably.

Transfer

  • A discovery is more valuable if it transfers beyond the exact search setup. Scaling autoresearch therefore means turning one-off wins into reusable methods, skills, and harness components.

  • Transfer should be evaluated across several axes:

    • Across seeds: The candidate should remain strong under different random seeds or sampling settings.
    • Across budgets: The candidate should still help under shorter or longer training budgets, or under stricter inference budgets.
    • Across tasks: A harness improvement should help task families not seen during search when possible.
    • Across models: A harness should ideally improve multiple base models, not only the one used during search.
    • Across hardware: Training-code improvements should be checked on different accelerators or batch-size regimes before being treated as general.
    • Across datasets: A model-training change should be tested on another dataset or distribution when feasible.

  • Meta-Harness-style results are especially interesting when a discovered harness transfers across held-out models or out-of-distribution tasks, because that suggests the search found a reusable procedure rather than a benchmark-specific shortcut.

Cost control

  • Scaling without cost control leads to waste. Agents may run redundant experiments, repeatedly test invalid candidates, use excessive context, or rerun expensive evaluations before cheap checks.

  • Cost control should include:

    • Deduplication: The system should compare proposed diffs and reject near-duplicates before evaluation.
    • Progressive evaluation: Candidates should pass cheap checks before receiving full-budget evaluation.
    • Early stopping: Training runs or harness evaluations should stop early when they clearly violate constraints, diverge, or fail required interfaces.
    • Context budgets: Harnesses should have explicit context-token budgets, and candidates should be penalized or rejected when they exceed them.
    • Retry budgets: Agents and harnesses should have capped retries so they cannot convert cost into score invisibly.
    • Cache policy: Expensive retrieval indexes, compiled kernels, prepared datasets, and benchmark environments should be cached when this does not compromise comparability.
    • Invalid-rate monitoring: If too many candidates fail validation, the system should shift budget from exploration to debugging or instruction redesign.

  • A campaign-level efficiency metric is:

    \[\mathrm{SearchEfficiency} = \frac{ \max_{t \le T} \Delta s_t }{ B_{\mathrm{spent}} }\]
    • where \(\Delta s_t\) is the best improvement achieved by time \(t\) and \(B_{\mathrm{spent}}\) is the consumed budget. This metric discourages search strategies that eventually improve but burn excessive resources.

Staleness

  • At scale, information becomes stale. A run-family summary may describe an old evaluator version. A best candidate may be dominated under a new cost metric. A failure pattern may no longer apply after a structural rewrite.

  • The system should handle staleness explicitly:

    • Versioned evaluators: Every metric should be tied to an evaluator version, and candidates from different evaluator versions should not be compared without migration.
    • Versioned datasets: Data and benchmark versions should be recorded so improvements are not confused across changed task sets.
    • Versioned instructions: Agent instructions should be versioned because changes to the research policy affect the search distribution.
    • Summary timestamps: Family summaries and frontier notes should record when they were produced and which run IDs they cover.
    • Deprecation markers: Old candidates should be marked as deprecated when they are dominated, invalid under new rules, or tied to obsolete infrastructure.
    • Revalidation: Important frontier candidates should be rerun after evaluator, hardware, model, or dataset changes.

  • Staleness is not a bookkeeping issue. It directly affects whether the system’s memory is trustworthy.

Finalization

  • Scaling should end with a finalization phase. The goal is to convert a large search history into a small set of validated claims.

  • A finalization phase should include:

    • Candidate selection: Choose the best candidate or frontier candidates using only predeclared search-time criteria.
    • Clean reruns: Rerun selected candidates from clean checkouts under the final evaluator.
    • Variance estimation: Repeat evaluations when metrics are stochastic or margins are small.
    • Ablation: Remove or isolate major components of the winning candidate to verify causal contribution.
    • Held-out testing: Evaluate on held-out tasks, models, datasets, or seeds that were not visible during search.
    • Audit report: Document evaluator hashes, data versions, leakage checks, budget compliance, and suspicious-run handling.
    • Release artifact: Package the winning source code, configuration, evaluation script, run logs, and reproduction instructions.

  • The final claim should be proportional to the evidence. A candidate tuned on a public benchmark can be claimed as a strong benchmark-specific harness. A candidate validated on held-out tasks and models can be claimed as a more general method. A candidate that improves only one noisy run should be treated as a lead, not a result.

Practical blueprint

  • A practical autoresearch system should be small enough to audit, structured enough to run unattended, and strict enough that improvements remain meaningful. The blueprint below combines the single-GPU training pattern with the Meta-Harness pattern: start with a narrow editable artifact, add a durable archive, enforce evaluator invariants, then scale toward harness search and multi-agent coordination.

Minimal stack

  • The smallest useful stack has five pieces: an editable artifact, an evaluator, an archive, an agent instruction layer, and an audit layer. The editable artifact is what the agent changes. The evaluator scores the artifact under a fixed budget. The archive stores every attempt. The instruction layer tells the agent how to behave. The audit layer rejects candidates that violate the experiment contract.

  • A minimal implementation should include:

    • A fixed substrate: The system should keep data preparation, validation construction, metric computation, and result parsing outside the editable region so candidate comparisons remain valid.
    • A single editable artifact: The first version should usually let the agent edit one file, such as a model-training script or a harness implementation, because single-file diffs are easier to inspect and debug.
    • A bounded evaluator: Every candidate should receive the same time budget, token budget, task budget, context budget, or tool-call budget, depending on the domain.
    • An append-only archive: Every run should write a source snapshot, diff, metrics, logs, traces, and notes to a permanent run directory.
    • A compact ledger: The agent should be able to read a small table of all prior runs, including score, cost, status, parent, edit family, and pointers to full evidence.
    • A proposer instruction file: The agent should be told what files are editable, what success means, how to inspect prior runs, how to write hypotheses, and how to handle failures.
    • Validation gates: The system should run cheap checks before expensive evaluation, including import checks, smoke tests, interface checks, metric-path checks, and evaluator-hash checks.
    • A final audit path: Frontier candidates should be rerun cleanly, checked for leakage, and ablated before being treated as discoveries.

  • This stack is enough to convert manual iteration into an autonomous loop while keeping the system understandable.

Build order

  • Autoresearch should be built in stages. A system that tries to start with many agents, many tasks, and flexible harness search will be hard to debug. The first milestone should be a boring, reliable loop.

  • A sensible build order is:

    • Start with the frozen evaluator: First implement the data loading, task loading, metric computation, budget enforcement, and result schema. Do not add an agent until a human can run candidates reliably and compare results.
    • Add source snapshotting: Next, make every run copy the editable artifact and store a patch against its parent so that results are reproducible.
    • Add the ledger: Then create an append-only table that records run identity, parent, status, score, cost, and notes path.
    • Add validation gates: Before full evaluation, enforce syntax, import, interface, and smoke-test checks so invalid candidates fail cheaply.
    • Add a single agent: Give the agent read access to the ledger and prior runs, write access to only the editable artifact, and a clear instruction file.
    • Add trace logging: Once scalar metrics are reliable, log richer traces such as training curves, prompts, tool calls, parser outcomes, and task-level failures.
    • Add frontier tracking: Move from a single best score to a Pareto frontier when quality-cost tradeoffs matter.
    • Add ablation and confirmation: Reserve budget to rerun and ablate the best candidates so that improvements are not just noise or bundled confounds.
    • Add parallelism last: Only after single-agent runs are reliable should multiple agents or distributed evaluators be introduced.

  • This order reduces the chance that the system scales a flawed measurement setup.

Default loop

  • The default loop should be intentionally repetitive. Repetition is useful because it creates comparable evidence. Each iteration should inspect history, form a hypothesis, edit one coherent component, evaluate under the fixed contract, and write a postmortem.

  • A good default loop should require the agent to:

    • Inspect the current frontier: The agent should read the leaderboard and identify the current best candidate, relevant frontier candidates, and recent failed candidates before editing.
    • Choose a parent deliberately: The agent should state whether it is editing the current best, branching from a promising non-best candidate, transplanting a component, or performing an ablation.
    • State a hypothesis: The agent should explain the expected mechanism, expected metric movement, and main risk before changing code.
    • Make a coherent edit: The candidate should target one mechanism, such as optimizer stability, throughput, retrieval quality, parser robustness, or context reduction.
    • Run cheap checks: The agent should run validation gates before using the full budget.
    • Run full evaluation: The evaluator should execute the candidate under the fixed budget and produce structured metrics.
    • Write evidence: The system should save source, diff, logs, traces, metrics, and notes.
    • Update selection state: The system should update the leaderboard, frontier, family summaries, and failure database only after integrity checks pass.
    • Decide the next action: The agent should decide whether to refine, ablate, transplant, debug, or explore based on the result.

  • The loop can be summarized by:

    \[\mathcal{D}_{t+1} = \mathcal{D}_t \cup \left\{ c_t,\Delta_t,m_t,\ell_t,n_t \right\}\]
    • where \(c_t\) is candidate code, \(\Delta_t\) is the diff, \(m_t\) is the metric record, \(\ell_t\) is the raw evidence bundle, and \(n_t\) is the postmortem note.

Harness extension

  • Once the training-code loop works, the same pattern can be extended to harnesses. The editable artifact becomes the code that controls prompts, retrieval, memory, parsing, tool use, and state updates. The evaluator becomes a task suite. The archive stores task-level traces rather than only training logs.

  • A harness-oriented system should add:

    • A stable harness interface: The harness should expose a fixed entry point, such as running one task and returning a structured result with answer, reward, cost, trace path, and failure type.
    • Prompt logging: Every model call should save the prompt, model output, token counts, sampling settings, and stop reason.
    • Retrieval logging: Every retrieval step should save the query, retrieved items, scores, filters, and final context inserted into the prompt.
    • Memory logging: Every state update should save what was stored, compressed, dropped, or retrieved later.
    • Parser logging: Every parser decision should record extracted answers, schema errors, repair attempts, and final parse status.
    • Tool logging: Every tool call should record arguments, outputs, exit codes, touched files, timeouts, and cost.
    • Task-level scoring: The evaluator should store reward, failure type, latency, context tokens, model calls, and tool calls for each task.
    • Leakage scanning: Candidate code and traces should be scanned for task IDs, held-out answers, forbidden files, and benchmark-specific shortcuts.

  • This is where Meta-Harness becomes directly relevant: the proposer should inspect prior harness source code, task-level traces, and scores through the filesystem rather than relying on scalar rewards or short summaries.

Checklists

  • Checklists help make the loop reliable. They should be used by agents, evaluators, and human reviewers.

  • A pre-run checklist should include:

    • Editable-boundary check: Confirm that only allowed candidate files changed.
    • Interface check: Confirm that the candidate still exposes the expected functions, result schema, and output paths.
    • Smoke check: Confirm that a tiny run or a small task subset completes successfully.
    • Budget check: Confirm that time, token, context, retry, and tool-call limits are configured correctly.
    • Metric-path check: Confirm that the run will write fresh metrics to the correct run directory.
    • Frozen-hash check: Confirm that evaluator files, validation manifests, and task files match the expected hashes.

  • A post-run checklist should include:

    • Metric validity: Confirm that metrics are fresh, complete, schema-valid, and tied to the current run ID.
    • Budget compliance: Confirm that the run used the expected time, task count, validation tokens, context budget, and retry limits.
    • Failure classification: Confirm whether the run was evaluated, invalid, crashed, timed out, leaked, or violated metric integrity.
    • Evidence preservation: Confirm that source, diff, logs, traces, metrics, and notes were stored.
    • Comparison target: Confirm that the run was compared against the correct parent, current best, and frontier.
    • Audit status: Confirm whether the candidate is eligible for the leaderboard, requires review, or must be rejected.

  • A finalization checklist should include:

    • Clean rerun: Re-evaluate selected candidates from a clean checkout.
    • Variance check: Repeat evaluation when stochasticity or small margins matter.
    • Ablation check: Isolate the major components of the winning candidate.
    • Held-out check: Evaluate on held-out tasks, seeds, models, or datasets when available.
    • Leakage check: Scan source, prompts, traces, and retrieval corpora for forbidden information.
    • Release check: Package source code, configs, environment, evaluator version, metrics, traces, and reproduction instructions.

Common traps

  • Most autoresearch failures come from weak evaluation contracts, missing logs, or over-permissive agents. These traps are predictable.

  • Common traps include:

    • Changing the evaluator: A candidate that changes validation data, metric code, byte accounting, parser logic, or task selection is not comparable to earlier candidates.
    • Trusting tiny gains: A small improvement should not be treated as real until it is compared against noise from repeated runs.
    • Keeping only winners: Failed candidates contain useful evidence about unstable regions of the search space, so deleting them makes future search worse.
    • Overcompressing history: Short summaries can hide the exact trace detail needed for credit assignment.
    • Allowing unlimited retries: A harness that improves pass rate through unbounded retries is spending more budget, not necessarily becoming smarter.
    • Ignoring costs: A candidate that improves accuracy by doubling context or latency may be dominated under deployment constraints.
    • Skipping ablations: A bundled winning diff may contain one useful change and several irrelevant or harmful changes.
    • Parallelizing too early: Multiple agents amplify coordination bugs, stale parents, and duplicate work if the single-agent loop is not reliable.
    • Confusing benchmark discovery with generalization: A harness optimized repeatedly on a public benchmark may be valuable, but it should not be claimed as held-out generalization without separate evidence.

Design patterns

  • Several design patterns make autoresearch systems more robust.

    • Single editable surface: Start with one file or one harness module so diffs are inspectable and accidental evaluator changes are less likely.
    • Append-only memory: Preserve every run, including invalid candidates, because negative evidence supports future credit assignment.
    • Filesystem experience: Store full evidence on disk and let the agent retrieve selectively, rather than packing all history into one prompt.
    • Frontier over winner: Maintain a Pareto frontier so quality-cost tradeoffs are not collapsed too early.
    • Cheap checks before expensive runs: Use validation gates to reject broken candidates before consuming full compute.
    • Ablate before claiming: Treat a winning candidate as a hypothesis until its components are isolated.
    • Audit by default: Scan for metric drift, leakage, budget violations, stale results, and forbidden file edits after every candidate.
    • Version everything: Evaluator versions, dataset hashes, instruction versions, model versions, and candidate hashes should all be recorded.
    • Human review on anomalies: Large or surprising improvements should trigger human inspection before promotion.

Example rollout

  • A realistic first rollout might look like this:

    • Day one setup: A human freezes the evaluator, defines validation BPB, creates the run archive, writes the ledger schema, and verifies that several manual runs produce comparable metrics.
    • Day two single-agent loop: The agent receives permission to edit only the training file, reads the baseline, proposes small architecture and optimizer changes, runs bounded experiments, and writes postmortems.
    • Day three reliability pass: The system adds evaluator hashes, stale-metric checks, NaN classification, source snapshotting, and repeated baseline runs to estimate metric noise.
    • Day four trace enrichment: The evaluator starts storing training curves, throughput curves, gradient norms, memory usage, and richer failure labels.
    • Day five frontier search: The agent begins maintaining a Pareto frontier over BPB, throughput, memory, and parameter count rather than only one best score.
    • Day six ablation: The best candidate is decomposed into component changes, and the system tests which parts actually matter.
    • Day seven finalization: The top candidates are rerun from clean checkouts, compared against baselines, audited for evaluator integrity, and summarized into a reproducible report.

  • For a harness project, the same rollout replaces training curves with task traces, parser logs, retrieval logs, context-token accounting, and per-task reward analysis.

End state

  • The mature end state is not a black box that silently claims improvements. It is a research machine with visible evidence. It proposes code, runs experiments, stores traces, compares candidates, audits itself, and produces a frontier of validated artifacts.

  • A well-built autoresearch system should leave behind:

    • A reproducible best candidate: The final source, config, environment, and evaluator should be sufficient to rerun the winning result.
    • A Pareto frontier: The report should include alternatives that trade quality, cost, latency, memory, and robustness.
    • A complete archive: The search history should preserve both successes and failures so future work can build on the evidence.
    • A causal story: Ablations and traces should explain why the winning candidate worked.
    • A safety record: The report should document budget compliance, evaluator integrity, leakage checks, and suspicious-run handling.
    • Reusable skills: The system should extract patterns that can be reused in later campaigns, such as stable optimizer groups, parser repair strategies, retrieval filters, or audit checks.

  • The practical goal is not to replace scientific judgment. It is to make the experimental loop fast, cumulative, and inspectable enough that agents can do the repetitive search while humans supervise the claims.

References

Autoresearch core

Meta-Harness

Autoresearch extensions

Code search and evolutionary discovery

Prompt and text optimization

Agent reasoning and memory

Harnesses and LM systems

Retrieval and context

Benchmarks and evaluation

Optimization and training

Ecosystem articles

Background

Citation

If you found our work useful, please cite it as:

@article{Chadha2020DistilledAutoresearchMetaharness,
  title   = {Autoresearch and Metaharness},
  author  = {Chadha, Aman and Jain, Vinija},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}