# Don’t Learn Harness Engineering. Learn This Instead

[Watch on tuber](https://tuber.guzus.xyz/video/AKovdUFvThA/summary) · [YouTube](https://youtu.be/AKovdUFvThA)

**Score: 74/100 · recommended**

The video's answer is no: two new papers show that harness tuning can be automated, but harness work isn't going away. Both automated loops still depend on someone defining what "better" means, so the speaker says engineers should learn evals (graders) first and spend less time on specific harness tricks.

- **0:02-0:45** — **The provocation and definitions.** A 4B-parameter "proposer" model rewrote an agent's harness better than its 35B teacher did, scoring 0.62 vs 0.56 on puzzles it had never seen. Pedro Domingos read this as "the model swallows the harness," while big channels keep telling engineers to learn harness engineering. The harness is the code around a model that decides what it sees, what tools it can run and what happens next. The paper defines it as the executable program managing model calls, tool use and data flow.

- **0:45-1:52** — **Why harnesses exist.** Anthropic's post on long-running agents says each new session starts with no memory, so the agent has to rebuild its picture of the project every time. The listed failures: trying to do too much, declaring the job finished too early, marking features done without end-to-end testing, and leaving broken or undocumented state. In the example, an untested login page is marked done, the next session builds a profile page on top of it, and now two features are broken with no note explaining why. Anthropic's fix is an initializer agent, a progress file plus Git history, and a feature list. The "swallowing" claim assumes models outgrow these weaknesses. The speaker stresses that this is a prediction neither paper tests.

- **1:52-2:25** — **Harness engineering as a real job.** OpenAI's Ryan Lopopolo described (Feb 11, 2026) a small team that shipped a beta with about 1M lines of code and no handwritten source. The engineers' work shifted to designing environments, specifying intent and giving feedback. Since Domingos' post is opinion and not a test, the video turns to two papers from Sept 28 and Oct 8, 2026, each automating a different part of harness work.

- **2:25-3:50** — **Paper 1: a learned harness proposer.** A frozen "solver" does the tasks. A "proposer" reads the execution feedback (the runs and results) and rewrites the harness, and the solver's new task score becomes the proposer's reward. Training uses reinforcement learning (GRPO). The proposers are a 4B Qwen 3.5 for Reasoning Gym puzzles and a 4B Qwen 3 for Q&A. The teacher is a 35B Qwen 3.6 A3B (35B total, about 3B active per token by naming convention), which supplied training examples for the puzzle tasks only. The headline 0.62 vs 0.56 is an average for a single revision across 21 unseen puzzle families.

- **3:50-4:50** — **Further findings and limits.** When the proposer repeatedly fixes its own last version, the results approach or exceed the best of 8 independent proposals. A Q&A proposer trained on HotpotQA also improved harnesses on MuSiQue and 2WikiMultiHopQA, so the skill transfers at least within Q&A. The limits the paper states itself: the solver is fixed, humans wrote the revision prompts, and no test involves a coding agent. The key point is that the reward is task performance, and a person chose the tasks and wrote the scoring.

- **4:51-6:25** — **Paper 2: Sakana's MASS, inner loop.** One base model (27B Qwen 3.6) proposes a team of agents (the workflow), runs it, and judges the result. In the inner loop, the same model acts as evaluator: it compares the new team's workspace (files and notes) against the current champion. Then, as optimizer, it revises the workflow, and the winner of each comparison becomes the champion. The made-up example: adding a plan-reviewer agent to a draft-and-check pair replaces the old champion if the judge prefers its workspace. In the outer loop, trajectories from the best workflows are used to LoRA-fine-tune the model, and the search restarts with the updated model. Headline results come after two cycles.

- **6:25-7:52** — **MASS results and weaknesses.** Performance per output token rose 1.2–1.6× on four open-ended benchmarks. A student trained on the team's trajectories beat a single-agent student trained on 1.4× more tokens. That unit measures efficiency, not intelligence, and only one 27B model was tested. On coding it got slightly worse: Terminal-Bench 2.0 at 0.96× the base model and SWE-bench Verified at 0.94×. The paper reports two weak spots: optimizer capacity rather than evaluator capacity can be the bottleneck (the model judges teams better than it improves them), and performance can regress sharply between iterations. The code is open. Its stages are search, collect, rank, select, render, train and export, and it requires vLLM, the 27B base model and Qwen Code 0.2.0 as the runtime.

- **7:52-9:03** — **What both papers share: a signal someone defines.** Paper 1 edits only the harness, while MASS edits the team and also trains the model. Both optimize against a "better" signal without questioning it: the task score in Paper 1, and self-judgment against the champion in MASS. In the game analogy, players get faster but someone still sets the scoring, and with agents you write that scoring yourself. A bad rule gets optimized just as hard. The MASS authors warn that shared blind spots may be the Achilles' heel of recursive self-improvement, since the worker and the judge are the same model. The speaker connects this to Anthropic's "marking done without testing" failure: self-judging is the weakness that created harnesses, and MASS has to manage it.

- **9:03-10:24** — **How to build the signal: Anthropic's evals vocabulary (Jan 9, 2026).** A trial produces a transcript, and a grader scores it. Code-based graders (tests, static analysis, tool-call checks) give the same answer every run but only check what you can write as a rule. Model-based graders use rubrics, pairwise comparison (what MASS does) and multi-judge consensus. Human graders do spot checks and A/B tests, and they catch drift in the model judges. Without them, a loop can drift unseen. The guide also warns that checking exact step sequences is too brittle, so grade the outcome rather than the path. The speaker reads Lopopolo's list (environments, intent, feedback) as close to eval work, and flags that this is his interpretation.

- **10:24-11:23** — **Inference and counter-view.** The speaker's inference, which no paper measures: whether a harness is handwritten, learned by a proposer or absorbed into the model, someone still has to define "better," and that skill carries across all three cases while individual harness tricks may not. Circuit Lee's essay argues the harness moves up the stack, and the operator's job becomes judging what to trust. The Tesla story is cited by both sides: Musk said v11 had 300K+ lines of C++ and v12 had basically none (his own unaudited claim, relayed by a blog). Circuit Lee points to the bigger fleet data pipelines built afterward, which suggests the work moved rather than disappeared. The speaker personally expects harnesses to shrink, while noting that a system still needs a goal and a check on its work.

- **11:23-12:27** — **Verdict and open question.** The papers support Domingos only on one point: harness tuning can be automated on puzzles, Q&A and open-ended benchmarks. They don't support the claim that the job disappears, and the big channels are right that harness work matters today. His recommendation is to learn evals first, write a simple harness now, and put learning time into graders. He would change his view if learned harnesses beat hand-built ones on production coding agents, which hasn't been shown, and the only coding numbers so far are slightly negative. The video closes on an open question: in MASS, who samples the transcripts and checks the judge?

**Practical takeaways:**
- If you're choosing what to study, put eval and grader design ahead of specific harness tricks, because defining "better" is needed whether the harness is handwritten, learned or absorbed into the model.
- If you're building agents today, still write a simple harness: progress file, Git history, feature list and an initializer step. These address memory loss between sessions and premature "done" claims.
- First step from the video: pick one agent task, write a code grader for it, add a model grader for what code can't check, then read transcripts yourself.
- Grade outcomes rather than exact step sequences, and use human spot checks to catch drift in model judges, especially when the model doing the work also judges it.
- Don't read automated-harness results as proof for coding agents: Paper 1 had no coding tests, and MASS was slightly worse on coding benchmarks.

## Audience signal

- No high-signal audience themes available.