An AI agent moves along a track towards five arches: the first is finished and the other four are only outlined. Behind it are several red marks, joined by lines to pieces already placed beside the track, and a hand sets down the next one.

Every agent mistake, a piece of the system: harness engineering at RosmarOps

by

I build, with AI agents, a tool to use AI agents more carefully. This week its bugs were the same ones RosmarOps looks for in yours. This is how we turned each one into a permanent piece of the system.

RosmarOps exists so you can see what your AI agents can do without asking you. And I build it, almost entirely, with AI agents.

This week that paradox took a second turn. I published two versions in two days, 0.10.1 and 0.10.2, to fix bugs no test had seen. My own computer found them: it has 48 projects and almost every agent RosmarOps knows. Looking at them closely, I saw that RosmarOps’s bugs were the same ones RosmarOps looks for in your agents: lists that stop matching, permissions nobody knows who decided, errors that stay silent, and an “I’m done” that nobody checks from the outside.

Fixing them was the easy part. What I want to share is what we are doing so they don’t come back.

Harness engineering, in one sentence

Mitchell Hashimoto named a habit: every time an agent makes a mistake, you take the time to build something so it can never make that mistake again. He calls the set of those pieces around the agent the harness.

Ichigo adds an order I have adopted as it is. When something fails, the fix is looked for in this order:

  1. Structure: make the mistake impossible to make.
  2. An automated check that always catches it.
  3. Text in the instructions, only where judgement is really needed.
  4. A person who reviews, at the end.

A line in the instructions is a wish: the agent will read it and almost always follow it. An automated test is a wall: it holds even if the agent forgets.

Six demons and their piece

1. The text that didn’t fit.

  • What happened: when Codex may use many desktop apps, the finding lists them all. On my computer the title went over the 200 characters the space accepts, and the program dropped the whole review.
  • The piece (check): a test that reads every length and count limit from the format itself and checks that the program respects them even when the data goes over. Today it covers the space’s lists and texts. In the next version, the limits will come straight from the format, so no new field is left out.

2. The lists that stopped matching.

  • What happened: the program could say “I detect this agent but can’t score it” for the Antigravity app, but that reason wasn’t in the list the report accepts. On my computer, the report wasn’t written. It happened three times in one week, with three different lists.
  • The piece (check): a parity test that compares 15 lists in the code with the format’s and fails if one falls behind. It is already in 0.10.2.

3. The tests that looked at another path.

  • What happened: the factory-score tests built the agents’ map by hand, along a different path from the one the real program uses. They were green because they tested something else. That’s how the missing rule in Cline’s settings slipped through and left my review without a score.
  • The piece (structure): a single way to build the map, the same for the tests and for the program, on a complete test computer with every agent and Windows paths. It arrived with 0.10.3.

4. The “I’m done” nobody checked.

  • What happened: everything was green and it didn’t work on my computer. A green test says something works where it was tested, not where it will be used.
  • The piece (check and person):
    • 0.10.2 wasn’t published until I ran it on my computer and it came out with a score and in my space.
    • Before publishing, an agent that hadn’t written that code reviewed it and approved it with five conditions that had to be met.
    • The next step is for the machine to do that test on its own: a clean Windows that, before every release, sets up what the agents leave when installed, runs the new version and checks that it finishes, that there is a score and that sending to the space works. If it fails, the version doesn’t ship.

5. The silence.

  • What happened: for days, the program dropped reviews and only noted it in a window that closed by itself.
  • The piece (check): no error stays only in the log. Either it shows in the report, or the review stops and says so. 0.10.2 already writes to the sends log every review that couldn’t be sent, and why. The full rule, with a test that searches the code for errors that are only logged, is part of the plan.

6. Haste with good intentions.

  • What happened: an agent added a change to the data format without asking. It was right in substance, and still it broke a rule. I myself moved a time on the server and left my space reading the previous day’s data.
  • The piece (instruction and person): any format change stops and waits for my signature. Today it is the weakest piece, because it’s an instruction. The next step is to turn it into a check.

The full harness: five phases

An engineering review by an agent that hadn’t written the code, comparing the project with the industry’s reference practices, left a plan in five phases. Each one closes a class of bug, not a single bug:

  1. A single path. The tests build the map the same way as the real program, on the complete computer. That’s 0.10.3.
  2. A broken version doesn’t ship. The Windows test before every release, and a procedure to withdraw a version if needed.
  3. A single registry of agents. Today, adding an agent touches about 18 places in 12 files. With the registry, agent number 15 won’t be able to break the other 14.
  4. Robustness on real computers. Badly written files, very long paths, unusual encodings. The worst that can happen is that something shows as “not read”, never as clean.
  5. Process and metrics. Every change with its independent review, every version with what it includes, and a measure of how many versions have to be withdrawn and how long the fix takes.

When all five are in place, another independent review will audit the whole code.

What I take away

In my AI-First guide I put it this way: a failure fixed where it was observed is hidden, not fixed. This week I’ve seen it in my own product. Every agent mistake was a question about the system it works in: if tomorrow it happens again somewhere else, does anything stop it?

The job isn’t to get an agent that never makes mistakes. It’s to build a harness where its mistakes can’t reach you twice.

If you use RosmarOps

Update to 0.10.3. If your space hadn’t received reviews for days, or showed different figures from your report, run the program once and it should catch up. Your score may change, because it now sees things that used to slip through.

Sources: Mitchell Hashimoto, “My AI Adoption Journey” (5 February 2026); Ichigo’s thread on X (5 November 2026); Albert Garcia Pujadas, AI-First Guide (in Spanish); notes of versions 0.10.1, 0.10.2 and 0.10.3 in the RosmarOps repository.

Written with the help of AI assistants. The experience, the mistakes and the decision to publish are mine.

Privacy · Legal notice · Terms · Cookie settings