A study examined 36,487 software repositories and confirmed autonomous AI agent loops in 217. More interesting than the percentage is what we know —and what we still don’t— about the controls and results of those systems.
In June a new phrase started going around: loop engineering. The idea sounded irresistible. Instead of keeping an eye on Claude Code, Codex or any other agent, you set up a system that hands it one task after another. You get on with something else and the agent keeps working. Within weeks there were threads, diagrams and recipes on how to do it. Almost all of them talked about productivity. Far fewer explained how to measure it.
This one is close to home for me. I’ve spent months building UXMachine with AI agents, and I’ve learned that letting them run ahead has its risks. Once, one of them started running checks nobody had asked for until it filled the disk. It wasn’t exactly one of those scheduled autonomous loops, but the problem looked a lot like one: it could keep doing things without it being clear when it should stop.
That’s why I’m more interested in the question that comes after automating: how do we know all those runs actually did us any good?
What loop engineering is, exactly
A coding agent already works in small cycles: it reads files, runs tools, looks at the results and decides what to do next. Loop engineering adds another layer. Instead of starting each session yourself, you set up a mechanism so that sessions start on a schedule or on an event: an issue comes in, a test fails, someone opens a pull request.
In its most careful version, the system should also know what goal it is pursuing, how to check the result, how much it can spend and when it has to stop or ask for help. But that definition is one thing, and what we find in real projects is another.
What they found in 36,487 repositories
In August, researchers from Heidelberg University and Singapore Management University set out to examine the phenomenon. Their paper, Loop Engineering: Building Blocks, Adoption, and Impact, was updated on 22 October. On 5 November, Ichigo brought it back in a thread on X that led me to read it more closely.
They started from 36,710 open-source software repositories. After removing duplicates and projects that were no longer reachable, 36,487 were left. Their searches found 256 possible cases and, after examining the configurations and the records available, they confirmed that 217 had run autonomous agent loops. That’s 0.59% of the sample.
It’s worth not confusing that figure with the share of developers who use agents this way. The study could only find certain public traces on GitHub. It couldn’t see what someone runs on their own computer, private configurations, or the part of the operation that lives outside the repository.
What struck me most was the gap between starting an agent and controlling what it does. In the classification of the 256 candidates inspected, done with the help of Claude Code and reviewed by one of the researchers, these mechanisms turned up:
- 101 had some limit on spending or turns.
- 76 had some kind of verifier.
- 55 included a goal or stop condition.
- 35 kept state between runs.
- Only 10 recorded the cost.
These figures don’t mean the others necessarily lacked those controls. They mean the researchers were able to identify them in the material they examined. In fact, direct searches of versioned files found hardly any trace of the memory and the other mechanisms that loop engineering guides recommend.
Another figure takes some of the shine off the image of an agent working all night on a big project. Of the 217 confirmed cases, 180 were triggered mainly by events in the repository, above all to review pull requests. Another 21 ran on a schedule and 15 combined both. A review automation can be useful, of course. But it’s not the same as handing over a complex task for hours and expecting the system to find its own way.
And then there’s the example of a system for triaging issues: it had piled up 6,290 runs. In the records the authors checked, none of them got as far as starting the agent step, because there were no issues to process. The workflow ran; that didn’t mean the agent had worked.
What the study does not prove
This is the part I miss most in many conversations about AI: telling apart what seems to work from what has been shown to work.
The paper by Lulla and colleagues does not show that loop engineering increases productivity. Nor does it show that it reduces it. What it says is that, in the literature and the cases they reviewed, systematic evidence to back up the claims about its benefits is missing. That’s exactly why the authors propose a follow-up experiment to measure results, code quality, effort and resource use.
It’s an exploratory study, not a census of every loop out there. Its heuristics could miss cases, and much of a loop’s run-time state is never published on GitHub. So it would be going too far to conclude that almost nobody applies controls. What the study does show is something more precise: it is much easier to find the configuration that starts an agent than public evidence of how it is supervised, what it remembers and how much it costs.
There are three different loops, and it’s best not to mix them up
Ichigo’s thread proposes a distinction I find especially useful:
- The agent loop. Inside a session, the model calls tools, interprets their responses and carries on. That’s how today’s agents normally work.
- The work loop. The agent delivers something, an external mechanism checks it, feeds back what failed and allows another attempt. Here it isn’t enough for the agent to say “done”.
- The system loop. When you spot a recurring problem, you change the rules, the checks or the way of working so that the next attempt starts from better conditions. You’re no longer just fixing one result: you’re improving the process that produces it.
Most of the cases the study found are automations that start agent sessions. That can save work. But on its own it doesn’t amount to having those three levels working.
The last two are, for me, the interesting ones. They’re also the ones that take more human work than many videos of agents coding on their own suggest.
What do we gain by leaving an agent working alone for hours?
I don’t have a universal answer. There are repetitive, well-bounded tasks where autonomy makes complete sense. And there are others where piling up attempts can mean piling up errors, costs or changes that someone will have to review later.
If we only count runs, we’ll miss much of the story. We should also count which changes were useful, how many attempts were thrown away, what it cost to check them and how much human time it takes to understand and accept the result.
My experience with UXMachine has made me a little wary of the word “done”. An agent saying it has finished a task doesn’t mean it’s done well. And the fact that it did it while I wasn’t looking adds no guarantee at all.
Maybe we’ve started with the easy part: learning to start agents and keep them from stopping. The hard part comes after. Knowing what they’ve done, being able to check it and deciding when it’s enough.
That’s what the next article in this series will be about: The real work starts when the agent says “I’m done”.
Main source: Jai Lal Lulla, Vahram Nersesyan, Seyedmoein Mohsenimofidi, Christoph Treude and Sebastian Baltes, “Loop Engineering: Building Blocks, Adoption, and Impact”, arXiv:2608.21884, version 3 (22 October 2026). Preprint submitted to JAWs@ASE 2026.
Written with the help of AI assistants. The reading of the study, the examples and the decision to publish are mine.