Exec summary

• Despite the hype, public examples of true self-improving loops are few and far between.

• A self-improving loop is only as good as its verifier, and a verifier is only as good as the real-world data grounding it.

• Building verifiers is hard, slow, and complicated, and where most of the competitive opportunity is today. Most AI engineering is trying to side-step them.

• Before you sponsor a project that builds one, align the resources: (1) what real-world data grounds the verifier? (2) who provides expert feedback on failures and what does that cost?

Subscribe now

Loops are having a moment. The poor souls who take it upon themselves to be their organization’s Most Plugged-In AI Person were in for a treat over the last few weeks: Boris Cherny, (creator of Claude Code), shared his Steps of AI Adoption, a de facto blueprint of what it takes to move past the bottlenecks in automating work; co-founder of Arize AI Aparna Dhinakaran broke down the theory of loops. Gergely Orosz of The Pragmatic Engineer quickly followed with a piece rightfully dispelling some of the hype.

No doubt many organizations are experimenting with self-improvement. But if you look for documented cases of self-improving loops in production, the list gets short quickly; among 9 use cases I found and looked into, only two disclose enough to verifiably call the loop “self-improving”.

Verifier is key

In its simplest form, a loop has three components:

1. The execution loop, which is whatever the automation actually does; say, monitors the bug queue and fixes them;

2. The verifier - checks the output of the execution loop against some standard you established (if in your mind verifier is served well by golden-data-evals alone - please read along);

3. Feedback - if the verifier flags an issue with the execution loop, the context of the issue is fed back into the execution loops for another spin.

And it’s in the verifier the real bottleneck of “self-improving” lives.

Why verifiers are expensive and difficult

Verifier, despite what seems to be an increasing trend in the industry, must be grounded in a steady trickle of real-world data; if it isn’t, it can (and will) optimize in a random direction.

For deploying code, this “steady stream of real-world data” is relatively straightforward, even if it requires some creative stitching: tests can pass or fail, production telemetry can catch anomalies, code history can be audited, A/B tests can run, and user feedback and bugs queue can be monitored.

As you move further outside of code and into knowledge work, figuring out the “steady trickle of real-world data” needs some creativity.

This need for creativity is why I like to see how various businesses have tackled the problem, but evidence on in-production loops is pretty thin. Kudos to the folks behind the two examples I break down below: Replit’s product development agent and Tax AI agent, built by OpenAI’s FDEs (PS. are FDEs a revival of 2010s implementation consulting?).

Agent drafts, human ships: current state of self-improving loops

Exhibit 1. Replit’s self-improving coding agent

The use case for Replit’s agent is straightforward: identify product issues, write the code and submit the build. From here, verifier takes over, including Replit’s benchmark built specifically for vibe-coding: it simulates the app and user’s clicks within it; if passed, the proposed change runs through A/B testing, and results are sent to a human for review to ship, iterate, or drop.

For me, an even more interesting element of Replit’s loop is Telescope - their proprietary traces clustering tool1, which allows to analyze traces at scale without unnecessarily exposing user data to random eyeballs. Plus, it looks cool.

Exhibit 2. Tax AI agent, built by OpenAI x Thrive

Tax AI set out to tackle the ambiguous and spottily-documented process of tax preparation. Since wading through hundreds of documents to prepare a draft is the most labor-intensive part of the CPA’s job, the appetite for automation was high enough to dedicate the time to weeks of painful - but effective - mechanical turking.

What I delightfully noted in OpenAI’s use case is that the agent improved drastically post-launch2. The percentage of drafts that were at least 75% correct went from roughly a quarter at launch to 86% - vast majority - at six weeks. The result comes from FDEs pulling teeth asking CPAs to explain each decision where their judgment differed from the agent.

Think about this process: one tax draft has dozens - or hundreds - of fields, each requiring data and judgement - sometimes formal, sometimes not. Imagine those six weeks and the amount of good will required on both sides, the FDEs sending draft after draft to the CPAs, asking them to explain why and how the agent is wrong, field-by-field. Slow, complicated, and expensive, and inevitable in navigating the muck layer of knowledge work.

This is the exact challenge with self-improving agents: is it possible to make agent’s failures visible and understandable? Is it feasible to get the context from the experts on enough of the failures to meaningfully improve the agent? This time and buy-in might be one of the biggest bottlenecks in AI adoption today.

If your team has solved this creatively, tell me more. If your team is struggling with figuring this out, also reach out, I’d love to workshop some solutions.

What about automated evals?

Inevitably, someone will say: we already have verifiers - we have an automated eval system, thank you very much, all taken care of.

Auto-evals are sort of a trojan horse of agent development - they are cheap, fast, simple, privacy-preserving, loops can spin indefinitely without being slowed down by humans - tempting to set them up and call it a day.

Hamel Husain, an ML engineer known for his work on LLM evals, recently ran an open test that proved autoevaluators did relatively well on known criteria. But because of criteria drift, it’s in the “undefined” where the customers weep and dollars leak:

The premise of criteria drift is that you don’t know how to evaluate an agent until your human eyeballs observe it interacting with the real world.

Even if you assume you do know all possible failure modes of your agent, there’s the increasingly ripe problem of reward hacking, where an agent optimizes for a score but not the intended behavior - for example, if rated on politeness, the agent might hang up on an angry customer; if rated on sales, the agent might give 99% discounts, or close $0 sales, or offer hallucinated perks - all of which you can catch, but not until you know it can happen.

Which needs real-world input.

Autoevals are one way of shooting yourself in the foot and sidestepping the verifier problem instead of solving it; other ways include: LLM-as-judge, synthetic data, “we’ll add evals later”, and golden data that stays static.

Don’t let the hype tell you that everyone’s automated their business with loops already. Before you sponsor a self-improving loop project, make sure you understand the messy middle between the launch and results, and that you’ve aligned the resources: (1) what real-world data grounds the verifier? (2) who provides expert feedback on failures and what does that cost?

That is, only if you have good verifiers.

I write for people betting careers and businesses on AI from the perspective of having shipped across three waves of emerging tech in startups and Big Tech - if that’s you, subscribe to get your weekly under-5-minutes-to-read-weekly-email.

Subscribe now

Disclaimers: opinions my own. Data for this piece was sourced from public sources and private AI communities.

Two weeks ago, Center for AI Safety announced that Claude’s Fable 5 increased 6-fold how many remote “jobs” LLMs can perform, going from 2.5% in October 2025 to, now, 15.8%1. It’s a major update to their Remote Labor Index - a measure how much of the freelance jobs market can be handled, end-to-end, by AI.

I find the jump to 15.8% noteworthy but unsurprising, and the details of how the index is composed will tell you why.

The Remote Labor Index is one of the most useful benchmarks to watch, because it’s one of the closest ones to answer the question “what does LLM progress mean for human jobs and real-world businesses?”

To build it, researchers pulled 240 projects, amounting to about ~$140k in economic value, off of sites like Upwork. The specific projects are unknown to the public, so benchmarks can’t be gamed. It then tracks what percentage of these projects that can be done autonomously by an agent, end-to-end; it’s important there’s no partial credit for “some completion”. The approach is prudent and discriminative to not-really-real progress, and yet - there’s a big gap between climbing the RLI and taking over large swaths of business work.

Since RLI measures freelance remote work only, there’s already some removal of the human element that exists in regular jobs. Even if you set aside the explicit carve-outs that RLI calls out: requiring physical labor, long-term evaluation, or direct client interaction, there are nuanced differences between freelance and, well, loadbearing work.

Jobs and projects posted on sites like Upwork are materially different from full-time jobs, in-person or remote. They are easy to slice up to contract out, straightforward to evaluate, require limited amount of trust and accountability. Nobody is losing sleep at night if they go wrong, they just get another contractor2.

In contrast, most knowledge jobs or business workflows are not like that.

As I’ve mentioned in a few comments on Import AI or Don’t Worry About the Vase, my experience as a technology practitioner (vs. researcher) puts me in a skeptical seat when it comes to the promises that AI will take over most of the economy, or even knowledge work or business activities: yes, we might see human-light and AI-heavy companies; we will see “work” augmented on a massive scale. But

1) humans and economy will evolve quickly in response, moving our labor and knowledge to more effective uses, and

2) even in the existing economy, there’s enough muck layer in most jobs - the interhuman messiness - that the agent-consumption of broader economic activity will be resistant to major changes3.

On one hand, the RLI is the closest benchmark we have to tracking real world impact, so increases here are notable. On the other hand, “Freelance Contractor Index” would be a more precise name, and 15.8% of that is a very different number than 15.8% of labor.

Lastly, I’m sure this will change in the future, but for the time being, the projects captured in the Remote Labor Index are static, meaning they were captured at the end of 2025 and LLMs are evaluated against the same set. This gives us a frozen reference point, but of course the remote labor landscape will change precisely as quickly as LLMs evolve, and projects will materially change. So even if 2030’s LLM can do 100% of 2025’s remote jobs, that number will be different for 2030’s remote jobs. wrote about this recently taking the opposing position:

But I’m more optimistic about human ambition and hunger to evolve, and more bearish on the “mucky” part of jobs and labor: ongoing relationships, ambiguous long-term horizons, and the right-hemisphere or instinctive parts of doing business.

PS. Muck layer is a concept I will return to often, which is why I wrote a post about it; I update it often.

The best insights into the future and scope of forward-deployed engineers hide in the boom (and no bust) of implementation consultants of the 2010s.

In the mid-2010s, if you were an enterprise using software, you were in the middle of two major shifts. First one was moving away from maintaining your own servers and moving them to cloud; the second was abandoning the upgrade hell of perpetual licenses sold by Oracle et al., and moving to SaaS that updated itself in the background and could be canceled any time.

Old software was pretty good for the seller and cumbersome for the buyer - bought a perpetual license, didn’t use it? The onus is on the buyer to implement it better.

The then-new model - subscriptions - gave buyers more flexibility, but came with a new risk for the seller, shifting the responsibility for implementation onto them. If the buyer doesn’t use it or see the ROI - churn!

And, because as an enterprise buyer, you have a vast but brittle ecosystem of workflows and data, haphazardly stitched together in your old systems, you need someone to hold your hand as you transition them over to SaaS. The mere thought of breaking something, of revenue lost with every minute of downtime, is keeping you up at night. You’ll pay good money to do away with that risk.

So a market of boutique consulting agencies blossomed: Cloud Sherpas, Bluewolf, Appirio, Meteorix, Vlocity - started out independently, eventually partnered closely with the seller (e.g. Salesforce or Workday) and a pretty relationship ensued:

• customer got the innovation they wanted,

• agency got the business,

• seller got the sale and intel on how to build a better, more marketable product.

Eventually, these agencies became such a crucial cog in the software industry machine that a wave of acquisitions followed in the mid-2015:

Sept 2015 Cloud Sherpas est. $350M-$450M) ~1,100 employees

Sept 2015 Meteorix ~$80M - $100M ~180 Workday experts

Mar 2016 Bluewolf $200M+

Oct 2016 Appirio $500M

Dec 2020 Vlocity $1.33B

Do the billions of dollars here foreshadow what’ll happen with FDEs? OpenAI’s recent acquisitions of Northslope and Tomoro, two implementation agencies, are an indicator here. Their $4B fund is a bet to own the hand-holding layer, a concession that the implementation is the hard part now, not the models.

Eventually, as software ossified and became relatively static, the implementation consultant industry evolved:

1. Software became much easier to adopt, partly because it injected lessons from onboarding the first waves of customers back into the product, and partly because as old-generation companies died and new-generation companies were started, they adopted the SaaS tools and Cloud natively - no complexity to port over;

2. The jobs of implementation consultants moved in-house, diffused between several different roles: customer success1 (for overall relationship maintenance, retention, and upsell), solutions architects (for the technical implementation and onboarding work), and product managers (for passing the requirements and features back and forth between the customer and the seller). This in-house support focused on the biggest enterprise customers, and outsourced all other work to the partner network.

Implementation consultants filled the service gap - between the muck and complexity of real workflows and SaaS’s blank slate - with a dedicated middleman.

Forward deployed engineers fill this service gap for agents.

The industry is trudging through the muck of real businesses and real jobs.

• The approach of throwing tokens at the wall turned out to be extremely expensive, and, unsurprisingly to anyone who’s heard of the Productivity J-Curve, the ROI is not immediate. [Uber, MSFT]

• Remote Labor Index - likely the most useful benchmark for quantifying how much of an impact AI will have on most enterprise workflows - still sits at <5% is, as of Fable 5, at about 16%, and even that is on upwork-style tasks that require, for example, no direct client interaction.

It’s not surprising then that the answer to this problem is that labs will build out their own “agencies” to implement the product. Turns out harness engineering - making sure the agent does what you want it to do at scale - is insanely hard. It’s figure-outable for the buyers, but their existing employees already have full-time jobs. So as the frontier labs are looking for sticky use cases and retention in real enterprise workflows - they will shepherd the willing (and sufficiently big) customers along through the hands of FDEs. As implementation consultants have proven, this model scales once it finds its footing.

There are other signals that the industry is highly optimistic about this “send an AI-consultant into a company” model:

• Y Combinator looking for AI native agencies to sponsor

• Claude launched the small business suite, sort of providing what FDEs would do for bigger businesses - an agent that “plugs into” existing data and workflows. Because of the ossification I mentioned before, lots of small businesses run on the same stack with similar workflows, so this is in theory highly scalable, but I’d need to see retention numbers to really know (adoption of some of the plug-ins is at 700-800k installs, so lots. I suspect maybe ~20% is retained and running actively). The alternative is tiny agencies that build the workflows in specific industries.

Will this work? Probably; it’s a pretty safe bet that it will, to the extent that agents can handle workflows with high ROI; the actual question is whether the ROI these companies will see is worth the cost of both tokens and the maintenance required.

With SaaS and Cloud in 2010s, that math was simpler because it was deterministic: cut $150,000 in server costs and $2M per year in Oracle software updates in exchange for a subscription of $200,000 per year. With agents, the math is fuzzy. Can you eliminate some employee or contractor spend and save the money? Some portions of them? For how much money in tokens? How directly does it translate to business growth? How do you quantify risk or future opportunity cost?

Aaron Levie, CEO of Box reported after a bunch of meetings with enterprise IT leaders that this is one of the top open problems in the industry.

All of these are much fuzzier questions, and FDEs will be under a lot of pressure to deliver results quickly, sometimes too quickly to validate whether a new way of operating is working or not; especially where it comes with change management for humans.

Afterthoughts and side quests

1. Can FDEs uncover enough “standard” processes to ossify them into products that can be sold (out-of-the-box agents, workflows, scripts)? FDEs are essentially a discovery surface for new products.

2. The success of individual company’s FDE teams depends on how they’re structured. How much of this is “I actually work on site next to the buyer, watch them work, see the muck they won’t verbalize” vs. “I visit on-site occasionally and we review and collect feedback” vs. “I meet with them on Zoom every two weeks”. The muck is hard to spot if it relies on the customer’s ability to identify the “automatable” or “solvable” problems, cutting into the exact expertise an FDE is there to provide.

3. Here’s what the rest of the industry says about FDEs:

   1. expects the demand to increase, and, quoting Aaron Levie, points to them as an opportunity to break into the industry, especially for new grads: two observations I agree with.

   2. Vas, CEO of Varick Agents, wrote an entire piece on the FDE demand and how to capitalize.

   3. Quick scan through engineering forums on reddit seem to echo that FDEs are essentially solutions consulting but cool

      

Do not overcomplicate is an evergreen series that introduces concepts that 1) quickly take over the industry, 2) feel like something everyone should be doing, and 3) quickly spiral out of control in terms of complexity, when in reality, in majority of use cases, they are quite simple - or at least only as complicated as you need them to be.

If you read nothing else because you’re headed out for the 4th of July weekend, take away this:

1. Harness is the difference between a demo and a functioning automation;

2. Similarly to evals, they are the first step to a functioning product, not the last;

3. Start simple: markdown, .csv, and human eyeballs.

Terminology around AI and agents evolves so quickly that it’s hard to figure out which concepts to treat seriously because they will eventually run the business, and which one to let slide (MoltBook). The rule of introducing new food to toddlers seems applicable here: if you’ve seen it on your plate 15-20 times, it might be safe to take a bite.

Building AI agents for enterprise is my day job. Because of this, I am fairly skeptical of most stories about autonomous agents running complex real-world operations end-to-end, like automating entire jobs, because, well, I work on agents that try to do exactly that.

Most of the risk is in preventing agents from making silly or disastrous mistakes. Research moves quickly, because reliability ties up a lot of ROI. Real-world case studies are few and far between, and the never-ending streams of arXiv papers make it seem like you need a dedicated ML whiz on staff to build out the sophisticated infrastructures of pipelines and evaluation algorithms.

But just like evals can be deconstructed to their minimal (and often most effective) functional baseline of a spreadsheet and eyeballs, so can the whole concept of a harness. More of a lego box of blocks to build from rather than a prescription to start from.

Doesn’t it make sense, then, that most public “LLM failures” are harness failures?

1. The Chevy Tahoe for $1: no output validator (or a bad one);

2. Replit’s agent deleting the production database: tool authorization too loose and broad;

3. Cursor’s “I quit” incident, where the agent refused to keep coding and told the user to learn to program: bad output validators;

4. Lawyers submitting hallucinated citations: issues with retrieval, prompt, and validators to check citations;

5. Air Canada’s chatbot’s refund - similarly, issues with retrieval - grounding claims in the actual policy document, prompt, and validators.

I’d love to deconstruct a real-life harness in the next post, so if you have one you can share (yours or public) let me know. Harnesses already are becoming the moats and tightly-protected secret sauce, and “harness engineering” a real job, even if hiding under a different title.

PS. If agent performance issues are a harness failure, is OpenAI’s social media manager job posting a harness opportunity? Because, honestly, I thought LLMs could do this already.

(I say this tongue-in-cheek, because I’m a believer that most tech and knowledge jobs - regardless how automatable they might seem on the surface - have a real moat that protects them from being replaced by automation, a deep “muck layer” that humans navigate intuitively and LLMs are not suited for. This “muck layer” is a concept I’ll return to often, which is why I wrote about it.

Sources:

• https://www.preprints.org/manuscript/202604.0428

• https://www.mindstudio.ai/blog/agent-harness-engineering-wrapper-matters-more-than-model

• https://medium.com/@MongoDB/the-agent-harness-why-the-llm-is-the-smallest-part-of-your-agent-system-bce68414ccfd

• https://github.com/HKUDS/OpenHarness

• https://arxiv.org/pdf/2603.28052 - “Changing the harness around a fixed LLM can produce a 6× performance gap on the same benchmark.”

• interpersonal and interorganizational politics,

• gut feelings,

• EQ,

• unclear fail / success scenarios,

• real-world complexity of systems, both digital and atom-based

• organizational and network inertia

Because of the muck layer, I suspect that the large swath replacement of human employees with AI that’s been predicted over the last few years will not happen.

Here are the signals:

• Forward deployed engineering heating up proves the muck layer - this happened with implementation consultants in the 2010s. Forward deployed engineers are a sign that the promise of AI automation is, to some extent, failing for the time being. AI is supposed to be empowering, and automation in a box, and yet all frontier labs are growing forward deployed engineers on staff to provide what the LLM can’t - reliable performance.

• Complexity of harnesses to get useful and safe results; an average business owner / manager is not going to take this on

• I (and many others) have written about Jevons Paradox and how it changed writing code. Human tasks change, but demand on the overall volume of human effort isn’t decreasing en masse.

• Head on the line: an important roadblock in deploying AI is liability for when things go wrong, not just executing whatever the task is. If the automated response is a mistake, someone has to hold the bag. And as long as there’s a liable human in the loop, that human will need an ecosystem of humans around them.

• Messy data and unclear procedures: sort of a Polanyi Paradox on an enterprise scale and not a new problem, something Digital Transformation efforts have been working on for at least a decade.

• Real world friction, like orchestrating humans

Caveats

• There’s a high number of researchers that will tell you this is only relevant insofar as humans are involved in the economy; it’s possible that economy and capital take off independently, leaving labor behind; “work” as in economically profitable activities, will be taken over completely by machines, which effectively eliminates the Muck Layer. This is a viable take if for no other reason than because a lot of smart people see it as viable. But after 200+ hours dedicated specifically to the question of “how likely is this?” I am less and less convinced that this is a real scenario. Elements of the economy, sure, like biotech R&D or maybe the boilerplate-code-writing industry. The entirety of the economy? No way.

• Of course that doesn’t mean that the career landscape will stay static, jobs will change as they always have. One of the earliest signals is the growth in AI engineers, forward deployed engineer roles. But knowledge work as a profession is not going anywhere.

Related datapoints:

• “LLMs are less like sophisticated, strategic advisors and more like enthusiastic newcomers who skimmed the textbooks, plugged into the ether of buzzwords, and enthusiastically offer their advice.” HBR

During the many months of meandering around AI discussions and literature1, I realized that this false and (somewhat convenient) conviction is exactly the problem. I’ve just finished the BlueDot Impact‘s AGI Strategy course with the intention of figuring out how to bridge the gap between the ambiguous scenario of a future AI risk, and the hammer-and-nail of how AI is being deployed today.

AI safety doesn’t need another mediocre researcher that I maybe could become in a couple of years - it needs AI-safety-oriented-people everywhere else, and since I’m already close enough - working on applied AI in enterprise - I’m starting where I am.

The vast majority of life runs on enterprise workflows - how the government operates, how most things are built, how banks handle your money or how a hospital deals with patient data. My day job is weaving various automations into these workflows; and because of the messiness of the human element inside them, there’s a lot of AI risk that sits right at the deployment layer.

The near-term risks - which we’re already seeing - and the plausible-but-yet-unrelized long-term catastrophic scenarios, like superintelligence taking control over our infrastructure, aren't separate categories, but the same dynamics at different points on a timeline. So the path to the big bad future scenarios runs through the small failures that are already happening.

Which means a lot of the highest-leverage work has to happen at the deployment layer, and it has to happen by people who know that layer: the PMs, engineers, the procurement folks, the operations leads, and the general “middle” of the org chart that gets handed an LLM to figure out how to implement it across their side of the business. And, because all of this has to happen quickly to bring about the ROI, there’s a looming brittleness, that, if multiplied across finance, healthcare, institutional underpinning of everyday life, looks pretty bleak.

Which is why I’m throwing my hat into the AI chatterbox: the intention is safety, but the topics span across the concepts and infrastructure that you as a PM, engineer, exec should know: how do agent harnesses work? What is the Productivity J-curve and why does it apply here? How to think about forward deployed engineers? If I want to know why an agent made a specific decision, can I just look at the chain of thought?

Welcome!