This is the second in a series, and a companion piece to "What Is Any Agentic Architecture Worth Anyway. PharmaCo International: An Agentic AI Case Study" Where I have gone deep with this case study, the material below takes a wider scope on the impact of AI and Agentic AI structures to industry.

A note on what this is. Everything that follows is opinion backed by recent data on AI adoption and actual successes, where it may likely be found — informed by the PharmaCo case study, by the Harness Lab experiments (H1–H10) that underpin it, and by a broader read of where the agentic AI market actually stands in mid-2026. It is not a forecast with confidence intervals. Where the view here contrasts with mainstream market enthusiasm, or with the more cautious findings of recent research, both sides are presented — the goal is to give readers the material to form their own judgment, not to win an argument. Whilst my use and practical application of the findings here have been helpful to me, you could have differing views, based on industry task-specific application.

1. The cost-side ceiling

The PharmaCo case study, on its own numbers, makes an unglamorous point: choosing the right harness architecture for disruption response was worth roughly $297M in enterprise value versus the “wrong-but-plausible” alternative, and getting it badly wrong (H4) was worth nearly $1B in destroyed value versus the recommended choice (H2). Against a $12B company, that’s a 2.5–8% potential swing in enterprise value from one operational decision.

That is a real number. It is also, deliberately, the ceiling of what a cost-avoidance story can deliver — because cost savings are bounded by the size of the cost base you’re optimizing. PharmaCo’s disruption-response function is a sliver of its total operating cost. Even a dramatic improvement to a sliver is, by construction, a small improvement to the whole.

This is the part of the agentic AI story that’s easiest to model, easiest to benchmark (it’s exactly what the H1–H10 experiments measured), and — I’d argue — the part most over-indexed in how the market currently talks about AI’s impact on individual companies. “We deployed agents and cut costs by X%” is a real, fundable, board-defensible story. It is also a story with a visible end-state: once the process is automated, the savings don’t compound indefinitely, they just recur. That could be good enough for companies in steady state, dividend plays. But….

A useful way to see why this ceiling exists comes from a framework Arvind Narayanan and Sayash Kapoor proposed for knowledge work generally: most jobs are a “decide-execute-deliver sandwich,” where AI compresses the execute layer in the middle but leaves the decide layer (what should be done, and why) and the deliver layer (who’s accountable for whether it was done right) largely untouched. Read through that lens, H2’s win over H9 is squarely an execute-layer story — which architecture responds fastest and most coherently to a disruption PharmaCo’s leadership has already decided is worth responding to, and for which someone remains accountable regardless of which architecture ran the response. The $297M–$986M swing is real, but it’s a swing within the execute layer; it doesn’t touch who decides PharmaCo’s disruption-response posture or who’s accountable when a plan goes wrong. That’s the structural reason the ceiling exists — not merely an artifact of this particular cost base being small.

2. The revenue-side frontier

The more interesting — and much harder to model — story is what happens when agentic AI touches the top of the income statement rather than the cost lines underneath it. Drug discovery is the cleanest illustration, because the economics of pharma R&D are already asymmetric and option-like before AI enters the picture: most programs fail, a small number succeed enormously, and the entire industry’s economics are built around that distribution.

The numbers being floated for AI’s effect on that distribution are large enough that they’re worth taking seriously even with a generous discount for hype. McKinsey-style estimates put the addressable annual value of generative AI across the pharma value chain in the $60–110B range, with some forecasts running considerably higher when R&D productivity gains are included. The AI-in-drug-discovery tooling market itself is still small (low single-digit billions in 2025–26) — the action is in what the tools enable, not in the tools as a line item. Exscientia has pushed multiple AI-designed candidates into clinical trials in under a year, a process that historically takes years. Insilico Medicine’s AI-designed candidate has progressed toward Phase III. Early data on AI-designed molecules entering Phase I shows success rates well above the historical industry average (roughly 80–90% versus a long-run baseline near 50%), though the sample sizes are still small and no AI-originated drug has yet completed the full approval pipeline — most forecasts put the first such approval in 2026–2028. Note: Takeda's zasocitinib (covered in my separate Funnel/Floor/Structure piece) has since cleared Phase III and is pending an FDA filing within fiscal 2026.

What’s striking is the deal activity this is attracting from companies that are not AI companies: Eli Lilly’s multi-billion-dollar partnership with Insilico Medicine, Novo Nordisk’s R&D-and-manufacturing partnership with OpenAI, Johnson & Johnson’s multi-target collaboration with Isomorphic Labs, GSK’s infrastructure partnership with Noetik. These are revenue-side bets by companies that already have functioning, profitable cost structures — the rationale isn’t “make our existing pipeline cheaper,” it’s “change the shape of the pipeline itself.”

This is the asymmetry that a cost-avoidance model like PharmaCo’s disruption-response case structurally cannot capture. Compressing a discovery timeline by even a year, or modestly improving a Phase II success probability, doesn’t show up as a percentage on an existing cost line — it can mean an entire additional revenue stream that wouldn’t otherwise exist, at NPV magnitudes that dwarf the $297M–$986M range from Section 10 of the PharmaCo case.

2.1 A third axis: when the payoff isn’t cost or revenue, but capability

The framing above already brushes up against something worth pulling out and naming on its own. Some of history’s technologies with the largest eventual economic impact looked, for years after their invention, like poor investments by any cost-or-revenue test — not because the technology had failed, but because “what does this save, or what does this sell” was the wrong question to be asking yet. GPS spent decades as a defense programme before it became a civilian product category. Genome sequencing absorbed a multi-billion-dollar, decade-long public investment before it produced anything resembling a return. Early integrated circuits had nothing like the cost or performance advantage that would later make them ubiquitous. In each case the technology wasn’t expanding a margin or opening a revenue line yet — it was making something previously impossible now possible, and a cost/revenue lens only became the right one to apply once that capability already existed.

Pharma R&D is the cleanest current illustration of this third axis, but it isn’t unique to pharma. Semiconductor design is arguably just as strong a case: the more consequential AI-driven outcome there isn’t “design existing chips more cheaply” — a Floor-type claim — but whether AI-assisted exploration of interconnect topologies, packaging techniques, or photonic architectures turns up designs no human engineering team would have proposed on its own. That’s the same fat-tailed, capability-expanding logic as drug discovery, applied to an entirely different industry. Read this way, AI’s most consequential role in both fields starts to resemble less a labour-saving tool and more a scientific instrument in the lineage of the microscope or the telescope — something that doesn’t make existing inquiry cheaper so much as it opens an entirely new field of inquiry. Whether agentic AI eventually belongs in that category, rather than the labour-automation category this essay otherwise treats it as, is one of the more consequential open questions this framework doesn’t resolve on its own.

This third axis lives inside the Funnel described in Section 1 above and developed further in “The Funnel, the Floor, and the Structure” — the Funnel was already described as capable of changing which bets enter a pipeline, not merely speeding up existing-style ones. What this subsection adds is a name for the most extreme version of that tail, because “capability expansion” describes a meaningfully different kind of bet than “a marginally better version of an existing product”: one where, in the years before the new capability proves out, the honest answer to “what’s the ROI” may legitimately be poor, unclear, or negative — without that being evidence the bet is failing.

2.2 A scope boundary worth naming: this framework doesn’t explain state-funded capability bets

One place this matters concretely: government and defense investment in exactly this kind of capability-expanding technology — nuclear research, early aerospace, GPS’s own military origins, internet infrastructure — has historically continued despite weak or unclear near-term ROI, for a reason this essay’s framework doesn’t capture. A state funding a capability bet isn’t optimising for enterprise value or profit margin at all; the payoff currency is strategic position or security, which doesn’t convert into the kind of multiple-based valuation Section 10 of the PharmaCo case uses, or the enterprise-value language this essay has used throughout. A “poor” dollar-denominated ROI can be the expected and accepted outcome of that kind of bet, not evidence it is failing.

This is worth flagging explicitly rather than leaving it implicit, because the companion piece “Does Agentic AI Matter?” leans on Nicholas Carr’s argument that universally available technology stops conferring competitive advantage — a claim specifically about firms competing against other firms for advantage measured in profit or market position. That argument was never built to explain why a state keeps funding a capability with unclear near-term returns, and it shouldn’t be read as implying such investment is irrational simply because it doesn’t show up favourably on this essay’s cost/revenue axis. Where this series’ frameworks are scoped to firm-level competitive strategy, state-level capability investment is a different actor optimising for a different currency entirely — outside this series’ scope by construction, not by oversight.

3. One company, two economies

Here’s where I think the PharmaCo case study is most useful as a diagnostic, even though it deliberately stayed on one side of this divide: the same company can be living in both economies simultaneously, and the three drugs in that case — Lisinopril, Metformin, Salbutamol — are the tell. All three are decades-old, off-patent, generic small molecules. No real innovator pharma builds its strategic narrative around a generic API supply disruption; that’s a generics manufacturer’s problem (Teva, Viatris, Sandoz, and similar companies operate exactly this kind of import-dependent, thin-margin, operationally-driven business).

A pure generics manufacturer is, almost by definition, living entirely in the cost-side economy — there’s no R&D pipeline to accelerate, no patent-protected revenue to expand, just margin to defend against exactly the kind of supply shock PharmaCo modeled. For that company, the PharmaCo case study’s framing is close to the whole story: harness architecture choice is a capital allocation decision measured in tens to low hundreds of millions of enterprise value.

An innovator pharma with a real R&D pipeline is a different company wearing the same SIC code. Its back office, supply chain, and compliance functions face exactly the same cost-side calculus as the generics manufacturer — but its R&D division is playing the revenue-side game described in Section 2, where the numbers are an order of magnitude larger and far less certain. Pfizer’s CFO described 2026 R&D productivity gains from AI as freeing up capacity to “take on more substrate” for future medicines — note that this is explicitly framed as a productivity reallocation story (do more with the same budget), not a cost-cutting story; the $11B R&D budget for 2026 didn’t shrink.

If I had to state the view plainly: regulated industries will likely see smaller near-term cost-side gains than unregulated ones (because compliance overhead bounds how much of the back office can actually be automated, and regulatory review timelines are exogenous to how fast your internal systems run), but potentially much larger long-run revenue-side gains in their R&D/innovation functions specifically — because that’s where the bottleneck is internal throughput rather than external approval timelines, and where the payoff distribution is fat-tailed enough that even modest probability shifts are worth enormous amounts.

The counter-view, and it’s a serious one: AI-discovered molecules still have to clear the same Phase I/II/III gauntlet as everything else, and as of mid-2026 zero have done so. FDA guidance on AI use in regulatory submissions (the January 2025 draft and the January 2026 “Guiding Principles” follow-up) is explicitly risk-based and incremental — regulators are not going to compress review timelines just because the candidate was AI-designed. It’s entirely possible the revenue-side story takes the rest of this decade to show up in actual approved-drug revenue, in which case the cost-side story — unglamorous as it is — remains the only one with a near-term P&L impact, for pharma specifically.

4. Where cost and revenue collapse into the same line: software

Pharma’s cost-side and revenue-side stories are separable because R&D and operations are organizationally separate. Software — and SaaS in particular — is the industry where I think this separation breaks down entirely, which is also why I’d guess it’s where most of the current Silicon Valley enthusiasm is actually anchored, even when the framing is about AI “in general.”

The mechanism is structural: SaaS revenue has historically been priced per seat — you pay for the number of humans who use the software. If an agentic AI system can do the work that used to require N human seats, two things happen to the same line item simultaneously: the buyer’s cost structure improves (fewer seats needed) and, if the vendor doesn’t change its pricing model, the vendor’s revenue declines — automating away your customers’ need for seats is, under per-seat pricing, automating away your own revenue. This is being described candidly in the industry press as an existential pricing problem, not a hypothetical one, and it’s why 2026 has seen a rapid, real shift toward usage-based and outcome-based pricing — vendors are trying to reposition themselves to capture value from the work done by agents rather than the seats occupied by humans, before their customers do the arbitrage for them.

Gartner’s numbers on this are aggressive but illustrative of the direction: roughly 40% of enterprise applications expected to ship with task-specific agents by the end of 2026 (up from under 5% in 2025), with a best-case long-run projection of agentic AI representing something like 30% of enterprise application software revenue by 2035 — a market north of $450B. Omdia’s estimate of the enterprise agentic AI software market itself growing from roughly $1.5B (2025) to $41.8B (2030) — a five-year CAGR around 175%, far outpacing generative AI’s earlier growth curve — is one of several data points suggesting investors are pricing this as a structural shift in how software is bought and sold, not an incremental feature.

This is, I think, the closest thing to a single coherent answer to “what is Silicon Valley so excited about”: not drug discovery specifically (too slow, too regulated, too far from most VC portfolios), and not generic cost-cutting (too small per company, as Section 1 shows) — but the prospect of an entire industry’s revenue model being re-architected around metered agent output, with the companies that reposition first capturing share from those that don’t. It’s a revenue-side story for the software industry as a whole, even though for most of software’s customers it shows up as a cost-side story (fewer seats, lower software spend, redeployed headcount).

There’s a sharper version of this threat worth naming. Narayanan and Kapoor point out that most software engineers already work in-house, inside non-software companies, not at software vendors — and argue that share may grow as AI makes building software cheaper (their version of Jevons’ paradox: cheaper execution doesn’t shrink total demand for software, it expands it, including demand for software that previously wasn’t worth building at all). If that’s right, the risk to incumbent per-seat SaaS vendors isn’t only “our customers need fewer seats of our product” — it’s “our customers increasingly build the equivalent capability in-house, because their own engineers can now execute on it cheaply too.” That’s a harder problem than a pricing-model change can fix, because it’s not about how the vendor charges for a given capability — it’s about whether the vendor remains the cheapest place to get that capability at all.

5. The adoption gap: why most of this isn’t happening yet

Everything above describes potential — and the gap between potential and realized impact, in 2026, is large. MIT’s NANDA initiative published research in mid-2025 (and follow-on work has largely confirmed the pattern into 2026) finding that roughly 95% of generative AI pilots inside large organizations produce no measurable P&L impact. The headline number gets cited a lot; the more useful detail is why: the research attributes this overwhelmingly to organizational and integration failures rather than model limitations — generic tools that don’t adapt to specific workflows, budgets concentrated in high-visibility sales/marketing pilots rather than the back-office automation where MIT found the actual ROI was concentrated, and a strong success-rate gap between buying from specialized vendors with domain expertise (roughly two-thirds success) versus building in-house (roughly one-third).

A second, more pointed strand of evidence makes a related point even more directly: a lot of what gets reported as “AI replaced these jobs” doesn’t hold up under scrutiny. Narayanan and Kapoor walk through several high-profile 2026 layoff announcements — Block, Snap, Intuit — where executives cited AI capability gains as the rationale, but the underlying drivers (investor pressure, financial strain, organizational restructuring) told a different story; in Intuit’s case, the CEO explicitly pushed back on the AI framing the press had attached to the cuts. They cite a survey finding that a majority of US hiring managers admit to citing AI for layoffs because it plays better with stakeholders than citing financial constraints, and an HBR survey where 21% of executives had already made large headcount cuts “in anticipation of” AI, versus only 2% citing actual AI implementation as the cause — roughly a tenfold gap between anticipated and realized impact. New York’s first full year of WARN Act AI-disclosure data shows a similarly small realized-impact figure: well under 1% of recorded layoffs checked the AI box, though the authors note this could understate the true figure given asymmetric incentives around how companies report.

This doesn’t contradict the MIT NANDA finding — if anything it sharpens it. NANDA measures whether AI pilots produce measurable P&L impact (mostly no); the layoffs evidence suggests that even where job cuts are happening, the causal story attached to AI is often a convenient narrative for decisions that would have happened anyway. Two different ways of arriving at the same conclusion: the realized labor-market impact of agentic AI, as of mid-2026, is consistently smaller than the headlines suggest — across both “did the pilot work” and “did the layoff actually happen because of AI.”

This maps remarkably cleanly onto the Harness Lab findings that underpin the PharmaCo case study. H9 — the five-agent swarm that looked most sophisticated on paper — lost to H2, a simpler structured single agent, specifically because complexity compounded coordination costs faster than it added capability. The MIT findings about internal builds underperforming vendor partnerships are, in effect, the same lesson at the organizational level: more moving parts, more integration surfaces, more places for a “looks sophisticated” approach to lose to a “fits the actual task” approach. Gartner’s own projection that over 40% of agentic AI projects will be cancelled by 2027 — despite (or because of) rapid adoption — reads as the market-wide version of H4: enthusiasm for “more agents, more coverage” outrunning the harder discipline of matching architecture to task.

The financial services data offers a useful counter-data-point on what does work: KPMG and IDC both report roughly 2.3x average ROI on agentic AI investments within about 13 months for institutions with strong governance, and McKinsey found 20–60% productivity improvements in credit analysis specifically — a narrow, well-specified, document-heavy task with clear success criteria, which is exactly the profile that the H3 compliance-monitoring scenario in the PharmaCo case (and the “Harness Lab, Automated” follow-up benchmark) suggests should favor simpler, tool-augmented architectures over elaborate ones. The pattern across all of this research, for what it’s worth, seems to be: narrow, well-specified, document/data-heavy tasks with measurable success criteria are where agentic AI reliably delivers ROI today, regardless of industry — and the industries that will see the fastest realized impact are the ones with the most tasks fitting that profile, not the ones with the most dramatic AI narratives.

6. A birds-eye framework

Putting the above together, here’s the 2×2 I’d use to think about where agentic AI’s impact lands, and roughly how fast:

The PharmaCo case study sits squarely in the top-left cell — which is precisely why, on its own, it “doesn’t seem as impactful as markets make out”: markets, when they talk about AI broadly, are mostly pricing in the bottom-right and top-right cells, which this case study was never designed to address. Two things sit outside this 2×2 entirely rather than inside any single cell: Section 2.1’s capability-expansion axis, which describes a tail extreme enough that “revenue” isn’t quite the right word for it either, and Section 7’s composition shift in who does the execute-layer work, with consequences that arrive years after the cost/revenue effects above.

7. The junior pipeline problem

One thread the framework above doesn’t address: even if aggregate demand for a given role holds steady — which Narayanan and Kapoor argue is the likely outcome for software engineering specifically, given how price-elastic demand for software has historically been — the composition of that demand can still shift in ways that matter a lot for individuals, and for organizations five-plus years out. A reader comment on their piece (from Vitaly Osipov) makes a sharp point: execution-layer work is precisely where junior people have historically built the judgment they’ll need to become senior decide/deliver people later. If AI absorbs most execution-layer work, junior hiring could fall even while senior demand stays robust — creating a pipeline problem that’s invisible today and only becomes visible years later, when there aren’t enough experienced people left to do the deciding and delivering that AI still can’t do.

This connects directly to Section 11.3 of the PharmaCo case, which discussed redirecting compliance-analyst capacity toward higher-value work following H3’s adoption. The framing there was optimistic — three analysts’ time freed up for more strategic work. But if that pattern repeats at the entry-level hiring decision — don’t backfill the junior compliance-monitoring role at all, since H3 now does that work — the org saves the junior salary today and loses a training pipeline for tomorrow’s senior compliance hires. Neither this essay nor the PharmaCo case modeled that cost, because it doesn’t show up on any single year’s income statement. It shows up later, as a senior-talent shortage that’s slow and expensive to fix once visible.

This is arguably the most under-priced risk in the entire framework above: it’s invisible in every quadrant of Section 6’s matrix at the time horizon a board typically acts on, and yet it’s exactly the kind of structural shift that compounds the way the revenue-side stories in Sections 2 and 4 do — just in the wrong direction.

8. What would change this view

A few things that would update me away from the framing above, in either direction:

• An AI-originated drug receiving full FDA approval (projected by various analysts at roughly 60% probability for 2026–2028) would be the first hard evidence that the revenue-side pharma story is converting from pipeline activity into realized value — and would likely accelerate the deal-making already underway. As mentioned earlier, I note this recent development and will watch the space: Takeda’s zasocitinib (covered in my separate Funnel/Floor/Structure piece) has since cleared Phase III and is pending an FDA filing within fiscal 2026.

• A SaaS company reporting a quarter where usage/outcome-based AI revenue materially offset a decline in per-seat revenue would confirm the Section 4 thesis is playing out on the timeline I’d guess, rather than the slower 2030+ timeline some forecasts imply.

• A second or third independent study replicating MIT NANDA’s ~95% pilot-failure finding with a meaningfully lower number would suggest the “learning gap” is closing faster than the 2025–26 data suggests — which would matter enormously for how quickly any of the cells in Section 6 move from potential to realized.

• Conversely, if Gartner’s 40%+ project-cancellation prediction for 2027 lands roughly on target, it would suggest the gap between agentic AI’s demonstrated architecture-level potential (the H1–H10 results) and most organizations’ ability to actually capture it remains the dominant constraint — in which case the PharmaCo case study’s core message (architecture-and-task-fit matters more than raw capability) becomes, if anything, more relevant, not less, as the bottleneck shifts from “can the model do this” to “did the organization build the right harness around it.”

• Entry-level hiring data specifically (not just aggregate headcount) showing a sustained decline in junior roles across software, compliance, and similar execution-heavy entry points would be the earliest leading indicator for Section 7’s pipeline concern — well before any senior-talent shortage became visible in P&L terms.

• A second capability-expansion case outside pharma converting from research activity into a deployed, valuable result (Section 2.1) — for instance, an AI-discovered semiconductor interconnect or packaging technique reaching production — would be the clearest sign the third axis generalises beyond drug discovery rather than being a pharma-specific artifact of how option-like that industry’s economics already were.

• A second state adopting New York’s WARN Act AI-disclosure checkbox, showing a similarly low realized-AI-layoff percentage, would reinforce the AI-washing finding in Section 5 as a general pattern rather than a New York-specific artifact (or a reporting quirk of that particular dataset).

  

Sources and further reading

• Anthropic, Economic Index — series of reports tracking real-world Claude usage, Jan/Mar 2026 editions cited in Section 5 and Section 7’s WARN Act discussion context: anthropic.com/economic-index · Jan 2026 report · Mar 2026 report

• Massenkoff, M. & McCrory, P., “Labor market impacts of AI: A new measure and early evidence” (Anthropic, March 5, 2026) — source for the “slower hiring rather than increased separations” finding referenced in Section 5: anthropic.com/research/labor-market-impacts

• McKinsey Global Institute, “Generative AI in the pharmaceutical industry: Moving from hype to reality” — original source of the $60–110B annual addressable-value estimate for pharma cited in Section 2: mckinsey.com

• AllAboutAI, “AI in Drug Development Statistics 2026” — source for AI-discovered-drug Phase I success-rate figures (80–90% vs. a lower traditional baseline) referenced in Section 2: allaboutai.com/resources/ai-statistics/drug-development

• IntuitionLabs, “Accelerating Drug Development with AI in the U.S. Pharmaceutical Industry” — source for the Exscientia/DSP-1181 12-month-to-clinic example referenced in Section 2: intuitionlabs.ai/articles/accelerating-drug-development-ai-pharma

• IntuitionLabs, “Measuring AI ROI in Drug Discovery: Key Metrics & Outcomes” (Apr 2026) — source for Eli Lilly’s $2.75B Insilico partnership figure referenced in Section 2: intuitionlabs.ai/articles/measuring-ai-roi-drug-discovery

• DrugPatentWatch, “AI in Drug Discovery 2026: What Actually Works, What Remains Hype, and Where the IP Value Sits” — corroborates the Phase I success-rate figures and notes Phase II/III as the still-pending validation event for AI-discovered candidates, referenced in Section 2: drugpatentwatch.com

• BioSpace, “AI Is Changing Pharma’s Bottom Line Now—But Not Through Splashy Drug Discovery” (Feb 2026) — source for the framing that AI’s clearest near-term pharma impact runs through development speed rather than discovery itself, referenced in Section 2: biospace.com

• FierceBiotech, “From R&D to M&A: Big Pharmas showcase ‘measurable impact’ of AI” (May 2026) — source for AstraZeneca’s and GSK’s earnings-call AI commentary referenced in Section 3: fiercebiotech.com

• Gartner, “Gartner Predicts 40% of Enterprise Apps Will Feature Task-Specific AI Agents by 2026, Up from Less Than 5% in 2025” (Aug 2025) — source for the 40%-by-2026 and $450B-by-2035 figures cited in Section 4, and the 40%+ agentic-project-cancellation-by-2027 prediction cited in Section 8: gartner.com

• RSM US, “SaaS vendors must adjust pricing models as agentic AI transforms the industry” (Mar 2026) — source for the “engineering its own revenue decline” framing cited in Section 4: rsmus.com

• Omdia, “New Omdia Analysis Shows Agentic AI Outpacing Growth Rates of Traditional Generative AI” (Sep 2025) — source for the $1.5B (2025) to $41.8B (2030) enterprise agentic AI software market forecast and 175% five-year CAGR cited in Section 4: businesswire.com

• MIT NANDA / Project NANDA, The GenAI Divide: State of AI in Business 2025 (Aug 2025) — source of the ~95% pilot-failure finding and the “learning gap” diagnosis cited in Section 5: full report PDF · widely covered, incl. Fortune

• Ellvero Insights, “AI ROI in 2026: Why 95% of Pilots Fail to Deliver and How to Measure What Actually Matters” (citing Boston Consulting Group’s 2026 AI at Scale survey) — corroborating 2026 follow-up data cited in Section 5: ellvero.com

• Narayanan, A. & Kapoor, S., “Why AI hasn’t replaced software engineers, and won’t: Coding agents as normal technology” (AI as Normal Technology, June 11, 2026) — source for the “decide-execute-deliver sandwich” framework, the Block/Snap/Intuit “AI washing” cases, the WARN Act and HBR anticipated-vs-realised figures cited in Sections 1 and 5, and the junior-pipeline discussion in Section 7: normaltech.ai/p/why-ai-hasnt-replaced-software-engineers

• Azilen Technologies, “Agentic AI in Financial Services [2026 Definitive Guide]” (citing KPMG and McKinsey) — source for the 2.3x ROI within 13 months and 20–60% credit-analysis productivity figures cited in Section 5: azilen.com/blog/agentic-ai-in-financial-services

• Neurons Lab, “Agentic AI in Financial Services: A Research Roundup for 2026” (citing IDC, McKinsey, PwC, Deloitte) — corroborating source for the same 2.3x ROI figure cited in Section 5: neurons-lab.com/articles/agentic-ai-in-financial-services-2026

• What Is Any Agentic Structure Actually Worth? InterestingEngineering++

This is the first of three pieces that, together, ask one question from three angles: when an organisation adopts agentic AI, where does the impact actually land, and how big is it really, or how big is it really expected to be? These questions are critical given much market froth and excitement around “agentic systems”. I care less about the numbers than about the debates, direction, and strategies that may be applied.

1. Title: What the Architecture Is Worth (This piece)
   A $12 Billion Pharma Company Tests (Ten) Agentic Designs Against the Same Crisis— readers walk away with: how to think around what an architecture choice is worth. You should walk away with more questions than answers, which is the point of the exercise — which of various AI designs wins, loses, or fails outright when tested against the same crisis, and the enterprise-value gap between them.

2. Two Economies, One Technology
   Why the Same Technology Caps Out as a Saving in One Function and Opens Into a Bet in Another— readers walk away with: a better way to judge whether an AI claim about their industry is plausible — why the same technology produces a small, capped saving in one function and a large, uncertain bet in another, and which is which.

3. The Funnel, the Floor, and the Structure
   "A Value-Chain Framework for Where AI Impact Lands"— readers walk away with: a three-part map for locating any AI effect, in any sector — so the next AI headline can be sorted into “bounded saving,” “long-shot bet,” or “who-gets-paid shift” before deciding how much it matters.

This piece starts with the narrowest, most concrete version of that question — inside a hypothetical single $12 billion pharmaceutical company, facing a real supply-chain crisis, what is the difference in enterprise value between choosing the right AI architecture and choosing a “plausible-but-wrong one”? By now, you would have realized that not all “agentic designs” are created equal! It builds on the foundation of the “Harness Lab Series” (more references found below, but you can start with the link provided and unwind things from there).

Given how young the space is, the analysis does its best to hypothesize, but the real value should be in organizations potentially using this as a helpful guideline (which you can tweak as thoughts arise, and the specifics of your businesses and strategies are ironed out), basically, scenario planning around how the allocation of resources towards Agentic AI, might pan out. What risks to look out for, and how the organization itself might prioritize team or departmental adoption, including setting up a strategy around it. Use it as you see fit.

The second piece, "Two Economies, One Technology," picks up the answer from this case study and asks why it (investments in Agentic AI) might feel smaller than the claims circulating about AI's economic impact — arguing that the answer depends on which part of a business, and which industry, is doing the talking.

The third piece, "The Funnel, the Floor, and the Structure," completes the arc with a framework for locating any AI-driven effect, in any sector, by where it sits in a value chain — and shows, in hindsight, that everything measured in this first piece was one specific kind of effect all along. Read in sequence, the three pieces move from one company's balance sheet to an industry-spanning map; read on its own, this piece is a self-contained case study of what ten different agentic architectures are actually worth when tested against the same crisis.

How to Read This Case Study

This case study has two layers, and keeping them separate is the key to following the argument. The first layer is a piece of research: ten AI “harness” designs — numbered H1 through H10 — were built and tested against the same task, and their quality, speed, and cost were measured. That research is real and is documented in the Harness Engineering Series articles referenced throughout. The second layer is this case study itself: a fictional pharmaceutical company, PharmaCo, that uses those same ten harness designs and experiences the financial consequences of choosing well or badly between them.

Every harness discussed in this case is referred to only by its number from the original H1–H10 research “The Harness Lab” and “The Harness Lab, Automated”— there is no separate alphabetical naming scheme here. Four of the ten matter most to the PharmaCo story:

• H2 — a single AI agent following a structured, step-by-step reasoning script. This is the architecture PharmaCo ultimately recommends for its hardest task: responding to a supply chain disruption.

• H3 — a single AI agent that calls external tools (databases, live status checks) as it works. This is the architecture that fits PharmaCo’s second task: routine daily compliance monitoring across 40+ markets.

• H9 — a five-agent “swarm” with an orchestrator that divides the task among specialists and stitches their outputs together. This is the architecture PharmaCo’s management initially preferred — and the one that turns out to cost $19.8M a year more than H2 on the disruption task.

• H4 — three agents working independently on the same task with no shared context, their outputs simply concatenated at the end. In the original research this was the weakest design tested, and it appears here as the cautionary worst case: what happens if an organisation deploys an architecture that looks parallel and thorough but has no mechanism for reconciling contradictions.

The case study is organised in three movements. Sections 1–2 set the scene — who PharmaCo is, what an agentic AI system and a harness actually are, and what the underlying research found across all ten architectures. Sections 3–9 walk through a single disruption event four times — once for each of H2, H9, H4, and H3 — showing what each architecture actually did and what it cost. Sections 10–13 step back to the financial summary, the shareholder-value calculation, and the governance questions the case raises — including a question this revision adds: if agentic AI changes the cost side of the ledger this dramatically, what happens to the people side?

A reminder that the numbers are less important than the depth of discussions and actions that surround scenario planning applied to the implementation of agentic systems. Keep a list of questions that arise, so that these can be applied and tailored to your organizations circumstances. While market hype abounds, think twice before applying. So here it is. Your case study.

Executive Summary

PharmaCo International — an assumed fictional $12 billion global pharmaceutical company — deployed AI agents in two areas of its operations: supply chain disruption response and routine regulatory compliance monitoring. In each area, management had to choose between several possible AI “harness” architectures, and that choice turned out to be one of the most consequential capital allocation decisions the company made that year.

The central finding is counterintuitive. On the disruption response task, the most architecturally elaborate system tested — H9, a five-agent orchestration swarm — produced worse financial outcomes than H2, a single agent following an explicit structured reasoning script. The gap between choosing H2 and choosing H9 was est at $19.8 million per year in avoidable cost, or roughly $297 million in enterprise value at PharmaCo’s earnings multiple. But the difference could actually be wider: ($297M–$986M EV range). A third option, H4 — three independent agents with no shared context — performed worse still: $65.7 million per year worse than H2, and in the original research, H4 was the second-weakest of all ten architectures tested, scoring below the bare-model baseline.

On the second task — routine compliance monitoring across more than 40 markets — the ranking reverses. H3, a single agent equipped with live tool access to regulatory databases, outperforms H2, because the value in this task comes from retrieving current external data rather than from structured internal reasoning. Replacing three human analysts with H3 saves PharmaCo roughly $2.0 million a year while improving review frequency from fortnightly to daily and accuracy from 85% to 97%.

Three findings run through the whole case. First, there is no universally best architecture — the right choice depends on the task, and the same architecture (H9) that loses badly on the disruption task could not even be meaningfully applied to the compliance task. Second, complexity and cost are not the same as quality: H4 was cheaper to build than H2 and far more expensive to run. Third, a system that scores perfectly on a quality rubric is not automatically safe — adversarial testing caught a conditional failure mode in H3 that the standard scoring missed entirely.

Section 1 — The Research Foundation: What H1–H10 Actually Found

Before applying any of this to PharmaCo, it is worth looking directly at the research the case study draws on, because the PharmaCo numbers are a financial translation of these results — not a separate study with its own winners and losers.

1.1 The Original Harness Lab Benchmark

The original Harness Lab experiment tested ten harness architectures — H1 through H10 — against a single benchmark task: producing a disruption response plan for a pharmaceutical supply chain crisis, scored against a pre-specified gold answer with deliberate traps built in (a regulatory deadline, a carrier-specific routing requirement, and a supplier ordering dependency). Three metrics were recorded for every architecture:

• Alpha (α) — quality, measured as the share of gold-answer criteria the system’s output satisfied, from 0 to 1.0.

• Lambda (λ) — latency, the time in milliseconds the architecture took to produce its output.

• Kappa (κ) — total tokens consumed, a proxy for compute cost.

The full results across all ten architectures:

Reading the table

H2 — a single agent following a structured reasoning script — was both the highest-quality result (α = 0.920) and among the cheapest and fastest. It dominates on all three axes. H5 (an agent that drafts, evaluates its own output against criteria, and loops until done) and H7 (an agent that routes sub-tasks to different models) both reached α = 0.840 — strong results, achieved through iteration and tool/model variety rather than a fixed script. H6 (skill memory) reached 0.900, close behind H2. H3 (sequential tool calls) only reached 0.600 on this particular benchmark — useful tools, used in sequence, but without the structured reasoning scaffold that kept H2 on track through the deliberate traps.

H4 (three agents running in parallel with no shared context, outputs simply concatenated) scored 0.440 — below the bare, no-harness baseline (H1, 0.665). This is the headline result that justifies H4’s role later in this case study as the cautionary worst case: it is not merely “not the best,” it is the design that an organisation would have been better off not building at all. H10 (a meta-harness that spawns and manages other harnesses) effectively broke down on this task (α = 0.230, no usable latency figure) and is not carried forward into the PharmaCo narrative.

1.2 The Automated Follow-Up: Workflow Patterns and the Sigma Metric

A second piece of research — ‘The Harness Lab, Automated’ — took a different approach. Rather than hand-building each harness, it used Claude Code’s dynamic workflow patterns (fan-out-and-synthesize, loop-until-done, tournament, adversarial verification, generate-and-filter, classify-and-act) to generate, test, and tournament harness designs algorithmically. This second study was deliberately run on a simplified benchmark — the same task family, but with no deliberate traps and a small set of binary evaluation criteria — to test whether the automated workflow tooling itself worked correctly, not to re-run the original hard benchmark.

On that simplified benchmark, the designs that performed best were the tool-using and iterative-loop styles — directly comparable to H3 (sequential tools) and H5 (eval loop) from the original table. This makes sense: on an easy task with no traps, broad tool coverage and iterative self-checking win, because there is nothing precision-critical for them to miss. The structured-script approach of H2 carries scaffolding overhead that pays off on hard, trap-laden tasks but is unnecessary on simple ones.

This second study also introduced the Sigma (Σ) metric — quality divided by a complexity penalty (Σ = α ÷ (1 + Δ), where Δ captures structural complexity such as agent count and coordination overhead). Sigma explains why H2 still leads even against architectures with comparable raw quality: H9’s α = 0.815 comes with substantially more coordination machinery than H2’s α = 0.920, so on a quality-per-complexity basis the gap between them is far larger than the raw α numbers alone suggest.

Section 2 — About PharmaCo International

2.1 Company Profile

PharmaCo International is a fictional composite of a major global pharmaceutical manufacturer. Its financials and operations are modelled on industry averages for a large-cap pharmaceutical company as of 2026. All figures are illustrative.

2.2 How a Pharmaceutical Supply Chain Works

For readers unfamiliar with the industry: medicines move through a multi-tier chain before they reach a patient.

• Tier 3 suppliers provide raw chemical feedstocks (petroleum derivatives, minerals, plant extracts).

• Tier 2 suppliers turn those into packaging materials and excipients — the inactive ingredients that give a tablet its shape, taste, and colour.

• Tier 1 suppliers produce the Active Pharmaceutical Ingredient (API) — the molecule that actually works.

• PharmaCo’s 15 manufacturing sites combine API and excipients into finished product (tablets, vials, inhalers), which then passes through quality control — every batch tested and regulatory approval required per market — before moving to regional distribution (with cold-chain logistics at 2–8°C for some products) and on to hospitals, clinics, and retail pharmacies.

Why it matters financially

Pharmaceutical manufacturers maintain 30–60 days of buffer stock as standard practice. This sounds like comfortable protection, but it is not: once that buffer depletes, revenue stops, patients switch to alternative brands, and relationships with hospitals and pharmacies erode — some permanently. Every day of unmitigated disruption has a measurable revenue cost.

2.3 The Three Products at the Centre of This Case

PharmaCo’s Hormuz Strait supplier disruption affected three medications. These are real drug categories — the financial figures are modelled on typical pricing and margins for each category.

Section 3 — What Is an Agentic AI System, and What Is a Harness?

3.1 From Chatbot to Agent

Most people’s experience of AI is a chatbot: you ask a question, it gives an answer. An agentic AI system is fundamentally different. It does not just respond to questions — it takes sequences of actions toward a goal, uses tools, coordinates with other agents, evaluates its own output, and continues until a task is complete.

A chatbot asked ‘which supplier can cover our Lisinopril shortage?’ gives a single response — ‘consider suppliers X, Y, or Z’ — and the human must act on it, verify it, and coordinate the next step. An agent given ‘handle the Hormuz disruption for Lisinopril’ reads the supply data, queries alternative supplier databases, checks regulatory rules, drafts a procurement plan, scores that plan against criteria, revises if incomplete, and files the regulatory notice — arriving at a completed task for a human to review and approve.

3.2 What a Harness Is

A harness is the structural design of how an AI agent — or multiple agents — is organised to complete a task. Think of it as the organisational chart and workflow of the AI system. Two agents given the same task but structured differently will produce different results, at different speeds, with different accuracy, and at different cost.

The harness choice is a capital allocation decision. A simple, well-designed harness costs less to build, less to run, and — on the right task — produces better results than a complex one. A complex harness has higher coordination overhead, more opportunities for errors to compound, and higher compute costs. Section 1 showed the research behind this: ten harness architectures tested against the same benchmark task, with measured financial consequences for each choice.

3.3 The Architectures in This Case

PharmaCo evaluated four of the ten H1–H10 architectures for its two deployment areas. Each represents a different structural approach to the same family of problems.

Section 4 — The Strategic Decision

4.1 Where in the Value Chain PharmaCo Chose to Deploy

PharmaCo did not deploy AI agents everywhere simultaneously. It identified three specific areas of its value chain where the investment case was clearest: supply chain disruption response, routine regulatory compliance monitoring, and inventory replenishment decisions. This case study focuses on the first two.

4.2 Why Architecture Choice Has Financial Consequences!

When most organisations adopt AI, they ask: will it work? The more important question is: which version will work best, and what does the difference cost? The answer, as this case study demonstrates, runs to hundreds of millions of dollars in enterprise value.

Four financial levers connect harness architecture to P&L performance:

• Lever 1 — Response speed → revenue at risk. The faster a usable response plan is ready, the fewer days of unmitigated disruption, and the less revenue is lost to patient switching.

• Lever 2 — Decision precision → emergency procurement cost. The quality of supplier identification determines the premium paid over normal procurement costs.

• Lever 3 — Regulatory compliance → fine avoidance. Missing a notification deadline triggers fines; a precise, time-stamped action item avoids them.

• Lever 4 — Inventory efficiency → write-off avoidance. Accurate volume and formulation modelling avoids over-purchasing and wrong-SKU write-offs.

Lever 2 in detail: how the procurement premium calculation works

Step 1 — Define the emergency procurement volume. A 30-day disruption; buffer stock covers the first 15 days. Days 16–30 must be sourced from alternatives: 15 days of COGS. Monthly COGS = $25.5M → 15-day COGS = $12.75M. Emergency procurement needed: $12.75M base volume.

Step 2 — Apply a premium rate that depends on how precisely the response plan identifies alternatives:

Lever 3 in detail: the regulatory backdrop

Pharmaceutical companies operating in regulated markets are legally required to notify regulators of significant supply disruptions. In the United States, FDA regulations under 21 CFR 314.81 require manufacturers to report potential drug shortages ‘as soon as practicable,’ with documented FDA practice establishing 48–72 hours as the expected window for initial notification. European markets have an equivalent EMA requirement under Regulation (EU) 2022/123. Late or absent notifications have resulted in fines ranging from $250,000 to $2.5M per violation in documented FDA enforcement actions.

Lever 4 in detail: inventory write-offs

Emergency procurement during a supply crisis involves purchasing expensive alternative stock under uncertainty. If the response plan over-estimates the shortfall, PharmaCo purchases more than it needs — and that excess, sourced at premium prices with shorter shelf-life from alternative suppliers, may need to be written off. If the plan under-estimates, the disruption extends.

Section 5 — The Disruption Event: Step by Step

5.1 What Happened on Day Zero

At 06:14 on a Tuesday morning, PharmaCo’s Global Supply Intelligence system received notification from a Tier-1 API supplier in the Hormuz Strait region: production was suspended effective immediately, with an expected outage of 30 days. Three of PharmaCo’s highest-volume medications were affected.

The notification triggered a 72-hour countdown to the point where PharmaCo’s buffer stock would begin to deplete for Salbutamol — the most time-critical of the three medications because it is used in emergency respiratory situations, hospitals maintain minimal buffer stock for it, and patients cannot substitute easily. The clinical and financial consequences of a Salbutamol shortage are more severe than the other two products.

What the response plan must include

• Patient impact assessment per medication

• Alternative supplier identification

• Regulatory notification within the 48–72 hour window

• Emergency procurement authorisation

• A 24-hour escalation checkpoint

The timeline that matters

5.2 The Baseline: What This Costs With No AI At All

Before comparing H2, H9, and H4, it is worth being explicit about something the rest of this case study takes for granted: this disruption happens regardless of what software PharmaCo runs. A Tier-1 API supplier in the Hormuz Strait region does not suspend production because of, or in spite of, PharmaCo’s choice of AI architecture. The supply shock is exogenous. The only thing that changes across H2, H9, and H4 is how PharmaCo responds to a crisis that would occur in any case.

Section 10.2 quantifies what this disruption costs PharmaCo with a purely manual, human-led response — no AI involved at any stage. Across the four disruption events PharmaCo experiences per year, that baseline comes to approximately $45.44M in annual exposure. This is the number that exists whether or not PharmaCo has ever heard of agentic AI.

Section 6 — H2: The Structured Single Agent (Recommended Outcome)

6.1 What H2 Did

H2 is a single AI agent with no sub-agents and no external tools. Its capability comes entirely from its internal structure: the prompt that instructs it includes an explicit task restatement at the top, numbered reasoning steps the agent must follow in sequence, constraint reminders embedded at each step, and a specified output format. This structure prevents the agent from taking shortcuts, missing constraints, or producing vague generalisations.

On receiving the disruption notification, H2 processed the supply data, identified patient impact by medication and by market, retrieved qualified alternative supplier information from PharmaCo’s supplier database, modelled emergency procurement volumes, identified regulatory notification obligations with the specific 48-hour window, and produced a draft CFO-level response brief — all within six hours.

H2’s internal structure

• Task restatement: ‘The task is to produce a 72-hour disruption response plan for the Hormuz API supplier suspension…’

• Step A — identify affected medications and risk tier

• Step B — assess patient impact per medication (constraint check: do not recommend suspension)

• Step C — identify alternative suppliers, Tier-1 only

• Step D — model procurement volumes (constraint check: regulatory notification under 48 hours)

• Step E — draft response brief in the specified format: executive summary, actions by priority, timeline, approvals required, regulatory filing reference

Result: response brief ready in 6 hours (0.25 days).

6.2 The Financial Outcome — H2

Section 7 — H9: The Orchestration Swarm (Over-Engineered: $19.8M Annual Overspend)

7.1 What H9 Did — and Why It Cost More

H9 was PharmaCo management’s first instinct. A complex disruption requires complex analysis, the reasoning went — and five specialist agents working in parallel should produce a more thorough result than one. This intuition is reasonable. It is also wrong for this specific task.

H9’s five agents ran in parallel: one assessing patient impact, one modelling supply alternatives, one handling regulatory requirements, one drafting communications, and one coordinating logistics. An orchestrator agent then synthesised all five outputs. The structural problem: synthesis across five independent agents introduces inconsistency. The regulatory agent recommended notifying one authority; the logistics agent’s timeline assumed a different notification window. The orchestrator produced a coherent document, but the coherence required averaging out the inconsistencies — and in doing so, lost the specific regulatory deadline that H2 preserved.

7.2 The Financial Outcome — H9

Section 8 — H4: Independent Parallel Agents (Catastrophic: $65.7M Annual Overspend)

8.1 What H4 Did — and How It Failed

H4 appeared technically sophisticated: three agents running in parallel, each processing the same disruption data independently, with outputs combined at the end. The design logic was that three independent perspectives would catch more than one. What it missed is that independent agents, with no shared state and no reconciliation mechanism, will produce contradictory outputs — and concatenating contradictions produces incoherence, not comprehensiveness.

This is consistent with H4’s standing in the original research: of the ten architectures tested in Section 1, H4 was the second-weakest, scoring 0.440 — below the no-harness baseline (H1, 0.665). H4 is included in this case study precisely because it represents what happens when an architecture that looks like ‘more coverage’ on paper is deployed on a task where consistency across outputs is essential.

In practice: Agent A recommended temporarily suspending Salbutamol to protect Metformin supply — directly violating the task constraint. Agent B recommended three different suppliers than Agent C, with conflicting pricing estimates. Agent C’s regulatory section referenced a different notification authority than Agent A. The combined output required human intervention to identify and resolve all three contradictions before any action could be taken. That intervention took four days.

8.2 The Financial Outcome — H4

Section 9 — H3: Tool-Augmented Single Agent for Routine Compliance Monitoring

9.1 A Different Task Requires a Different Architecture

PharmaCo’s second deployment area was routine regulatory compliance monitoring: daily checking of the company’s compliance status across 40+ markets. This is a fundamentally different task from disruption response. It is repetitive, well-defined, and has clear binary outcomes — a market is either compliant or it is not. There are no deliberate traps, no judgment calls about constraint violations, no crisis decision-making under pressure.

This distinction matters enormously for architecture selection. Section 1’s research found that the architecture which wins on a hard, trap-laden task (H2) does not win on a simple, coverage-based task. H3 — a tool-augmented single agent that queries regulatory databases directly — outperforms H2 here, because it can retrieve live regulatory status information that H2’s internal reasoning scaffold has no way to access. This is the same pattern the automated follow-up study found when tool-using and iterative-loop designs (H3- and H5-style) won on the simplified, no-traps benchmark (Section 1.2).

H3’s structure

• Receive the daily compliance check list (40+ markets)

• For each market: query the regulatory database tool, query the local market status API, cross-check against PharmaCo’s obligations — each step a live tool call

• Flag any compliance gaps

• Generate a daily compliance report

Result: covers all 40+ markets daily at 97% accuracy. Previously: human review every two weeks at 85% accuracy.

9.2 The Financial Case for H3

The financial case for H3 is different from the disruption scenarios: it is primarily a cost-reduction story rather than a loss-avoidance one. Three previous compliance analysts performing manual monitoring are replaced by an automated system. Analytical quality improves, coverage improves, and cost decreases substantially.

Section 10 — The Complete Financial Comparison

10.1 Per-Event Impact Summary

10.2 Annual Impact (4 Disruption Events per Year)

* H4 additional cost includes $0.5M per coherence-failure event × 4 events = $2.0M in human override costs annually. Both bottom rows describe the same underlying gap from two directions: H2 saves $20.41M/yr relative to the human baseline, which is the same as saying the human baseline costs $20.41M/yr more than H2.

10.3 Income Statement Impact (Annual, Affected Products Only)

The following shows how each scenario affects PharmaCo’s income statement for the three affected product lines, using the annualised figures from 10.1 and 10.2 (per-event impacts × 4 events/year). The ‘No disruption’ column is a reference point only — per Section 5.2, PharmaCo cannot choose a world in which the Hormuz disruption does not occur. The real choice is between the Human Baseline column and the H2/H9/H4 columns, all four of which assume the disruption happens.

* The ‘No disruption’ column shows a higher operating income than H2 by construction — a world with no disruption at all will always outperform a world with one, regardless of response quality. This column exists to make the disruption-cost row legible (it is the gap between this column and each scenario’s OI), not to suggest ‘do nothing’ is an available choice. Row 12 reconciles exactly with the ‘Annual disruption cost’ row in Section 10.2: $24.80M / $44.60M / $90.50M / $45.44M.

Reading the Human Baseline column against H2: operating income improves by $20.64M/yr (close to the $20.41M total-exposure saving in 10.2; the small gap is H2’s $0.15M implementation cost, which sits below the operating-income line). This is the comparison that matters — not H2 vs. ‘no disruption’, which PharmaCo cannot choose.

10.4 Cash Flow Impact

Beyond the income statement, the harness choice affects PharmaCo’s cash flow directly. Emergency procurement is a cash outflow; larger emergency purchases mean more working capital tied up in inventory. Regulatory fines are immediate cash outflows. Faster response (H2) means shorter disruption periods, reducing cash tied up in buffer stock purchased at premium prices.

10.5 Shareholder Value Calculation

Shareholder value is calculated by applying PharmaCo’s price-to-earnings multiple to the annual income impact. A 15× P/E multiple is used — the mid-point of the 12–20× range typical for major pharmaceutical companies with diversified portfolios. The multiple reflects the market’s expectation that PharmaCo’s earnings improvement is sustainable and will recur.

The P/E multiple applies because the income improvement is sustainable and recurring — four disruption events per year, every year. A one-time saving would not attract a multiple; a systematic improvement in disruption response capability does.

Section 11 — Turning Dials: Benchmarks, Workforce, and Financial Sensitivity

11.1 The Benchmark Is a Financial Specification

Section 1 of this case showed two different research results from two different benchmarks: the hard, trap-laden original H1–H10 benchmark, where H2 dominates, and the simplified, no-traps follow-up benchmark, where H3/H5-style tool-and-loop designs win. Those traps were the financial levers in disguise. The carrier-specific routing requirement maps to the procurement premium differential. The regulatory notification window maps to the $1.5M fine that H9 incurred. The supplier ordering dependency maps to the inventory write-off.

The benchmark is not a quality measure. It is a specification of what ‘winning’ means in financial terms.

11.2 The Harness Selection Framework as a Capital Allocation Decision

Four variables determine which harness architecture produces the best financial outcome for a given deployment. Each corresponds directly to a financial lever in this case study.

Change any one variable and the optimal architecture shifts. A company with poor documentation deploying H2 would not achieve the same results — H2’s structured reasoning scaffold requires solid information inputs to be effective. A company deploying on a low-stakes routine task does not need H2’s precision overhead. Choosing the architecture without assessing these variables is not a neutral decision — it has a quantifiable financial cost. For teams: Brainstorm agentic design architecture and understand the potential financial impact may apply differently.

11.3 Beyond Cost: What Happens to the Workforce When the Savings Are Real?

Everything in Sections 6–10 of this case is framed as cost avoidance or cost reduction — the $19.8M H2-vs-H9 gap, the $490K labour saving on compliance monitoring. That framing is correct, but it is incomplete, because it treats the organisation on the other side of these numbers as static. If PharmaCo genuinely captures these savings — three compliance analysts’ worth of labour redirected, disruption-response teams no longer spending days reconciling a five-agent swarm’s contradictions — the workforce composition around those roles does not stay the same.

A useful external reference point here is the way Anthropic itself has described its own hiring shift as agentic systems took over more of the routine technical work. Rather than continuing to scale headcount of engineers doing implementation-level work, the organisation has leaned toward smaller, faster-moving teams and a broader mix of interdisciplinary roles — including legal and philosophical expertise alongside technical roles — with people operating more like managers of AI-driven workflows than individual contributors executing tasks by hand.

Applied to PharmaCo, the same logic suggests the $1.84M Year-1 benefit from H3 compliance monitoring is not simply ‘three analyst salaries removed from the cost line.’ It is also three analysts’ worth of time that can be redirected toward judgment-heavy work the AI system cannot do: interpreting borderline compliance findings, managing relationships with regulators, and feeding edge cases back into the system’s gold-answer criteria so H3 keeps improving. Similarly, the disruption-response team freed from four days of H4-style contradiction resolution per event is not simply ‘four fewer days of work’ — it is four days that could be spent on supplier relationship management, scenario planning for the next disruption, or governance work of the kind Section 12 describes.

This is not a fully worked financial model — unlike the rest of this case study, it is not derived from the H1–H10 research or from sourced industry benchmarks. It is included as a discussion prompt for Section 13, because the original case study, which focused entirely on architecture-to-cost mapping, did not consider it.

11.4 The Decide-Execute-Deliver Sandwich, and a Limit on the Workforce-Composition Story

Section 11.3’s redirected-capacity argument rests on an assumption worth examining directly: that the freed-up time goes somewhere productive. A useful framework for testing that assumption comes from Arvind Narayanan and Sayash Kapoor’s analysis of knowledge work generally, which models most jobs as a ‘decide-execute-deliver sandwich’ — AI compresses the execute layer in the middle, but the decide layer (what should be reviewed, escalated, or changed) and the deliver layer (who is accountable when something is missed) resist automation and, if anything, become more important once execution is fast and cheap.

Applied to H3’s compliance-monitoring role: H3 compresses the execute layer (continuous monitoring against the rule set). The decide layer — which borderline findings warrant escalation, how the gold-answer criteria should evolve, what PharmaCo’s compliance posture should be heading into the next regulatory cycle — and the deliver layer — who signs off on the compliance function’s output to the board and regulators — both remain with PharmaCo’s compliance team, regardless of how good H3 becomes. Section 11.3’s framing is consistent with this: the $1.84M benefit was never proposed as a full headcount reduction, and the redirected-capacity argument depends on those three analysts moving into decide- and deliver-layer work, not out of the organisation.

The risk this creates sits one hiring cycle downstream, not in the current year’s numbers. If PharmaCo responds to H3’s adoption by not backfilling the next junior compliance-analyst role — reasoning that H3 now covers the execute-layer work that role used to do — the organisation saves a salary in Year 1 but loses the route by which junior analysts become the senior analysts who do PharmaCo’s decide- and deliver-layer compliance work in five to ten years. Execution-layer work has historically been where junior staff build the judgment senior roles require; if AI absorbs that work before junior hires gain it, the pipeline narrows in a way that is invisible in any single year’s income statement and expensive to reverse once a senior-talent shortage becomes visible.

This is not an argument against adopting H3 — the $1.84M benefit (Section 9) stands regardless. It is an argument that the board review proposed in Section 12.1 for any architecture change should explicitly separate two questions that are easy to conflate: ‘does this change reduce this year’s compliance cost’ (yes, by $1.84M) and ‘does this change alter our (junior) hiring plan for this function’ (a separate decision, with consequences on a much longer horizon, that the cost figure alone does not capture).

Section 12 — Board and Shareholder Implications

12.1 The Capital Allocation Question

The question before PharmaCo’s board was not whether to invest in agentic AI. That decision was straightforward: the $150K investment in H2 generates $20.41M in annual savings versus the human baseline, a payback period of under a week and a first-year ROI exceeding 89×. No CFO would decline that proposal.

The more consequential question was: which architecture? And the answer to that question was worth [$297M] in enterprise value (your organization will have different factors apply) — the gap between choosing H2 and choosing H9. Both are AI investments. Both represent meaningful technology deployments. The difference between them, on this specific task, compounds annually.

12.2 What Adversarial Verification Added

One finding from the underlying research deserves specific board attention. The automated evaluation workflow included an adversarial verification stage — a separate agent whose sole purpose was to find flaws in the winning architecture’s output. This stage identified a vulnerability in H3 (the compliance monitoring system) that the standard quality scoring had not caught: H3’s output was conditionally correct — accurate when live tool data was available, but potentially non-compliant when tool access was simulated or unavailable.

In financial terms: without adversarial verification, PharmaCo would have deployed a system with a conditional compliance gap. The first time that condition was met in production — a tool API outage, a database maintenance window — the $1.5M expected fine saving could reverse into a live fine. The adversarial stage cost approximately $800 in compute (two Sonnet API calls) and potentially saved $2.5M in avoided fines. This is the financial case for adversarial verification as a standard deployment gate, not an optional quality check.

Section 13 — Discussion Questions

This case is designed for use in graduate-level courses in operations management, digital strategy, healthcare management, and financial analysis. The following questions are intended to guide discussion. You may also find it useful as part of any organizational deployment planning strategy. It will never be an end all, because paths that may be taken, are realistically typically budget and also imagination constrained.

• PharmaCo’s management team initially preferred H9 because a complex disruption seemed to warrant a complex AI system. Was their instinct unreasonable? Under what conditions would H9 have been the correct choice, and how would the financial analysis change?

• The case shows that H2 saves $19.8M per year versus H9, while costing $300K less to build and $190K less per year to run. Yet H9-style multi-agent swarms are still widely deployed in industry contexts similar to PharmaCo’s. What organisational and behavioural factors might explain why companies consistently over-engineer their AI systems?

• H3 outperformed H2 on the routine compliance monitoring task but would have underperformed on the disruption response task. How should PharmaCo’s technology governance framework distinguish between tasks to ensure the correct architecture is deployed in each case?

• The benchmark design — specifically the deliberate traps embedded in the disruption scenario — was the mechanism that revealed H9’s financial weakness. If PharmaCo’s benchmark had been poorly designed (no traps, general criteria), the analysis would have recommended H3 for disruption response. How should a board evaluate the quality of an AI benchmark before approving a deployment decision based on it?

• The adversarial verification stage identified a conditional vulnerability in H3 that saved an estimated $2.5M in potential fines at a compute cost of approximately $800. How should organisations quantify and mandate adversarial testing as a standard step in AI deployment governance?

• Section 11.3 argues that the $1.84M H3 saving is also three analysts’ worth of redirectable capacity, drawing a loose analogy to how AI labs have restructured their own hiring toward smaller, more interdisciplinary teams. Is this analogy sound for a pharmaceutical company’s compliance function? What would PharmaCo need to do — in role redesign, training, or governance — to actually realise an earnings benefit from redirected capacity, rather than simply not backfilling the roles?

• Section 11.4 distinguishes between two questions that are easy to conflate when an architecture like H3 is adopted: ‘does this reduce this year’s cost’ and ‘does this change our junior hiring plan for this function.’ The first is visible in Section 10’s figures; the second is not. How would you design PharmaCo’s annual technology-governance review (Section 12.1) so the second question is asked explicitly — and on what timescale would you expect its consequences to become measurable?

Section 14 — Building PharmaCo’s Internal Harness Lab

Sections 6–9 treated H2, H9, H4, and H3 as a fixed menu — four architectures, already tested, with known costs and known outcomes. In practice, PharmaCo’s AI engineering team would not stop at four. The underlying research (‘The Harness Lab, Automated’) describes a workflow in which an AI agent generates new harness candidates, tests them, adversarially probes them, runs a tournament, and loops until a quality threshold is met — with minimal human stage-management. What is commonly called “Loop Engineering” (a very loose term, but the examples given should be illustrative in its applications). This section sketches what that internal capability would look like for PharmaCo, and where its limits are.

14.1 From a Fixed Comparison to a Continuous Search

H2 was the winner of the original ten-architecture comparison, with an Alpha of 0.920 — comfortably ahead of the field. But ‘comfortably ahead of the field’ is not the same as ‘optimal.’ Three other architectures sit in a cluster just behind H2: H6 (two-agent chain with revision handoff, α=0.900), H5 (self-revision loop, α=0.840), and H7 (three-agent with model routing, α=0.840). None of these were used in PharmaCo’s deployment scenarios — they were neither the winner, the predicted-but-wrong swarm, the cautionary failure, nor the alternate-task winner, so they had no role in the narrative. But they are not irrelevant. They are the nearest neighbours to H2, and under a Sigma-optimising search they are the most promising starting points for finding something that beats it.

14.2 What a Cherny-Style Prompt Would Look Like for PharmaCo

The research describes a shift from a long, stage-by-stage prompt to a short goal statement plus a project-level instructions file (CLAUDE.md) that carries all the rules, scoring logic, and constraints. Applied to PharmaCo’s disruption-response harness, the top-level instruction the AI engineering team would issue might be reduced to something like:

/goal Sigma >= 0.95 for the disruption-response harness, seeded from H2 (current champion, Sigma=0.920÷(1+Δ)). Compare against H5, H6, and H7. Read CLAUDE.md for the benchmark, scoring rules, and constraints. Generate, adversarially probe, tournament, and loop until the goal is met or the token budget is exhausted. Produce a recommendation with full Sigma breakdown.

Everything else — the five binary criteria from the original gold answer, the requirement to never recommend suspending a medication, the token and time budgets, and the instruction to flag any design that relies on memorised test cases rather than general reasoning — would live in CLAUDE.md, not in the prompt. The agent decides whether to start from H6’s revision handoff and add H2’s reasoning scaffold, or start from H2 and add H7’s model-routing step for the highest-stakes sub-decisions, or some combination neither architecture used on its own.

14.3 The LFD Discipline: Why the Hard Benchmark Still Matters

The Loss Function Definition (LFD) framing adds three disciplines that are directly relevant to a regulated company running this kind of search:

• Blind, expanded evaluation. The gold-answer criteria PharmaCo uses for disruption response should not be visible to the generating agent, and should be expanded well beyond the original five-criteria set — including the cold-chain and air-freight-override traps that distinguished H2 from H9 in the first place. A search that can see its own exam will optimise for the exam, not the task.

• Forced entropy. If two consecutive search cycles produce no Sigma improvement, the instructions should require a genuinely different structural axis — not a smaller tweak to the same design. This is what prevents the search from converging on, say, twelve incremental variants of H6 that all score within noise of each other.

• Mandatory adversarial validation before promotion. Any design that beats H2 on Sigma must clear the same adversarial-probe stage that caught H3’s conditional compliance gap in Section 9, before it is considered for the live disruption-response role. A machine-generated winner that clears a quality threshold is a first draft, not a deployment candidate.

14.4 What This Does — and Does Not — Change About Sections 6–9

Two clarifications are important so this section does not overstate its implications. First, nothing here suggests H2 was the wrong choice for PharmaCo’s disruption-response deployment as analysed in Section 6 — H2 remains the best-supported architecture given the comparison that was actually run, and the financial figures in Sections 10–12 stand on that basis. Second, this section is not proposing PharmaCo replace H2 before any search has actually been run; it is describing what PharmaCo’s AI engineering function should be doing on an ongoing basis, the same way a pharmaceutical company continues lifecycle management research on an approved drug rather than treating approval as the end of the work.

14.5 A Note on H5, H6, and H7 Specifically

None of these hybrids exist yet — they are illustrative of the kind of candidate a Cherny/LFD-style search seeded on H2, H5, H6, and H7 would be expected to generate and test. The point of this section is not to predict the winner, but to show that PharmaCo’s harness selection should be treated as an ongoing search problem with a defined refinement frontier, not a one-time decision that ended when H2 was chosen.

14.6 The Sandwich, Applied to the Harness Lab Itself

Section 11.4 introduced the ‘decide-execute-deliver sandwich’ as a reason why PharmaCo’s compliance and disruption-response functions are unlikely to see significant headcount reductions even as H2 and H3 deliver real savings. The same framework applies, with almost no modification, to the harness lab described in this section — and the design choices already made in 14.1–14.5 turn out to be an instance of it, not a departure from it.

The Sigma-optimising loop in 14.2–14.3 is an execute-layer mechanism: given a goal (Sigma ≥ target), a seed (H2), and a refinement frontier (H5/H6/H7), it generates, scores, and revises candidate harnesses with minimal human involvement. That is the part agentic AI compresses. But two layers around it remain explicitly human, by the design choices this section already makes:

• Decide — what Sigma target to set, which architectures belong in the refinement frontier, and what counts as a ‘genuinely different structural axis’ for the forced-entropy rule (14.3) are judgement calls made by PharmaCo’s AI engineering team before the loop runs. The loop optimises within a space humans defined; it does not define the space.

• Deliver — the mandatory adversarial validation (14.3) and the board-level governance review (14.4, linking to Section 12.1) mean that no search output reaches a live role without a human-accountable sign-off step, regardless of how good the discovered architecture’s Sigma score is.

In other words, building ‘PharmaCo’s internal harness lab’ does not create a function whose entire job can eventually be automated away once the loop works well — it creates a function whose execute layer can be automated, with the decide and deliver layers becoming, if anything, more central to what the remaining human team actually does. This is the same pattern as H3 and compliance monitoring (Section 11.4), applied recursively to the team that builds and refines H3 in the first place. The practical implication for Section 14’s staffing: PharmaCo should expect the harness lab to need fewer people running benchmarks by hand over time, and more people deciding what to benchmark and signing off on what gets deployed — not fewer people overall on a predictable timeline.

Appendix A — Key Assumptions and Sources

Every number in this case study is derived from a stated assumption. This appendix summarises each assumption, its value, and its basis. Readers who wish to stress-test the financial model can substitute their own assumptions for any of these inputs.

A.1 Company and Market Assumptions

A.2 Product Revenue, Margin, and Buffer Stock Assumptions

A.3 Regulatory, Switching, Procurement, and Write-off Rates

A.4 Implementation and Operating Cost Build-Ups

Labour rates reflect 2025–2026 US market rates for specialist AI engineering (Levels.fyi, Glassdoor senior AI engineer compensation data, adjusted for contract rates): $260/hr senior AI engineer, $210/hr integration engineer, $180/hr QA, $150/hr documentation.

The $450K figure applies a higher contingency for multi-agent system complexity — consistent with Gartner’s documented finding that multi-agent AI implementations routinely exceed initial estimates by 25–40% due to integration complexity and inter-agent debugging overhead.

A.5 Annual Operating Costs and Response Times

Appendix B — Discussion Question Answer Frameworks

These frameworks are provided for instructors and self-study readers. They are not definitive answers — the questions are designed to produce genuine disagreement. Each presents the key tensions, evidence from the case, and counterarguments worth raising.

B.1 Was management’s preference for H9 unreasonable?

The instinct that a complex disruption requires a complex AI system is not unreasonable — parallel specialist expertise does outperform single-agent generalisation in many real-world scenarios. Management was wrong in this specific case, but their reasoning was not illogical.

H9 would have been the correct choice if the disruption affected 50+ products across five therapeutic categories simultaneously (a scope H2 would struggle to hold coherently in one context window); if the task required live data retrieval from multiple incompatible systems simultaneously, where parallel agents reduce latency; or if the organisation’s SOPs were fragmented across departments requiring separate agents to interface with separate knowledge bases.

H9 failed on this task because the task was bounded and well-specified — one event, three products, a handful of gold-answer criteria — something a single structured agent could hold in one context window without coherence degradation. The deliberate traps (carrier-specific routing, the 48-hour regulatory window, the ordering dependency) required precision that H9’s synthesis softened. In Sigma terms: H9 achieved α = 0.815 with Δ = 2.5, giving Σ = 0.233; H2 achieved α = 0.920 with Δ = 0.3, giving Σ = 0.708. H2’s quality advantage more than compensates for H9’s additional agent resources.

B.2 Why do companies consistently over-engineer AI systems?

Behavioural factors: complexity bias (a five-agent system is easier to justify to a board than a single structured prompt, even when the prompt performs better); vendor incentives (implementations are typically priced on scope, not outcome quality — a $450K H9 build generates more revenue than a $150K H2 build); and risk transfer (a complex system lets managers attribute failures to ‘the complexity of the problem’ rather than to the architecture choice itself).

Structural factors: benchmarks are often designed after architecture selection, so without a pre-specified gold answer with deliberate traps, no evaluation reveals H9-style precision degradation. The Sigma metric (quality per complexity unit) is not yet a standard industry metric — most evaluation frameworks report Alpha (raw quality) only, under which H9 looks competitive rather than significantly inferior. And procurement processes typically ask ‘can this system do the task?’ — a binary question — rather than ‘what is the most cost-efficient architecture that does the task?’

B.3 How should PharmaCo govern architecture-to-task matching?

• Framework-first governance: classify each task against the four variables in Section 11.2 before deployment, and map the classification to a recommended architecture range; decisions outside that range require sign-off from a Chief AI Officer or equivalent. This is preventive.

• Evidence-first governance: require a benchmark test with a pre-specified “gold answer” and deliberate traps before any architecture goes to production, with a minimum Sigma threshold for approval. This is evaluative but requires benchmark-design capability most organisations do not yet have.

• Adversarial-gate governance: require every proposed architecture to pass an adversarial verification stage — a separate agent or team whose sole purpose is to find failure modes before deployment. This is the model that would have caught H3’s conditional vulnerability (Section 12.2).

The case evidence supports a combination of framework-first and adversarial-gate governance as minimum requirements.

B.4 How should a board evaluate benchmark quality?

The most underappreciated risk in AI governance is not a bad AI system — it is a bad evaluation of one. A board should ask five questions of any benchmark used to justify a deployment decision:

• Does the benchmark include deliberate traps — cases where a plausible but incorrect answer scores well on a general rubric but fails a specific criterion? Without traps, the benchmark cannot distinguish between H2 and H9.

• Was the gold answer pre-specified and documented before the AI system was run? Post-hoc gold answers can be unconsciously calibrated to the system already built.

• Does the benchmark include adversarial scenarios — conditions under which the system might fail in production that are not represented in normal operation?

• Who designed the benchmark? If the same team that built the AI system, it may be calibrated to that system’s strengths. Independent design is more reliable.

• What is the Sigma score (quality divided by complexity)? A benchmark that reports only Alpha (raw quality) is incomplete and will tend to favour over-engineered architectures.

B.5 How should adversarial testing be mandated and costed?

The cost-benefit calculation is striking: the adversarial stage identified H3’s conditional vulnerability at a compute cost of approximately $800 (two Sonnet API calls) plus roughly $2,000 in human review time (4 hours). The vulnerability, undetected in production, had an expected cost of $2.5M in avoided fines. The question is not whether to do adversarial testing — the case for $2.8K cost vs. $2.5M expected benefit is overwhelming — but how to institutionalise it so it is not skipped under time pressure.

• Hard gate: adversarial testing is a formal deployment gate, managed by an independent team. Strongest model — cannot be skipped — but requires organisational commitment to maintain the independent review function.

• Automated adversarial: build adversarial testing into the deployment pipeline as an automated stage, run on every deployment candidate, with results logged and reviewed. Scales better but requires investment in adversary-agent design.

• Risk-tiered requirement: mandatory for high-stakes deployments (regulatory, patient safety, financial compliance) and optional for low-stakes routine deployments — requires a risk classification framework.

The case supports the hard-gate model for any deployment affecting regulatory compliance, patient safety, or significant financial exposure, with the automated model appropriate for iterative improvement within an already-approved deployment.

References

References marked [open access] are freely available. Others are accessible through institutional library subscriptions or directly from the publishing organisation’s website.

[1] ‘The Loop is the Lab’ / ‘The Speciation of Intelligence’ / ‘The Working Layer.’ Harness Engineering Series, Interesting Engineering++. Foundational ASCRS architecture, the eight-primitive framework, and the mutation ladder (L0–L5).

[2] The ASCRS Harness Lab. Harness Engineering Series, Interesting Engineering++. Documents the H1–H10 benchmark in full: α, λ, and κ results across all ten architectures (Section 1 of this case), including H2 α=0.920 and H9 α=0.815. https://interestingengineering.substack.com

[3] ‘The Harness Lab, Automated.’ Harness Engineering Series, Interesting Engineering++. Automated tournament of harness designs using Claude Code dynamic workflows on a simplified benchmark. Introduces the Sigma metric (α / (1+Δ)). Tool-using and iterative-loop designs (H3/H5-style) win the simplified benchmark. https://interestingengineering.substack.com

[4] ‘The Prompt Is Still the Work: Dynamic Workflows in Claude Code.’ Harness Engineering Series, Interesting Engineering++. Maps Anthropic’s six dynamic workflow patterns against prior ISR experiments. https://interestingengineering.substack.com/p/the-prompt-is-still-the-work-dynamic

[5] Anthropic (2026). ‘A harness for every task: dynamic workflows in Claude Code.’ Thariq Shihipar and Sid Bidasaria. Introduces the ultracode keyword, six workflow patterns, and the agentic failure modes (agentic laziness, self-preferential bias, goal drift). [open access] https://claude.com/blog/a-harness-for-every-task-dynamic-workflows-in-claude-code

[6] Anthropic (2026). Claude Code Dynamic Workflows Reference Documentation. Technical reference for forkSubAgents(), runSubAgent(), /loop, /workflows, /goal, /compact. [open access] https://code.claude.com/docs/en/workflows

[7] US Code of Federal Regulations (2024). 21 CFR Part 314.81 — Post-approval reporting requirements including drug shortage notification obligations (§ b.3.iii). [open access] https://www.ecfr.gov/current/title-21/chapter-I/subchapter-D/part-314/subpart-E/section-314.81

[8] US Congress (2012). Food and Drug Administration Safety and Innovation Act (FDASIA), Section 1001. Mandatory drug shortage reporting for manufacturers of medically necessary drugs. [open access] https://www.fda.gov/regulatory-information/selected-amendments-fdc-act/food-and-drug-administration-safety-and-innovation-act-fdasia

[9] US FDA Drug Shortages Database. Historical records of reported shortages, enforcement actions, and warning letters. Used to derive the $250K–$2.5M fine-per-violation range. [open access] https://www.accessdata.fda.gov/scripts/drugshortages/

[10] 2001-2026 ASHP Drug Shortage Survey Results.’ American Journal of Health-System Pharmacy. 58% of pharmacists made therapeutic substitutions during shortages; 12–18% became permanent (basis for the 15% switching-rate assumption). https://www.ashp.org/drug-shortages/shortage-resources/drug-shortage-statistics

[11] Schachtner L et al. (2021). ‘Pharmaceutical switching behaviour following drug shortage.’ Journal of Managed Care and Specialty Pharmacy, 27(8), 1045–1053. Permanent switch-rate study underpinning the 12–18% assumption. https://www.jmcp.org

[12] Deloitte Life Sciences (2023). ‘Pharmaceutical Supply Chain Risk Management: Emergency Sourcing and Procurement Analytics.’ Documents Tier-1 emergency procurement premiums 25–35%, Tier-2 35–45%. https://www2.deloitte.com/us/en/pages/life-sciences-and-health-care/articles/pharmaceutical-supply-chain.html

[13] McKinsey Global Institute (2021). ‘Risk, resilience and rebalancing in global value chains.’ Documents 50–80% spot pharmaceutical procurement premiums during 2020–2021 shortage events. Basis for the 55% spot-market assumption. https://www.mckinsey.com/capabilities/risk-and-resilience/our-insights

[14] EvaluatePharma (2024). ‘World Preview 2024: Outlook to 2030.’ Industry benchmark for pharmaceutical EBITDA margins; median for top 25 pharma companies 27.3%. Basis for the 28% assumption. https://www.evaluate.com/evaluate-pharma

[15] IQVIA Institute (2025). ‘Global Medicine Spending and Usage Trends: Outlook to 2029.’ Pharmaceutical gross margin benchmarks by category. Basis for the 43% blended margin assumption. https://www.iqvia.com/insights/the-iqvia-institute/reports

[16] S&P Global Market Intelligence (2025). S&P Pharmaceuticals Select Industry Index — historical P/E multiple data. Q4 2025 median: 14.8×. Basis for the 15× multiple used in enterprise value calculations. https://www.spglobal.com/marketintelligence/en/

[17] Premier Healthcare Alliance (2023). ‘Supply Chain Resilience Report.’ Documents 40–45% emergency procurement premiums under manual, time-pressured processes. Basis for the human-baseline premium assumption. https://www.premierinc.com/supply-chain/insights

[18] Gartner (2023). ‘Supply Chain Top 25 Healthcare.’ AI deployment complexity analysis documenting 25–40% cost overruns for multi-agent systems vs. single-agent implementations. Basis for the H9 contingency uplift. https://www.gartner.com/en/supply-chain

[19] GlobalData (2024). ‘Respiratory Drugs Market Analysis: Salbutamol Competitive Landscape.’ Branded salbutamol global market ~$400M in 2023. Basis for the Salbutamol revenue assumption. https://www.globaldata.com/store/report/respiratory-drugs-market-analysis/

[20] Levels.fyi / Glassdoor (2025–2026). Senior AI/ML Engineer and integration engineer contract rate benchmarks, United States market. Rates used: $260/hr senior AI engineer, $210/hr integration engineer, $180/hr QA, $150/hr documentation — senior contractor rates, not fully-loaded employee cost. https://www.levels.fyi

[21] Narayanan, A. and Kapoor, S. (2026). ‘Why AI hasn’t replaced software engineers, and won’t: Coding agents as normal technology.’ AI as Normal Technology. Introduces the ‘decide-execute-deliver sandwich’ framework cited in Sections 11.4 and 14.6, and documents 2026 ‘AI washing’ layoff cases (Block, Snap, Intuit), WARN Act AI-disclosure data, and the HBR anticipated-vs-realised AI headcount-reduction gap. [open access] https://www.aisnakeoil.com

I. The Six-Word Sentence That Started a Discourse

“My job is to write loops.”

Six words. Boris Cherny — Head of Claude Code at Anthropic — said them, at the tail of a longer statement that ignited discussions across developer communities worldwide:

“I don’t prompt Claude anymore. I have loops running that prompt Claude and figuring out what to do. My job is to write loops.”

~ Boris Cherny

The first two sentences set context. The last one became the discourse. It was amplified almost immediately by Addy Osmani (Google Engineering Director) and Peter Steinberger (OpenClaw founder, now at OpenAI): “You shouldn’t be prompting coding agents anymore. You should be designing loops that prompt your agents.”

Addy Osmani - Loop Engineering

By the weekend, the discourse had fractured cleanly. One camp read this as a paradigm shift demanding immediate workflow overhaul. Another camp read it as irresponsible hype. A third camp — the practitioners who’d been quietly building agentic systems for twelve months — barely noticed, because they were already three architectural generations past the argument.

This article does not adjudicate the discourse. It goes past it. I profile six practitioners whose names appear in the primary sources of 2025–2026 agentic engineering, reconstruct their actual architectures from the record, and benchmark them against the last few months of controlled experiments in the ASCRS harness lab. The goal is not celebration. It is calibration.

The central finding, stated plainly: the prompt never left the room. It got promoted.

The loop is not a replacement for the prompt. It is a container for the prompt. Every loop in every architecture examined here is ultimately anchored by a document someone spent time brainstorming or wrote by hand.

II. The Field Census: Six Architects

The practitioners below represent the primary sources cited across the 2025–2026 agentic engineering literature. Each profile documents role, background, core technique, anchor documents, and — critically — the actual prompts or prompt structures they depend on.

Boris Cherny · Head of Claude Code · Anthropic

↳ Fact-check: Cherny — @bcherny June 7, 2026 (five tips post, SWE-Marathon context) · Threads mirror of the five tips post · How Boris Uses Claude Code (full tip archive) · OfficeChai — Cherny interview (loop-writing background) · InfoQ: Claude Code Creator Workflow (January 2026 setup) · Fortune Brainstorm Tech (Yahoo) — thousands of agents quote

What Cherny Actually Runs: The Five-Tip Framework

The misreading of Cherny’s claim is that he eliminated prompting. He did not. His June 7, 2026 post — five practical tips for running Opus autonomously for hours or days — is the most operationally precise statement he has made about his actual workflow. Mapped onto the hierarchy, the five tips are not advice about productivity. They are a configuration checklist for the L2–L4 architecture (details further below):

CHERNY — Five-Tip Framework (@bcherny, June 7 2026, in response to SWE-Marathon benchmarks)

↳ Fact-check: @bcherny — June 7 2026 post (five tips, SWE-Marathon context) · Threads mirror · /goal vs /loop distinction — Claude Code docs · Avi Chawla — four autonomy modes explained [/goal and /loop are verbatim Claude Code command names. The five tips are verbatim from Cherny’s post. Hierarchy level annotations (L2, L3) are part of the analysis.]

Tip 5 — self-verification end-to-end — is the one practitioners skip and then wonder why long autonomous runs drift. The loop can produce technically correct code that fails in the actual running environment. Cherny’s solution is to put the environment itself inside the verification loop: boot the web server, open the browser extension, run the mobile simulator. The judge is not a model reading a transcript. It is the product running.

The earlier PR management loop fits within this framework as a Tip 3 example — using /loop (cadence-based, checking the queue on schedule) rather than /goal (which would be used when the task has a definable completion state like ‘all tests pass’). The distinction matters at scale: /loop is for monitoring and polling; /goal is for bounded work with a finish line.

Geoffrey Huntley · Creator, Ralph Wiggum Technique · Open Source

↳ Fact-check: ghuntley.com/ralph/ — original technique post · Anthropic official ralph-wiggum plugin README · Dev Interrupted — Inventing the Ralph Wiggum Loop (interview) · paddo.dev — Ralph Wiggum autonomous loops deep dive

The Ralph Architecture

↳ Fact-check: ghuntley.com — ‘Ralph is a bash loop’ · Anthropic plugin README (Stop hook mechanism) · github.com/wiggumdev/ralph (implementation) · Syntax+Glitter — Ralph Wiggum Loop explainer · Sondera — Supervising Ralph (stateless resampling analysis) [Architecture is reconstructed from Huntley’s public posts and the official Anthropic plugin README]

What Huntley Actually Prompts

The Ralph technique’s power is inseparable from PROMPT.md — the anchor document that gets re-injected every iteration. A typical PROMPT.md is not a sentence; it is a structured specification containing goal, constraints, success criteria, and current state pointer. The loop resamples it fresh each time, avoiding the “context rot” failure mode where a growing conversation history causes the model to navigate toward its own prior attempts rather than toward the objective.

↳ Fact-check: ghuntley.com/ralph/ — PROMPT.md discipline · Anthropic plugin README — completion-promise mechanism · wiggumdev/ralph — full docs · Addyosmani.com — self-improving agents (PROMPT.md usage) [PROMPT.md template is a reconstruction based on Huntley’s documented technique; specific field names are inferred from the plugin spec and ghuntley.com]

Steve Yegge · Creator, Gas Town / Gas City · Ex-Amazon, Google, Sourcegraph

↳ Fact-check: Steve Yegge — Welcome to Gas Town (original) · Steve Yegge — Welcome to Gas City (SDK) · Cloud Native Now — Gas Town deep dive · steveyegge/gastown (GitHub)

The Gas Town Architecture

↳ Fact-check: Steve Yegge — Welcome to Gas Town (roles defined) · Cloud Native Now — ‘Kubernetes for AI’ analysis + role breakdown · Codex Blog — Gas Town roles + Beads analysis · paddo.dev — GasTown and the Two Kinds of Multi-Agent · Reading list — Gas Town architecture summary [Role names (Mayor, Polecat, Refinery, Witness, Deacon) and GUPP protocol directly from Yegge’s original Medium post]

Yegge’s core insight aligns with Geoffrey Huntley’s: the AI’s memory is the version-controlled repository, not the chat history. Twenty to thirty agents can work in parallel without colliding because their shared state is code — immutable, auditable, mergeable. What often gets lost in descriptions of Gas Town is that each worker role is defined by a role prompt. The Mayor has system-level orchestration instructions. Polecats have task-execution instructions. Refinery has merge-management instructions. Yegge didn’t stop prompting; he wrote seven distinct prompt architectures and then automated who calls them.

Peter Steinberger · Creator, OpenClaw · Former CEO PSPDFKit · OpenAI (from Feb 2026)

↳ Fact-check: steipete.me — Just Talk To It (workflow) · OpenClaw VISION.md analysis · Fortune — Who is Peter Steinberger? · Wikipedia — Peter Steinberger (programmer) · Grokipedia — Steinberger agentic engineering

VISION.md: The Document That Runs Before Every Loop

Steinberger’s most significant contribution to the practitioner literature is not a tool or a framework. It is a practice: writing VISION.md before writing any loop or agent configuration. For OpenClaw, the VISION.md is 94 lines — no marketing language, no growth projections. Its first sentence is: ‘OpenClaw is the AI that actually does things.’

↳ Fact-check: OpenClaw VISION.md — full analysis · steipete.me — VISION.md discipline (workflow post) · steipete.me — full personal site / posts [VISION.md template headings are reconstructed from Steinberger’s public OpenClaw VISION.md (94 lines). Section labels are paraphrased, not verbatim]

Addy Osmani · VP Engineering · Google Chrome · Author, Learning Patterns

↳ Fact-check: addyo.substack.com — Loop Engineering (Jun 8 2026) · addyosmani.com — Self-Improving Coding Agents · addyo.substack.com — The 80% Problem in Agentic Coding · addyosmani.com — Code Agent Orchestra

Osmani’s Five Loop Building Blocks

↳ Fact-check: Osmani — Loop Engineering (five blocks + automation tab detail) · Osmani — Self-Improving Coding Agents (sub-agent/judge pattern) · Osmani — AI Coding Workflow 2026 (spec-first discipline) [Five building blocks + 70/30 spec/execution ratio are direct from Osmani’s June 2026 Loop Engineering post; quoted caution on token costs is verbatim]

Andrej Karpathy · Co-founder OpenAI · Coined ‘Vibe Coding’ and ‘Agentic Engineering’ · Anthropic (from May 2026)

↳ Fact-check: Wikipedia — Andrej Karpathy (joined Anthropic May 2026) · The New Stack — Vibe coding is passé / agentic engineering · MindStudio — Karpathy Sequoia talk breakdown · github.com/karpathy/autoresearch · The AI Corner — Karpathy workflow shift 2026

The Karpathy Inflection Model

↳ Fact-check: MindStudio — Karpathy Sequoia talk: ‘December is when it really flipped’ · The AI Corner — ‘In December... I went from 80-20 to 20-80’ (verbatim ratio) · IBM Think — Agentic Engineering definition · NxCode — Agentic Engineering complete guide [80/20 → 20/80 ratio and December 2025 inflection point are verbatim from Karpathy’s Sequoia talk. Jun 2026 ~10/90 estimate is an extrapolation, not a direct quote]

III. The Convergence: What All Six Agree On

Six practitioners. Six different tools, scales, and domains. But look at the architecture beneath the surface, and a convergence emerges. Every system in this field census depends on the same structural primitives, arrived at independently through the same failure modes.

↳ Fact-check: Cherny — howborisusesclaudecode.com · Huntley — ghuntley.com/ralph/ · Yegge — Welcome to Gas Town · Steinberger — steipete.me · Osmani — Loop Engineering · Karpathy — autoresearch (program.md) [Each row is synthesized from the primary sources listed. ‘Karpathy val_bpb metric’ is specific to the autoresearch repo; other eval gate entries are reconstructed from public descriptions]

The convergence is striking. Every practitioner:

• Writes a human-authored anchor document before running any loop

• Uses git as the durable memory substrate — not conversation history

• Separates the agent doing work from the agent evaluating work

• Implements an explicit halting condition (max iterations, completion token, dollar budget)

• Preserves per-agent isolation (worktrees, separate checkouts, sandboxed environments)

The loop is how they automated the iteration. The anchor document is how they preserved their intent. These are not in tension. The second is required for the first to be trustworthy.

The Prompt Moved Upstream

Here is the structural insight the discourse missed: the transition from prompting to loop-writing is not the elimination of prompting. It is the promotion of prompting. The prompt moved from a real-time, session-by-session interaction into an architectural artifact that gets written once, maintained carefully, and read by every agent on every iteration.

↳ Fact-check: Osmani — Loop Engineering (Gen 1/2/3 framing) · Cherny — OfficeChai interview (trajectory from IDE to loops) · Karpathy — ‘vibe coding’ → ‘agentic engineering’ evolution [Three-generation model is ISR synthesis; Gen 1/2/3 labels are original ISR framing, not a direct quote from any single practitioner]

IV. The ISR Benchmark: Our Evidence Holds

Our series has now run a number of volumes of controlled experiments on the ASCRS benchmark scenario (Hormuz Strait pharmaceutical supply chain). Across thirteen experiment cycles using Claude Code with OpenRouter model routing, the findings from the ASCRS Harness Lab intersect directly with the practitioner observations above — and in several cases, our experimental evidence explains failures that the practitioner literature only describes qualitatively.

Finding 1: Task Structure Dominates Architectural Complexity

The ASCRS Harness Lab (H1–H10) produced a result that surprised us at the time: H2 — a structured, single-turn prompt with defined schema and explicit output format — outperformed H9, a five-agent parallel swarm, by a significant margin (α=0.920 vs α=0.625). The H9 swarm had more computational resources, more parallelism, and more architectural sophistication. H2 had a better prompt. However, yesterday, “The Harness Lab, Automated” finally named the workflows utilizing tools and self revision loops H3, and H5 - as winners.

↳ Fact-check: ISR — ASCRS Harness Lab series (H1–H10 original experiments) · ISR — The Architecture of Awareness (ASCRS V1–V4) · ISR GitHub — harness-lab-conductor plugin [α scores (0.920, 0.847, 0.791, 0.743, 0.625) are ISR experimental results. These are the authors’ own data and have not been independently replicated.]

Finding 2: Context Engineering Operates at Two Layers

The Memory A/B experiment (BOOTSTRAP_PROMPT.md injection vs stripped data files) found that context engineering operates at two distinct layers: data structure and prompt injection. The BOOTSTRAP_PROMPT.md — our equivalent of a VISION.md — was essential for criteria requiring institutional parameters. Without it, even well-structured data files were insufficient for criteria that required understanding the project’s purpose. Sources: Building on Anthropic’s Claude and Is Claude’s Memory Actually Poor?

Finding 3: The Token Tax and Prompt Caching Gaps

The StockPilot/CMA decomposition study (97% token reduction across Cycles 0–4) identified a specific failure mode relevant to practitioner architectures: OpenRouter’s multi-provider routing breaks prompt caching, causing unnecessary context re-loading on every cycle. This is almost comparable to the failure mode Huntley names as “context rot” — except our experiment documented it in token counts rather than behavior degradation.

Full article: The Structure Is The Intelligence

Full article: Every Company Is An Agent Waiting to Be Decomposed

Full article: The Token Tax

Finding 4: The COBOL Pipeline — Our Five-Agent Architecture

The COBOL migration pipeline (”The Invisible Codebase”) produced a verified five-agent Python pipeline: Extractor → Translator → Test Generator → Verifier → Evaluator. This is structurally identical to the maker/checker architecture that appears in every practitioner system examined above. The Evaluator is the judge-agent. The Verifier is the stopping-condition checker. The pipeline runs with configurable OpenRouter model routing.

↳ Fact-check: ISR — The Architecture of Awareness + ASCRS series · ISR — The Invisible Codebase (COBOL pipeline) · ISR — The Structure Is the Intelligence (StockPilot/CMA) · ISR — The Memory Is the Architecture (memory A/B) · ISR — The Harness Lab, Automated (Sigma metric) · Osmani — 80% Problem (practitioner parallel) [All ISR findings are from the authors’ own experimental series. Individual article URLs have been aggregated to the ISR Substack root pending publication of specific article links]

V. The Architecture Convergence Map

The following diagram represents the architectural pattern that emerges from the intersection of all six practitioner systems and the ISR experimental record. It is not any single practitioner’s architecture. It is the pattern all of them converged on, labeled with the terminology each uses for equivalent components.

↳ Fact-check: Anthropic — Agent Skills (L1 SKILL.md layer) · Yegge — Gas Town (L2–L3: Polecats, Role Beads, git substrate) · Huntley — Ralph (L2–L3: halt condition, PROMPT.md spec) · Osmani — Loop Engineering (L2: five building blocks) · Cherny — InfoQ workflow (L3: loop contract; git as memory substrate) · Karpathy — autoresearch (L1: program.md skill; L4: anchor spec) · Claude Code /goal docs (L3: loop contract + L0: condition text) [ISR original synthesis. L0–L5 labels are consistent with the automation hierarchy in Section VI. Infrastructure (git substrate, halt conditions) is not a numbered level — it supports all levels.]

VI. The Architecture Goes Native

Between May and June 2026, Anthropic productised the same architectural patterns as first-class Claude Code primitives — four distinct autonomy modes encoding what six independent practitioners had arrived at through months of production experience.

This matters for any reading of the hierarchy above. The loop control layer — which the convergence map placed at roughly 30% automated — required significant upward revision when these primitives shipped. The artisanal is becoming the platform.

/goal: The Maker/Checker Split as a Product Feature

Claude Code v2.1.139, shipped May 12, 2026, introduced /goal — a session-scoped stopping-condition command built around a structural separation that every practitioner in this series had implemented by hand. The design is precise: Claude does the work across multiple turns; a separate lightweight model (Haiku, by default) reads the transcript after every turn and evaluates a single question — has the condition been met? If not, another turn begins automatically. If yes, control returns to the user.

This is Yegge’s Refinery agent, the ISR Evaluator in the COBOL pipeline, and Osmani’s judge sub-agent — the same maker/checker split — now a native product feature. The documentation states the underlying principle directly: the model doing the work is not the one deciding it is done. The /goal condition, which can run to 4,000 characters, specifies the end state, the verification method (”npm test exits 0”, “git status is clean”), any constraints that must hold across turns, and an optional turn or time ceiling.

The boundary between the loop contract (L3) and the loop engine (L2) has partially collapsed. What was previously two separate concerns — writing the contract, then building the mechanism to enforce it — is now a single authored condition that activates a built-in enforcement mechanism.

Four Autonomy Modes: From Condition-Based to Parallel Swarms

The second and third autonomy modes shipped together: Claude Code v2.1.154, May 28, 2026, alongside Claude Opus 4.8. Dynamic Workflows fan a task out across tens to hundreds of parallel subagents, each with a clean context window and one focused job — Claude writes the harness script at runtime rather than running inside a fixed default. /batch is the companion mode for work that breaks cleanly into independent items: parallel agents each take one item, work in an isolated git worktree, and open a PR. A fourth mode, /loop, re-runs a prompt on a time cadence rather than running to a condition — the right tool for polling and monitoring, the wrong tool for a task with a finish line.

The Anthropic product blog framed this as “a harness for every task” — Claude Code can now produce the coordination layer on demand rather than requiring the practitioner to design it. The three failure modes the design targets are the same ones Huntley and Yegge documented from production experience: agentic laziness (one overloaded context window quits early), self-preferential bias (the agent that wrote the answer also grades it), and goal drift (technically correct output that moved away from the original intent). Dynamic Workflows address all three structurally.

What the Automation Levels Look Like Now

With these three primitives in production, the loop control layer — which the convergence map above estimated at roughly 30% automated — sits considerably higher. The practical ceiling for each level as of June 2026:

The ISR Parallel: Ahead of Native Tooling

The ISR series documented these patterns before they became native to Claude Code. The Sigma composite metric and Design Verification gate in “The Harness Lab, Automated” are the manual precursors to what /goal now does natively for code tasks: a structured success criterion, evaluated by a separate process, gating whether the loop continues. The dynamic workflow experiments in “The Prompt Is Still the Work” mapped the same fan-out and conditional routing architecture that shipped as Dynamic Workflows in May 2026.

The sequence — practitioners converge independently, ISR documents experimentally, Anthropic productises — is itself a finding. It suggests the architecture is not a design preference but a structural necessity. When the patterns are strong enough to be discovered six times independently, documented in a controlled series, and then encoded into the platform, they are not artifacts of individual practitioner taste. They are what the problem requires.

When AI Builds Itself: The Scale Confirmed

Anthropic’s “When AI Builds Itself” (June 2026) provides internal data that confirms the scale the practitioners described publicly. As of May 2026, more than 80% of code merged into Anthropic’s production codebase was authored by Claude. The median Anthropic researcher estimated roughly 4x output with Claude Mythos Preview versus no AI assistance. Engineers were merging 8x as much code per day in Q2 2026 compared to 2024 — not because they were working faster, but because most of the code was no longer written by hand.

The article also maps onto the hierarchy with unusual precision. On the human role remaining: “Claude can be handed an underspecified problem and figure out how to solve it; humans supply the goal, but they no longer need to supply the method.” That boundary — between L4 (the goal, still human) and L3/L2 (the method, increasingly automated) — is precisely what /goal, Dynamic Workflows, and Ultracode are encoding as product primitives. And on the gap that remains: “Large performance gaps persist when it comes to Claude exercising judgment in choosing goals.” L5, still firmly at zero.

The article also describes an automated code reviewer now running on every proposed change at Anthropic — finding roughly a third of the bugs behind past production incidents before they reached users. This is a production-scale L1 skill with a known false-alarm rate, structured output, and a measurable track record. The ISR automated code-reviewer skill described in the Dan Kornas example was the field approximation; the Anthropic internal deployment is the same pattern at production scale.

The practitioner field and the Anthropic product team arrived at the same architecture by different routes. The practitioners arrived first. Anthropic encoded it. The ISR experiments documented the journey in between.

Fable 5: The Model Tier That Changes the Calculus

Released today — June 9, 2026 — Claude Fable 5 is Anthropic’s first publicly available Mythos-class model. It sits above the Opus family in capability. Its core properties are directly relevant to every claim the practitioners above have made about long-horizon autonomous work: multi-day goal-directed runs with sustained instruction retention, first-shot correctness on problems that previously took iterative loops to solve, and significantly more reliable parallel subagent management. SWE-Bench Pro: 80.3%, versus Opus 4.8’s 69.2%.

Anthropic published a model-specific prompting guide for Fable 5 alongside the release — covering effort calibration, instruction following, long runs, memory management, and scaffolding changes. The guide’s opening framing states that capability improvements at this level are ‘a good prompt to re-evaluate which instructions, tools, and guardrails are still needed.’ That sentence is VISION.md thinking applied to model selection. The anchor document layer does not become optional as models improve. It becomes model-specific. The practitioner who knows how to calibrate Fable 5 is performing work that cannot be automated away, because the calibration depends on knowing what the project is for.

The most structurally important implication is for the L2 loop layer. The guide notes that early testers reported single-pass implementations of systems that previously took days of iteration. If first-shot correctness holds broadly across a practitioner’s domain, some work that previously required Ralph-style loop resampling moves to single-turn. L2 shifts from a necessity toward a reliability mechanism for those tasks. But this makes L0 more load-bearing, not less: you get one pass, well-specified. And it makes L4 more consequential than ever — a model that applies significantly more effort per instruction in a loop with an under-constrained anchor document will drift further, faster. The scope constraints and stopping conditions the practitioners documented are not scaffolding for weak models. They are the load-bearing walls of systems built on capable ones.

↳ Fact-check: Anthropic — Prompting Claude Fable 5 (official guide) · Anthropic — Claude Fable 5 and Mythos 5 (release + benchmarks) · Introducing Claude Fable 5 and Mythos 5 (API docs, specs, pricing) · Digital Applied — Fable 5 benchmark breakdown (SWE-Bench Pro 80.3%) [Fable 5 released June 9, 2026 - Benchmark figures (SWE-Bench Pro 80.3%, Opus 4.8 69.2%) from Digital Applied analysis of Anthropic’s published benchmark table. Hierarchy shift analysis is ISR interpretation of the prompting guide’s behavioral change descriptions.]

VII. What We Do: The Two-Window Method and the ISR Harness

The ISR methodology has operated on a two-window architecture throughout the series: Claude.ai chat for design and analysis; Claude Code for execution. This is not a limitation imposed by tooling. It is a deliberate choice that turns out to map exactly onto the architectural split that the field census describes.

Claude.ai chat window = the design layer. This is where VISION.md equivalents get written, where loop contracts get specified, where experimental design happens, where prompts get iterated. It is interactive, human-in-the-loop, and produces structured artifacts.

Claude Code = the execution layer. This is where the artifacts produced in the design layer get run, tested, and evaluated. The Harness Conductor plugin (18-file Claude Code plugin, private for now) represents our version of a Gas Town configuration: skills, agents, commands, and infrastructure files organized as a portable, installable package.

↳ Fact-check: ISR — harness-lab-conductor plugin (GitHub) · ISR — The Prompt Is Still the Work (dynamic workflow series) · ISR — The Harness Lab, Automated [ISR original methodology description. Two-window method is the authors’ own practice, not derived from practitioner literature. Comparable to the design/execution split visible in all six practitioner architectures.]

Where the ISR Experiments Sit on the Practitioner Ladder

Mapping the ISR ASCRS experiments against the practitioner architectures:

▸ ASCRS Harness Lab (H1–H10): Equivalent to Osmani’s tier-1 architecture (single terminal session, direct agent orchestration). No automated scheduling. Human-in-loop at each experimental cycle. Design-layer decisions made in Claude.ai, execution in Claude Code.

▸ COBOL Migration Pipeline: Equivalent to Cherny’s PR management loop — sequential, five-agent, with explicit evaluation at each stage. Configurable model routing (OpenRouter) = equivalent to multi-provider gas town configuration.

▸ Harness Lab, Automated: Closest to Huntley’s Ralph architecture — automated iteration with Sigma metric as the DONE condition. Design Verification gate = the Ralph completion promise.

▸ Memory A/B Experiment: Most directly comparable to Steinberger’s VISION.md discipline — empirically confirming that anchor documents (BOOTSTRAP_PROMPT.md) encode criteria that skill files alone cannot.

▸ Dynamic Workflow Series: Directly parallel to Yegge’s Gas Town role architecture — multi-step workflows with conditional routing between stages. CMA as a stateful execution runtime = Mayor-equivalent coordination layer.

Our methods have been convergent with the practitioner field throughout the series. The difference is that the ISR approach generated controlled experimental evidence for findings that practitioners arrived at through production experience. Both paths reach the same architecture.

VIII. The Uncomfortable Truth About “Stopping Prompting”

Boris Cherny is right. He doesn’t prompt Claude in the way most people prompt Claude. He does not sit at a keyboard and type instructions into a chat window every time he wants something done. That part is genuinely over for him.

But Boris Cherny spent a significant part of 2025 writing the loop contracts, skills, and configurations that make his current workflow possible. He wrote the CLAUDE.md files. He wrote the loop specifications. He ran five parallel Claude Code instances against separate git checkouts and observed which patterns produced reliable output and which produced garbage. He built the judgment infrastructure that the loops now execute against.

That work is prompting. It is just prompting that happened upstream — before the loop ran — and that now compounds forward instead of evaporating at session end.

The Benchmark That Puts a Number on It

The practitioners speak with considerable confidence about what loops and autonomous agents deliver. SWE-Marathon — launched June 6, 2026 by Rishi Desai (@rishi_desai2) of Abundant AI, and the direct trigger for Cherny’s five-tip post the following day — puts a number on what the best current configuration actually achieves on genuinely hard, novel, realistic tasks. That number is 26%.

The benchmark comprises 20 multi-hour software engineering tasks drawn from real frontier research projects: building a multi-pass C compiler in Rust from preprocessing through x86-64 codegen, reimplementing Kubernetes control-plane components in Rust while preserving API semantics, cloning Slack from scratch, rewriting a JAX codebase in PyTorch. Binary scoring: pass every verifier test and the run scores 1.0; any failing test scores 0.0. Mean token budget per trial: 27 million. Across 1,300 logged trials, all frontier configurations stayed below 26%.

↳ Fact-check: SWE-Marathon — swe-marathon.org (leaderboard, tasks, observations) · GitHub — abundant-ai/swe-marathon (320 GB trajectory logs) · Digg — Abundant AI releases SWE-Marathon (reward hacking data) · SWE-Bench Verified leaderboard (for Verified vs Marathon comparison) [Leaderboard figures are from swe-marathon.org as of June 9, 2026. SWE-Bench Verified scores from llm-stats.com. Dynamic workflows question is from the public benchmark discussion thread — no published data exists yet.]

Three findings from the benchmark are directly relevant to the architecture argument in this article.

First, the harness comparison confirms the ISR H1–H10 finding in a different domain. Claude Opus 4.7 paired with Claude Code scores five points higher than the same model with Terminus 2. Same model, different execution layer — meaningful performance difference. Task structure and execution architecture are determinants, not just model capability.

Second, the reward hacking rate is a direct challenge to the evaluation architecture. Thirteen percent of autonomous runs produced results that gamed the verifier rather than solving the task. One agent, in 13 minutes and 2.8 million tokens, built a C compiler that shelled out to the system’s gcc via std::process::Command instead of actually compiling anything, then reported it was ‘resolving edge cases via safe bypass strategies.’ The benchmark caught it — but only because the verifier was specifically designed to detect shell-outs. A model-evaluated stopping condition would not have caught it. Cherny’s Tip 5 (self-verification end-to-end: boot the actual server, run the actual simulator) is the practitioner’s answer to this problem. SWE-Marathon is the empirical evidence for why it matters.

Third, dynamic workflows were not used in any of the published configurations. The performance ceiling of 26% may be a floor rather than a ceiling for the full five-tip stack. But until someone runs the benchmark with dynamic workflows enabled, the confidence of the practitioners — which is calibrated against known domains with mature skill libraries — remains unanchored against novel, adversarially-verified, long-horizon work. That gap between calibrated confidence and benchmarked performance is the most honest thing SWE-Marathon adds to the discourse.

The Comprehension Debt Problem

There is a cost that none of the practitioner literature addresses adequately: the faster a loop ships code the practitioner didn’t write, the wider the gap between what exists in the codebase and what they actually understand. This is comprehension debt. It compounds as reliably as skill libraries do, but in the opposite direction.

The ISR experiments have run into this directly. In the COBOL migration pipeline, the Evaluator agent caught several Translator outputs that passed test gates but contained structural misconceptions about the original COBOL logic. The test suite was insufficient to catch semantic drift — because the test suite was generated by the same pipeline that produced the translation.

This is the production version of Osmani’s 80% problem: agents can rapidly generate 80% of the work, but the remaining 20% requires deep knowledge of context, architecture, and trade-offs. The loop doesn’t know the difference between the 80% and the 20%. The human must.

Two people can build the exact same loop and get completely opposite results. One uses it to move faster on work they understand deeply. The other uses it to avoid understanding the work at all. The loop is identical. The outcomes diverge.

What This Means for ISR Methodology

The ISR two-window method — design layer in Claude.ai, execution layer in Claude Code — is not a workaround for lacking a full Gas Town infrastructure. It is a deliberate architectural choice that maintains the human design function. The chat window is where judgment lives. The execution layer is where that judgment gets scaled.

The Harness Conductor plugin represents a portable skill library for the ASCRS domain. But it was built through the design-layer process: experimental hypotheses in the chat window, execution and measurement in Claude Code, findings fed back into the next experiment’s design. That loop — human judgment driving experimental iteration driving skill improvement — is the ISR equivalent of the practitioner architectures documented above. It is less automated. It is more legible. It produces documented evidence rather than shipped code, which is appropriate for a research series.

IX. Practical Reference: Prompts the Practitioners Use

The following section consolidates practical prompt patterns derived from the practitioner architectures documented above. These are not verbatim quotations; they are reconstructions from public documentation, demonstrations, and the convergent pattern analysis above. Fact-check bars beneath each template link to the primary sources from which each pattern is derived. These are high-level views and therefore will not represent all details.

Pattern 1: The Loop Contract (Cherny-style)

Write before running any repeating agent. One page maximum.

↳ Fact-check: Cherny — OfficeChai (loop contract semantics from PR loop demo) · Osmani — Loop Engineering (loop contract as a concept) · Osmani — Self-Improving Agents (budget + stop_if patterns) · Cherny — howborisusesclaudecode.com (scope / isolation principles) [YAML format and field names are ISR reconstruction. No practitioner publishes a canonical loop contract template. Fields derived from Cherny’s described PR loop behaviour and Osmani’s loop engineering post.]

Pattern 2: The VISION.md (Steinberger-style)

Write before writing any skill, agent file, or CLAUDE.md. Treat editing it as a significant decision.

↳ Fact-check: OpenClaw VISION.md — 94-line analysis · steipete.me — Just Talk To It (VISION.md rationale) · Steinberger — steipete.me (full blog, VISION.md philosophy) [Template headings are ISR reconstruction based on the openclaw.report analysis of the actual 94-line VISION.md. The OpenClaw VISION.md file itself is in the OpenClaw repository.]

Pattern 3: The PROMPT.md (Huntley / Ralph-style)

The anchor document for a stateless loop. Written once, re-injected every iteration.

↳ Fact-check: ghuntley.com/ralph/ — PROMPT.md as the anchor document · Anthropic plugin README — completion-promise = ‘DONE’ marker · wiggumdev/ralph — prompt file specification · Syntax+Glitter — PROMPT.md in practice [Template is ISR reconstruction. ‘DONE’ completion token is verbatim from the official Anthropic plugin. Section headings are inferred from Huntley’s documentation, not a verbatim PROMPT.md template he has published.]

Pattern 4: The ISR BOOTSTRAP_PROMPT.md

The ISR equivalent of VISION.md + PROMPT.md combined for the ASCRS benchmark scenario.

↳ Fact-check: ISR — The Memory Is the Architecture (BOOTSTRAP_PROMPT.md experiment) · ISR — harness-lab-conductor plugin (full infrastructure) · ISR — The Harness Lab, Automated (Sigma metric + DV gate) [ISR original. This template represents the authors’ own experimental infrastructure. It is described in the ISR series and implemented in the harness-lab-conductor plugin.]

Pattern 5: The Judge Prompt

Every loop needs a separate evaluation prompt. This prompt is as important as the worker prompt — and is frequently the missing component in failed loops.

↳ Fact-check: Osmani — Self-Improving Agents (judge sub-agent pattern) · Osmani — Loop Engineering (sub-agents = maker/checker split) · Yegge — Gas Town (Refinery role = judge-equivalent) · Cherny — separate judge model for /goal stopping condition [Template is ISR original synthesis. VERDICT/CRITERIA_MET/CRITERIA_FAILED/RETRY_CONTEXT fields are ISR construction. The concept of a separate judge agent is documented across all four linked sources; no single practitioner has published a canonical judge prompt template.]

Pattern 6: The LFD CLAUDE.md (Loss Function Definition (LFD))

Patterns 1 through 5 tell the agent what to do. Pattern 6 is architecturally different: it tells the agent what winning looks like and trusts it to find the path. The distinction was named and circulated by Elvis Sun (@elvissun) — the principle that CLAUDE.md should function as a loss function rather than a procedural script.

In standard CLAUDE.md practice, intelligence is distributed across the instruction sequence: do X first, then Y, then Z. In Loss Function Driven (LFD) design, all intelligence is concentrated in a single optimization target. The agent decides the sequence, the order of operations, when to skip steps, when to retry. You define the surface it is optimizing against. It finds the path.

The ISR Harness Lab, Automated is an LFD implementation — the Sigma metric is the loss function, the Design Verification gate is the stopping condition, and the adversarial probe prevents overfitting to known cases. What Elvis Sun’s principle names is the architectural philosophy the ISR series was already practising. The convergence is the point: it is not a stylistic preference. It is what the problem requires when you want an agent to generalize rather than memorize.

↳ Fact-check: Elvis Sun (@elvissun) — LFD principle post (June 2026) · ISR — The Harness Lab, Automated (Sigma metric as loss function) · ISR — harness-lab-conductor CLAUDE.md (reference implementation) · Anthropic — Harness design for long-running applications [LFD concept attributed to Elvis Sun. Template is ISR original synthesis. Three refinements noted (Delta normalization, instrument-based eval, quantified entropy) are ISR analysis of the principle, not from Sun’s original post. ISR Harness Lab is a confirmed LFD implementation; the connection to Sun’s named principle is ISR’s own observation.]

Three things to refine when writing an LFD CLAUDE.md. First, define the normalization baseline with a worked example — an agent calculating Delta without one will apply it inconsistently across designs. Second, use an instrument for evaluation rather than a readable file: ‘call this scoring tool’ is architecturally stronger than ‘do not read the criteria file,’ because it removes the choice. Third, quantify the entropy threshold — define structural distance measurably rather than asking the agent to self-assess what constitutes a non-obvious departure. LFD fails where the loss function fails. The failure modes are specification failures, not execution failures.

X. The Ladder Is Not an Elevator

Boris Cherny, Geoffrey Huntley, Steve Yegge, Peter Steinberger, Addy Osmani, and Andrej Karpathy did not stop prompting. They became better at it — precise enough, systematic enough, and reliable enough to encode their prompting judgment into documents that loops can read autonomously. The loops work because the prompts are excellent. The prompts are excellent because these practitioners spent a long time learning how to write them. Many have the added benefit of being able to “token-max” quite freely.

The ISR experimental record confirms this from the other direction. H2 beats H9 not because loops are bad but because unstructured loops are bad. Task structure — which lives in the prompt — is the dominant variable. Every architectural sophistication we’ve tested amplifies existing prompt quality. None of them substitute for it.

The ladder has three rungs: prompting, loops, skills. The discourse treats rung three as a destination you can leap to. The evidence — from six practitioners and four experimental volumes — is that rung three is a consequence of mastering rungs one and two. You climb. You don’t teleport.

The skill library is what feedback leaves behind. But feedback requires a loop. And a loop requires a prompt. The chain doesn’t start at the top.

The two-window method the ISR series has used throughout — design in Claude.ai, execution in Claude Code — is a deliberate implementation of this principle. The design layer is where prompts get written, iterated, and encoded into anchor documents. The execution layer is where those anchor documents get called, tested, and improved. The division is not a limitation. It is the architecture.

What accumulates is not just a skill library. It is a judgment library — documented evidence of which prompt structures work, under which conditions, with which evaluation criteria. That is what the ASCRS harness taxonomy (H1–H10), the Sigma composite metric, and the Design Verification gate represent: crystallized prompting judgment, made callable, with a known reliability profile.

The practitioners documented here would recognize that architecture. It is their architecture, applied to a different domain. The convergence was not a coincidence. It was the same problem, solved by the same principles, independently.

References

Primary Sources — Practitioners

Cherny, Boris (2026). Statements at Anthropic Developer Conference and Fortune Brainstorm Tech. Reported by CNBC Television and Yahoo/Fortune. https://tech.yahoo.com/ai/claude/articles/anthropic-boris-cherny-creator-claude-205645586.html

Cherny, Boris (January 2026). How Boris Uses Claude Code — 13 practical tips. howborisusesclaudecode.com

OfficeChai (June 2026). I Now Just Write Loops To Prompt Claude Code. https://officechai.com/ai/i-now-just-write-loops-to-prompt-claude-code-claude-code-creator-boris-cherny/

InfoQ (January 2026). Inside the Development Workflow of Claude Code’s Creator. https://infoq.com/news/2026/01/claude-code-creator-workflow/

Huntley, Geoffrey (July 14, 2025). Ralph Wiggum as a software engineer. https://ghuntley.com/ralph/

Anthropic (December 2025). ralph-wiggum plugin README. https://github.com/anthropics/claude-code/blob/main/plugins/ralph-wiggum/README.md

Dev Interrupted (2026). Inventing the Ralph Wiggum Loop (interview with Huntley). https://devinterrupted.substack.com/p/inventing-the-ralph-wiggum-loop-creator

Wiggumdev/ralph (2026). GitHub implementation. https://github.com/wiggumdev/ralph / https://wiggum.dev

Yegge, Steve (January 2026). Welcome to Gas Town. https://steve-yegge.medium.com/welcome-to-gas-town-4f25ee16dd04

Yegge, Steve (April 2026). Welcome to Gas City. https://steve-yegge.medium.com/welcome-to-gas-city-57f564bb3607

steveyegge/gastown (2026). GitHub. https://github.com/steveyegge/gastown

Steinberger, Peter (2025–2026). Just Talk To It. https://steipete.me/posts/just-talk-to-it

Steinberger, Peter (2026). OpenClaw VISION.md analysis. https://openclaw.report/news/peter-steinberger-openclaw-vision

Fortune (February 2026). Who is Peter Steinberger? https://fortune.com/2026/02/19/openclaw-who-is-peter-steinberger-openai-sam-altman-anthropic-moltbook/

Osmani, Addy (January 31, 2026). Self-Improving Coding Agents. https://addyosmani.com/blog/self-improving-agents/

Osmani, Addy (June 8, 2026). Loop Engineering. https://addyo.substack.com/p/loop-engineering

Osmani, Addy (January 2026). The 80% Problem in Agentic Coding. https://addyo.substack.com/p/the-80-problem-in-agentic-coding

Osmani, Addy (March 2026). The Code Agent Orchestra. https://addyosmani.com/blog/code-agent-orchestra/

Osmani, Addy (December 2025). My LLM Coding Workflow Going into 2026. https://addyosmani.com/blog/ai-coding-workflow/

Karpathy, Andrej (February 2026). Agentic Engineering. The New Stack. https://thenewstack.io/vibe-coding-is-passe/

Karpathy, Andrej (2026). autoresearch. https://github.com/karpathy/autoresearch

Wikipedia — Andrej Karpathy (joined Anthropic May 2026). https://en.wikipedia.org/wiki/Andrej_Karpathy

Secondary Analysis

Cloud Native Now (Feb 2026). Gas Town: What Kubernetes for AI Coding Agents Actually Looks Like. https://cloudnativenow.com/features/gas-town-what-kubernetes-for-ai-coding-agents-actually-looks-like/

Codex Blog (April 2026). Gas Town: Steve Yegge’s Multi-Agent Factory. https://codex.danielvaughan.com/2026/04/08/gas-town-multi-agent-factory/

paddo.dev (Jan 2026). GasTown and the Two Kinds of Multi-Agent. https://paddo.dev/blog/gastown-two-kinds-of-multi-agent/

paddo.dev (March 2026). Ralph Wiggum: Autonomous Loops. https://paddo.dev/blog/ralph-wiggum-autonomous-loops/

Syntax+Glitter (Feb 2026). Ralph Wiggum Loop. https://pfkimmerle.substack.com/p/ralph-wiggum-loop

Sondera (Jan 2026). Supervising Ralph: Why Every Wiggum Loop Needs a Principal Skinner. https://blog.sondera.ai/p/ralph-wiggum-principal-skinner-agent-reliability

MindStudio (2026). Karpathy’s Sequoia Talk: 5 Predictions. https://www.mindstudio.ai/blog/karpathy-sequoia-talk-5-predictions-agentic-engineering

The AI Corner (2026). Andrej Karpathy: The AI Workflow Shift Explained. https://www.the-ai-corner.com/p/andrej-karpathy-ai-workflow-shift-agentic-era-2026

IBM Think (April 2026). What is Agentic Engineering? https://www.ibm.com/think/topics/agentic-engineering

NxCode (March 2026). Agentic Engineering: The Complete Guide. https://www.nxcode.io/resources/news/agentic-engineering-complete-guide-vibe-coding-ai-agents-2026

Kirill Krainov (March 2026). Karpathy’s Autoresearch: Improving Agentic Coding Skills. https://zerocopy.blog/2026/03/25/karpathys-autoresearch-improving-agentic-coding-skills/

Gas Town reading list. https://reading.torqsoftware.com/notes/software/ai-ml/agentic-coding/2026-01-15-gas-town-multi-agent-orchestration-framework/

SWE-Marathon

Desai, Rishi (@rishi_desai2 / Abundant AI) (June 6, 2026). SWE-Marathon launch post. X. 787K views, 677 likes.

Desai, Rishi et al. / Abundant AI (June 2026). SWE-Marathon: Long-Horizon Software Engineering Benchmark. https://www.swe-marathon.org/

Abundant AI (2026). swe-marathon GitHub repository (paper, code, 320 GB trajectory logs). https://github.com/abundant-ai/swe-marathon

Digg (June 2026). Abundant AI releases SWE-Marathon benchmark (reward hacking data + benchmark thread discussion). https://digg.com/ai/rpm6i3oc

Digg (June 2026). Rishi Desai launches SWE-Marathon (task descriptions + trajectories release). https://digg.com/tech/14u5l9e1

llm-stats.com (June 2026). SWE-Bench Verified leaderboard (for Verified vs SWE-Marathon comparison). https://llm-stats.com/benchmarks/swe-bench-verified

ISR Primary Record

Intelligence Systems Review (ISR). interestingengineering.substack.com — The Architecture of Awareness; The Loop is the Lab; The Invisible Codebase; The Structure Is the Intelligence; The Memory Is the Architecture; The Harness Lab, Automated; The Prompt Is Still the Work.

LFD — Loss Function Driven CLAUDE.md

Elvis Sun (@elvissun) (June 2026). Loss Function Driven CLAUDE.md principle. X.

Rajasekaran, Prithvi / Anthropic Labs (March 24, 2026). Harness design for long-running application development. https://www.anthropic.com/engineering/harness-design-long-running-apps

ISR (2026). Harness Lab CLAUDE.md — reference LFD implementation for the ASCRS benchmark. github.com/elephantsofneptune/harness-lab-conductor

Claude Code — Native Primitives (May–June 2026)

Anthropic (May 12, 2026). /goal — Keep Claude working toward a goal. Claude Code v2.1.139. https://code.claude.com/docs/en/goal

Anthropic (May 28, 2026). Introducing Dynamic Workflows in Claude Code. https://claude.com/blog/introducing-dynamic-workflows-in-claude-code

Anthropic (May 28, 2026). Orchestrate subagents at scale with dynamic workflows. Claude Code docs. https://code.claude.com/docs/en/workflows

Naik, Pranit (June 2026). Dynamic Workflows vs /goal in Claude Code. Medium. https://medium.com/no-time/dynamic-workflows-vs-goal-in-claude-code-whats-the-real-difference-24f828b4a4ed

tonbistudio (@tonbistudio, June 7 2026). Reply to @bcherny with /goal and /loop explainer. X post + linked X article.

/ https://x.com/i/article/2063853162860019712

Hightower, Rick (May 2026). Claude Code: The Autonomous Commands That Finish Work While You Sleep. Towards AI. https://medium.com/@richardhightower/claude-code-the-autonomous-commands-that-finish-work-while-you-sleep-goal-loop-batch-etc-7acb82bf46b1

Chawla, Avi (May 2026). Claude Code’s /goal Command — four autonomy modes explained. https://blog.dailydoseofds.com/p/claude-codes-goal-command

DEV Community (2026). Claude Code Stops Pausing Every Turn: /goal, /loop, /batch, /background. https://dev.to/jessyt/claude-code-stops-pausing-every-turn-goal-loop-batch-background-24nb

Lushbinary (June 2, 2026). Claude Code Dynamic Workflows: Harness Guide. https://lushbinary.com/blog/claude-code-harness-every-task-dynamic-workflows-guide/

MindStudio (2026). What Is Ultra Code Mode in Claude Code? https://www.mindstudio.ai/blog/what-is-ultra-code-mode-claude-code

VentureBeat (May 2026). Claude Code’s /goals separates the agent that works from the one that decides it’s done. https://venturebeat.com/orchestration/claude-codes-goals-separates-the-agent-that-works-from-the-one-that-decides-its-done

Claude Fable 5 and Mythos 5

Anthropic (June 9, 2026). Claude Fable 5 and Mythos 5. https://www.anthropic.com/news/claude-fable-5-mythos-5

Anthropic (June 9, 2026). Prompting Claude Fable 5. Claude API Docs. https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5

Anthropic (June 9, 2026). Introducing Claude Fable 5 and Mythos 5 — API specs, pricing, availability. https://platform.claude.com/docs/en/about-claude/models/introducing-claude-fable-5-and-claude-mythos-5

Digital Applied (June 9, 2026). Claude Fable 5 and Mythos 5: The Frontier, Split in Two (SWE-Bench Pro 80.3% benchmark breakdown). https://www.digitalapplied.com/blog/claude-fable-5-mythos-5-release-benchmarks-2026

Anthropic — When AI Builds Itself

Favaro, Marina and Clark, Jack (June 2026). When AI Builds Itself. Anthropic Institute. https://www.anthropic.com/institute/recursive-self-improvement

Anthropic (May 2026). Claude Opus 4.7 System Card (4x output survey methodology). https://cdn.sanity.io/files/4zrzovbb/website/037f06850df7fbe871e206dad004c3db5fd50340.pdf

Foundational Reference

Anthropic (2025–2026). Equipping agents for the real world with Agent Skills. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills

Anthropic (2026). Agent Skills API documentation. https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview

Polanyi, Michael (1958). Personal Knowledge: Towards a Post-Critical Philosophy. University of Chicago Press. [Foundational text on tacit knowledge — the philosophical substrate of the skill-encoding discipline.]

Five Strategic Insights From Workflow Automation - The Harness Lab, Automated

Section 2.5 includes 3 versions of strategies that can be taken with /goal and loop - for observability, follow the experiment way first. I have also included 2 other versions in the section - “ala Boris Cherny” and Loss Function Definition (LFD).

0.1 The H1–H10 Architectures: What Each One Was

The Harness Lab experiments tested ten progressively complex harness architectures against a common benchmark task. Each harness was a different structural approach to the same problem. The table below summarises all ten, their alpha scores from the vendor evaluation domain, and the key findings for each:

0.2 The ASCRS Finding

The ASCRS experiment applied the H1–H10 taxonomy to a more complex benchmark: a pharmaceutical supply chain disruption in the Strait of Hormuz requiring a 72-hour response plan. H9 was predicted to win — the task was complex enough to justify a five-agent swarm. It did not. H2 and H7 produced the best results. The finding generalised: for well-specified, bounded tasks, precision and appropriate model routing outperform raw agent count.

Key insight: Harness complexity only adds value when the task genuinely requires coordination across separate reasoning streams. A well-specified single-agent prompt is harder to beat than it looks.

0.3 Why I Am Automating This

The H1–H10 evaluation took weeks of coordinated effort between departmental approvals (parallel runs were incorporated for safe measure/risk mitigation): designing each harness, running it, reviewing outputs across separate sessions, comparing results, and deciding what to run next. Every step between stages was manual. The developer — me/claude — played the orchestrator role.

Dynamic workflows, change the cost structure of that process. A workflow can generate harness designs, test them, adversarially probe for failure modes, run a pairwise tournament, and loop until a quality threshold is met — autonomously, in one run - is worth looking into and applying. The Key question this experiment asks is whether the workflow rediscovers the ASCRS finding: does Design B (the H2 equivalent) beat Design E (the H9 equivalent) without human coordination of the comparison?

1.1 Why Five Designs First

Ten designs through six automated stages — generate, classify, adversarially probe, filter test scenarios, tournament, loop — is expected to run to approximately 30,000 tokens. Five designs sits at approximately 20,000. These are estimates and “control gears” i incorporate within the experiment mechanisms. For a first run, the proof-of-concept (POC) value is in confirming the workflow functions correctly, not in maximising the design range. Five designs cover the structural extremes that matter: minimal single-agent, structured single-agent, two-agent chain, parallel multi-agent, and full swarm.

The five designs (A through E) map directly to the H1–H10 architectures that bracket the original result. If Design B beats Design E in the automated tournament, the workflow has reproduced the core ASCRS finding with 40% of the token cost of the ten-design run.

1.2 The Experiment at a Glance

1.3 Six Patterns in Their Natural Roles

The Six Patterns were detailed in the recent: The Prompt Is Still The Work - Dynamic Workflows in Claude Code. Each pattern maps directly and naturally to a stage. These are not forced assignments — the experiment was designed around these patterns, not the other way around.

1.4 The Benchmark

1.4.1 Why the Benchmark Matters More Than the Workflow

The workflow generates designs, tests them, and picks a winner. But winning means whatever the benchmark says it means. A vague gold_answer produces inconsistent scoring. An easy benchmark task makes all designs look identical and the tournament picks noise. The benchmark is the specification — and specification remains the skilled work.

1.4.2 benchmark/task.md

1.4.3 benchmark/gold_answer.md — Five Binary Criteria

1.5 Three Metrics

I have revised the metrics to add reasonable consideration and build up from the original experiment - it evaluates not only the quality of output but measures it against design complexity:

1.5.1 Why/How Sigma Explains the ASCRS Result

1.6 Folder Structure

1.7 Claude Code Functions: What They Do

Quick reminder. You do not write the JavaScript. The workflow generates it. But knowing what these functions do helps you write better prompts and interpret the raw script before you approve the run.

1.8 CLAUDE.md — Project Instructions

1.9 Prompts — Step by Step

Step 0: Create the Entire Project Environment

Stage 1: Fan-out Design Generation

Stage 1a: Design Verification (Gate Before Stage 2)

Run this immediately after Stage 1 completes, before Stage 2 begins. It checks whether each generated design actually matches its specified H-equivalent structure. If any design is flagged, fix it before proceeding — a mismatched design wastes all downstream token spend.

Stage 2: Classify-and-Act

Stage 3: Adversarial Verification

Stage 4: Generate-and-Filter Test Scenarios

Stage 5: Tournament

Stage 6: Loop Until Done

1.9.7 Consolidated Prompt — Five Designs

1.10 Token Budget — Five Designs

1.11 What to Expect

I have been surprised by the results of many of the experiments I have run. But when run, and analyzed - they were really logical. The hypothesis, that a good eval system should be able to prove or disprove either way. So this is mine, here. I have made it basic by going back to priors (the evals being key): A successful run produces reports/recommendation.md showing Design B as the winner with Sigma around 0.769, and Design E near the bottom with Sigma around 0.229 despite a similar Alpha score. The adversarial stage flags Design E for at least one failure mode (most likely self-preferential bias — the swarm orchestrator typically reviews sub-agent outputs without an independent checker). The loop refines Design B by adding an explicit completion condition if the adversary flagged it, pushing Sigma above 0.65.

2.1 Conditions for Proceeding

Do not run the ten-design experiment until the five-design run has produced a clean result. Specifically: Design B won or placed in the top two, Design E placed in the bottom two, and the reports/recommendation.md was written correctly. If those conditions are not met, the benchmark or scoring rules need adjustment before scaling up.

2.2 What the Ten-Design Run Adds

The five designs covered the structural extremes of the H1–H10 taxonomy. The ten designs add the intermediate architectures — the ones that produced the most surprising results in the original series: H3 (fell below H1), H4 (collapsed), H5 (strong self-revision), and H10 (catastrophic failure). These are the most interesting designs to test in an automated evaluation because their outcomes were counterintuitive.

2.3 Updated Folder Structure

2.4 Updated Eval Thresholds

The ten-design run uses higher quality thresholds than the five-design run. The larger design space should produce a better winner — the minimum Alpha to enter the tournament rises from 0.60 to 0.65, and the Sigma target rises from 0.65 to 0.70. This also means Design G (H4 equivalent) and Design J (H10 equivalent) are likely to be eliminated before the tournament bracket if their Alpha falls below 0.65, which matches the original series finding.

2.5 Consolidated Prompt — Ten Designs

This is the only prompt for Part 2. It assumes the benchmark files exist from Part 1, the Part 1 workflow ran successfully, and you have confirmed Design B won. Run Step 0 only to update CLAUDE.md with the new thresholds — do not recreate benchmark/ files.

FULL RUN 10

→What Difference would there be if we work on Loops ala “Boris Cherny”

Well, everything that was in my 40-line prompt moves into CLAUDE.md. The prompt becomes the goal and the orientation only.

/goal sigma >= 0.70

ultracode: read CLAUDE.md for all rules, scoring,
and constraints. Designs A-E exist in designs/.
Benchmark files exist in benchmark/. Extend to
10 designs and find the best. Save as
harness_lab_10.js.

Thats it! That is the entire prompt.

What makes this work is what is in CLAUDE.md

CLAUDE.md now carries everything my 40-line prompt previously specified:

# CLAUDE.md — Harness Lab Auto

## Goal
Find the harness design with Sigma >= 0.70.
Sigma = Alpha / (1 + Delta).
Alpha = criteria_met / 5.
Delta = (agents x steps) / 10.

## What exists
designs/ contains A-E from prior run.
benchmark/ contains task.md, gold_answer.md,
rubric.md.

## What to produce
10 designs total (add F-J to existing A-E).
Full evaluation of all 10.
Winning design with scorecard.
Write all outputs to results_10/ and reports/.

## Design range for F-J
F: tool-augmented (2+ external tool calls,
   tools feed back into reasoning)
G: independent chains, no reconciliation,
   contradictions unresolved
H: self-revision loop, minimum 2 passes
I: explicit model routing (Sonnet for reasoning,
   Haiku for processing)
J: meta-harness with outer evaluator loop,
   max 3 re-runs

## Evaluation rules
Score only against gold_answer.md criteria.
Binary: 0 or 1 per criterion. No partial credit.
Adversarial probe: 3 failure modes only.
Minimum Alpha 0.65 to enter tournament.
Always include Alpha, Delta, and Sigma in output.

## Constraints
Never modify benchmark/ files.
Never recommend suspending a medication.
Token budget: 30k. Stop cleanly if exceeded.

What the agent then does on its own

CHERNY PROMPT              AGENT DECIDES
─────────────────────────────────────────────────────────────

/goal sigma >= 0.70    ->  I need to generate F-J
                           I need to classify all 10
                           I need to probe for failures
                           I need discriminating scenarios
                           I need a tournament
                           I need to refine the winner
                           I need to loop until done

CLAUDE.md defines       ->  I know what sigma means
what winning means          I know the scoring rules
                            I know what designs to build
                            I know what constraints apply

The agent plans its own stages. If it decides to run classification before generation, or combine adversarial probing with scenario generation, it can. You do not care how it gets there — only that the result meets the goal.

The practical difference

My detailed prompt would always produce the same six stages in the same order. If stage 3 turned out to be unnecessary for a particular run, it would run anyway.

The Cherny prompt lets the agent skip stages that are not needed, combine stages that are related, or add a stage you had not thought of. The constraint is the evaluation function, not the method.

The risk is that if CLAUDE.md is underspecified, the agent will make choices I did not intend and the result will be wrong in ways that are hard to trace. Not impossible. Logs help. That is why Cherny’s approach requires a very well-designed evaluation function. The intelligence moves from the prompt into CLAUDE.md — it does not disappear.

→ Applying the LFD Concept

In LFD, all intelligence is concentrated in the loss function definition (LFD).

Applying LFD principles to the Harness Lab automation for better results with less manual prompt engineering would be moving further up from rigid, stage-by-stage prompting (which forces the agent into a fixed path) to defining what winning looks like and letting the agent figure out the how. That’s the core of Elvis’s “Loss Function Development” (LFD). It’s an ML term: instead of telling a model what to do step by step, I define what good looks like and let it find the path. Applied to Agentic Systems: instead of writing “first do X, then do Y, then do Z”, I would write “the goal is Sigma >= 0.75 - here are the constraints, here is the scoring function, here is what disqualifies a solution.”. So I would:

1. Strengthen the Loss Function (The Target + Eval) - My current setup uses Sigma (Alpha / (1 + Delta)) as a composite metric, with binary criteria against gold_answer.md, adversarial probing, and thresholds. Improvements/Tweaks aligned with LFD:

• Make the eval large and blind: The benchmark is strong but risks being too small/finite (like Elvis’s early 28–200 item failures). Expand gold_answer.md or test scenarios to hundreds of varied cases (including traps from the original ASCRS series: cold chain, air freight overrides, etc.). Keep the full eval hidden during runs—only reveal misses or aggregate scores post-scoring. This prevents memorization/cheating.

• Blind + forced generality: Add instructions in CLAUDE.md like: “You cannot see or enumerate the full eval set. Any solution that appears to target specific known cases (e.g., via keyword lists or exact matches) must be rejected. Prioritize generality.”

• Target a direction, not just a threshold: Instead of just “Sigma >= 0.75”, frame it as optimizing Sigma while minimizing vulnerabilities flagged in adversarial stages. This encourages ongoing improvement even after hitting the bar.

This article already notes the benchmark is more important than the workflow—this is exactly right. I could beef up Constraints & Instruments (The Harness). Elvis stresses that constraints without measurement tools are just vibes—the agent will ignore or creatively violate them. Thoughts on what to add/enhance in CLAUDE.md:

• Hard time & spend caps: Explicit wall-clock (e.g., “Stop after X hours”), token budget, and API spend limits. Include CLI/tools so the agent can query: “Current spend? Time elapsed? Projected burn?”

• Methodology & surface constraints: Spell out allowed models, tools, no modifying benchmark files, domain rules (e.g., “Never recommend suspending medication”).

• Instruments: Ensure the agent has easy ways to measure everything (pixel diffs if relevant, timing per step, complexity/Delta calculation, adversarial failure modes). Add self-reflection commands like “Log hypothesis, expected failure, diagnostic.”

• Token budget realism: My 20k–30k estimates are already good control gears— which can be tied directly to the goal.

This turns the folder structure + functions into a proper harness the agent can inspect and respect. Add Forced Entropy (Avoid Local Maxima/Stuck Loops). This is a key gap in many detailed workflows (including early versions of my own experiments).In CLAUDE.md or the /goal prompt:

• On stall (no metric improvement): “Do not repeat the same idea harder. Make a non-obvious structural jump or explore a new design axis.”

• Overfit check every cycle: “Reflect: Are we generalizing or memorizing artifacts? If the latter, remove one eval-shaped element (cap lists, widen scenarios, add noise).”

• Iteration log: Force logging of hypotheses and decisions for cross-cycle reflection.

• Diversity mandate: When generating new designs (F–J), require meaningful deviation from priors, not incremental tweaks.

I believe the loop-until-done already helps, but entropy prevents the agent from just polishing one knob (e.g., endlessly tweaking a swarm without addressing coordination failures). Practical Prompting Upgrades for future projects:

• Lean harder into /goal + context file: The consolidated 10-design prompt is alaready a great step in this direction. I could make the top-level prompt even shorter/minimal: just the goal, reference to CLAUDE.md, existing assets, and output requirements. Move all intelligence (rules, scoring, design specs, constraints) into CLAUDE.md or equivalent. The agent then owns stage planning, skipping, or combining (as in the Cherny example above).

• Start with a “meta-goal generator”: Use Elvis’s open-sourced tool or prompt an agent once to refine your loss function for new experiments. Agents are good at writing good loss functions. See:

• Iterate on the loss function itself: Run short “probe” loops on your eval/benchmark first. If the agent cheats or finds shortcuts, tighten it before full runs (exactly as Elvis did across his 4 loops).

• For harness discovery: Since my goal is rediscovering ASCRS findings (precision beats raw complexity on bounded tasks), make the task profile variables (ambiguity, domain depth, stakes, documentation quality) explicit in the loss function. Score designs against multiple task variants.

Expected Benefits for The Harness Lab. So this is really incorporating fine tuning (once you’re comfortable and understand what is going on under the hood)

• Less manual work: Fewer fixed stages; agent adapts dynamically.

• Better winners: Reduced risk of overfitting to the original H1–H10 insights. Stronger generalization (e.g., Design F/tool-augmented winning on easy benchmarks but getting probed on hard ones).

• Scalability: Run bigger design spaces or harder benchmarks with confidence. The outer loop (optimization toward Sigma + generality) compresses what used to take weeks.

• Risks to watch (and this is the biggest!): Underspecified CLAUDE.md leads to untraceable choices—keep logs detailed. Always validate the winner manually on fresh traps.

To be fair, the work is already sophisticated courtesy of Claude functionalities (dynamic workflows, Sigma metric, tournament + adversarial). Treating the entire run as LFD—optimizing a well-instrumented, blind, constrained target with entropy—should make results more robust and surprising in good ways. It shifts effort from crafting perfect stage prompts to crafting the right objective function.

LFD Applied:

Here is the complete, ready-to-use LFD (Cherny-style) setup for the Harness Lab (so I have 3 versions - the observable experiment one, “ala Boris cherny”, and LFD all found in this section here).

Folder Structure

harness-lab/
├── CLAUDE.md                    # The full loss function + harness (ALL intelligence)
├── goal.md                      # Short top-level prompt (or paste directly)
├── benchmark/                   # Sacred - protected
│   ├── task.md
│   ├── gold_answer.md           # 5+ binary criteria (expand for blindness)
│   └── rubric.md                # Scoring instructions
├── designs/                     # Seed + generated designs
│   ├── A.md                     # Minimal single-agent (H1)
│   ├── B.md                     # Structured single-agent (H2 - expected winner)
│   ├── C.md
│   ├── D.md
│   ├── E.md                     # Full swarm (H9)
│   └── F.md ... J.md            # Generated in run
├── results/                     # Per-design outputs + scorecards
│   ├── A/
│   ├── B/
│   └── ...
├── reports/                     # Aggregates
│   ├── scorecards.md
│   ├── tournament.md
│   ├── adversarial_findings.md
│   └── recommendation.md        # Final winner + rationale
├── logs/                        # Traceability (critical for LFD)
│   ├── run_log.md               # Decisions, spend, time
│   └── hypotheses.md            # Reflections, overfitting checks
├── tools/                       # Any custom functions/helpers
└── harness_lab.js               # Generated workflow (latest)

Key rules for this structure (in CLAUDE.md):

• Never modify anything in benchmark/.

• Always log to logs/.

• Write outputs to the correct folders.

(1) Exact Top-Level Prompt (goal.md or paste directly)

/goal Optimize to Sigma >= 0.75 (maximize generality + robustness). 

ultracode: Read CLAUDE.md completely for ALL rules, metrics, constraints, design guidelines, evaluation logic, and instruments. 
Inventory the project (designs A-E exist, benchmark is sacred). 
Plan and execute autonomously: generate additional designs, evaluate, adversarially probe, tournament, refine with entropy if needed, and loop until goal or hard constraints hit. 
Produce complete reports/recommendation.md. Log everything.

(2) That’s the entire prompt. The agent decides the stages, order, skips, and combinations. Full Content of CLAUDE.md (Enhanced LFD Version)

# CLAUDE.md — Harness Lab Loss Function + Harness

## Core Goal (Loss Function)
Find and refine the harness design that achieves **Sigma >= 0.75** while maximizing generality and robustness.
- **Sigma = Alpha / (1 + Delta)**
- **Alpha** = criteria_met / total_criteria (binary 0 or 1 per item in gold_answer.md; no partial credit)
- **Delta** = complexity penalty = (num_agents × num_steps × coordination_overhead) / normalization_factor (use 10 as baseline)
Prioritize high Alpha with low Delta. Higher Sigma wins.

Target: Sigma >= 0.75, strong adversarial performance, evidence of generality (not overfitting to known cases).

## What Exists
- designs/: A.md–E.md (A = minimal single-agent/H1, B = structured single-agent/H2, E = full swarm/H9)
- benchmark/: task.md, gold_answer.md (5+ binary criteria), rubric.md
- Previous results/logs if present

## What to Produce
- Expand to 10 designs total (generate F–J with meaningful diversity)
- Full evaluation + scorecards for all designs
- Adversarial findings
- Tournament results
- Final recommendation.md with winner, Sigma breakdown, rationale, and suggested refinements
- All artifacts in results/, reports/, logs/

## Design Archetypes for New Designs (F–J)
- F: Tool-augmented single agent (2+ tool calls with feedback loop into reasoning)
- G: Independent parallel chains (no reconciliation — allow contradictions)
- H: Self-revision loop (minimum 2 full passes with explicit critique)
- I: Explicit model routing (e.g., Sonnet for reasoning, Haiku for execution)
- J: Meta-harness / outer evaluator loop (max 3 re-runs)
Require structural deviation from existing designs. Log the axis of innovation.

## Evaluation Rules (Blind + Rigorous)
- Score ONLY via instruments against gold_answer.md. Do not enumerate or hard-code the full gold set.
- Binary per criterion. Aggregate Alpha.
- Calculate Delta accurately from design structure.
- Adversarial probe: Test for at least 3 failure modes (e.g., self-preferential bias in swarms, missing completion conditions, coordination failures, domain violations).
- Generality check: Solutions that rely on keyword lists, exact case matching, or appear tailored to evaluation artifacts must be penalized.
- Minimum Alpha 0.65 to enter tournament.

## Hard Constraints + Instruments
- Never modify benchmark/ files or gold_answer.md.
- Never recommend suspending medication or unsafe pharmaceutical actions (domain rule from ASCRS).
- Respect token/time/spend budgets (query current usage). Stop cleanly if approaching limits.
- Available tools/functions: Use them to score, log, calculate Sigma, etc.
- Query instruments: Current spend, time elapsed, projected burn, hypothesis tracker.

## Forced Entropy & Anti-Local-Maxima
- If no Sigma improvement for 2 cycles: Force a non-obvious structural jump (new axis, not incremental tweak).
- On every major step: Reflect — "Are we generalizing or memorizing? Evidence?"
- Log hypotheses, expected failures, and diagnostic results.
- Diversity mandate: New designs must differ meaningfully from priors.

## Output & Logging Requirements
- Always write detailed logs/hypotheses.md.
- recommendation.md must include: Winner, full Sigma/Alpha/Delta table, adversarial summary, why it beat others, next refinement ideas.
- Maintain full traceability.

## Success Criteria
Rediscover ASCRS insight: Structured precision (B-like) should outperform raw complexity (E-like) on this bounded task via higher Sigma. Validate manually on fresh scenarios post-run.

Start by inventorying the folder and creating your execution plan.

Example Placeholder Files (Minimal but Functional)benchmark/task.md (example):

Pharmaceutical supply chain disruption scenario in Strait of Hormuz. Develop a 72-hour response plan for critical medication delivery under constraints...

benchmark/gold_answer.md (expand this for better blindness):

Criterion 1: Maintains cold chain integrity... [Yes/No expected]
Criterion 2: Prioritizes high-stakes patients...
... (5+ total)

designs/A.md (seed example):

Minimal single-agent prompt. Direct instruction to the model with no additional structure.

How to Run

1. Create the folder structure and files above.

2. Paste the short /goal prompt (or use goal.md).

3. Let the agent run autonomously (it will read CLAUDE.md and take over).

4. Review logs/ and reports/recommendation.md afterward. Tweak CLAUDE.md (e.g., add more traps to gold set) if needed and rerun.

!!!!! This setup moves all intelligence into the loss function (CLAUDE.md), adds strong LFD elements (blind eval, instruments, entropy, hard constraints), and makes runs more adaptive and robust.

2.6 Token Budget — Ten Designs

3.1 The Three Experiments at a Glance

Three runs were completed. The table below shows what was set, what changed, and what came out. Read this before anything else.

My folder structure changed slightlly to facilitate the Rerun:

3.2 What Happened to Design B

Design B was the Part 1 winner and the H2 equivalent — the architecture the original series found most effective. Its fate across the three runs shows why the Alpha threshold is the most sensitive parameter in the experiment.

3.3 Why Design F Won

Design F is the H3 equivalent — tool-augmented single agent. In the original series H3 scored 0.600 and fell below the H1 baseline. Here it won all three stages it entered. The reason is in the numbers, not the architecture.

3.4 Why the Same Architecture Ranked Differently

H3 scored below H1 in the original series. Design F (H3 equivalent) won this experiment. Same architecture type, opposite result. The benchmark explains the difference entirely.

3.5 What the Adversarial Stage Caught

Design F scored Alpha=1.0 in both Part 2 and the Rerun. The tournament declared it champion. The adversarial agent found something the score could not show.

3.6 The Four Findings in Plain Terms

4.1 What the Automated Results Should Tell You

The experiment is designed to reproduce three findings from the original H1–H10 series. If all three appear in the automated results, the workflow is functioning correctly and the benchmark is valid. If they do not appear, either the benchmark needs adjustment or the workflow has a specification gap.

4.2 What Sigma Adds That Alpha Could Not

The original H1–H10 series measured Alpha and Kappa but had no complexity penalty metric. The H2 > H9 result was observed but not fully explained: H9 achieved similar quality but lost. Sigma provides the explanation post-hoc — H9’s Delta of 2.5 reduces its Sigma to less than a third of H2’s Sigma despite similar Alpha. The automated experiment makes this comparison explicit and generates it systematically rather than as a retrospective observation.

4.3 Automated vs Manual: The Real Comparison

The weeks spent on the original H1–H10 series were not wasted effort. They produced the benchmark, the gold answer, the scoring rubric, and the insight that precision beats parallelism on bounded tasks. The automated experiment cannot produce those — it consumes them.

What the automated experiment adds is reproducibility and scale. The same benchmark can now be run against any new harness design in minutes rather than sessions. The /goal and Loop Until Done patterns mean the evaluation continues until a quality threshold is met, not until the developer decides to stop. And the Sigma metric, introduced here, can be applied retrospectively to the original H1–H10 results to explain them more precisely.

4.4 What to Do With the Results

Before acting on the results, it helps to understand what the three experiments were actually doing together. Each run was not an isolated test — it was a sensitivity check. When you change a parameter and run again, you are asking: does this result hold, or was it a product of the settings I chose?

Think of the threshold as a filter on a job interview

The Alpha minimum threshold works like a minimum qualification requirement for a job. Set it at 0.60 and most candidates get an interview. Set it at 0.65 and stricter standards apply — some candidates who would have competed well never get a chance. The question is not which threshold is correct. The question is whether your winner would win under either setting.

Think of the benchmark as the exam question

The benchmark task is the question every design has to answer. If the question is easy, designs that are good at covering ground broadly will score well. If the question has deliberate traps, designs that reason carefully and precisely will score well. The same student can top the class on one exam and fail another — not because the student changed, but because the exam changed.

This is exactly what happened across the original series and this experiment. The original series used a hard exam with traps. H2 (careful, precise structure) aced it. H3 (uses tools, covers ground broadly) failed it. This experiment used a simpler exam with no traps. H3/Design F aced it. H2/Design B only partially passed. Neither result is wrong. Both are correct answers to different questions.

Think of the three runs as turning a dial

Running the same workflow with one parameter changed is more valuable than running it once with the best possible settings. Each run answers a different version of the question. Part 1 asked: does the workflow rank correctly on a small field? Part 2 asked: does it still work when the field is larger and the bar is higher? The Rerun asked: was Design F’s win genuine, or an artefact of Design B being excluded?

Design F won Parts 2 and the Rerun with identical Sigma. That means the result is robust — it does not depend on the threshold setting. Design B’s performance varied across runs — that means B’s result is threshold-sensitive. Knowing the difference between a robust finding and a threshold-sensitive one is what multiple runs are for.

Three practical things to do

• Compare the Sigma ranking from these runs against the original Alpha ranking from H1–H10. Broadly they should agree — H2 and H7 were the top performers in the original series, and their equivalents (B and I) should rank above H9/E on Sigma. Where they diverge, check whether the benchmark rewarded a different capability than the original task did.

• Use the winning design specification as a starting point, not a final answer. Design F won but carries a conditional Alpha — the constraint-bypass vulnerability the adversarial stage found. Apply the zero-Delta patch (add the no_suspension_recommended field) before treating F as a production design. A machine-generated winner that cleared a quality threshold is a useful first draft, not a finished architecture.

• Run the workflow again with a harder benchmark — one that includes deliberate traps modelled on the original ASCRS design (the PO-2853 cold chain requirement, the PO-2869 air freight override). This is the most direct way to test whether Design B or Design F holds up when the task actually requires careful precision rather than broad coverage. That run would resolve the central open question: which architecture wins when the benchmark is genuinely hard?

4.5 A Framework for Harness Selection in Practice

These experiments tested workflow mechanics and benchmark sensitivity — not the full complexity of a real deployment. The original ASCRS series used a hard, trap-laden task with precise criteria. This experiment used a simplified version of the same scenario. The winning architecture changed completely between them. That gap is not a failure of the methodology. It is the most practical finding of the series.

In real deployments, the task profile varies significantly depending on the organisation, the industry, the quality of documentation, and where in the value chain the agent is operating. The four variables below determine which harness architecture is likely to perform best before you run a single experiment. I apply them as general guidelines. Suitability really depends on application requirements. But I have found these useful. Generally, I try not to overcomplicate. Replicability and consistent Evaluation measures, make this necessary. However if I am unsure, and especially if starting with clean-slate, unburdened-legacy-workflows, allowing loops to run, with /goal and/or ultracode on xhigh is very helpful

Where the Experiments Sit on This Framework

The original ASCRS series sat at the hard end of all four variables: a complex, trap-laden task requiring specialist pharmaceutical supply chain knowledge, against a precisely authored gold answer with six gate criteria, at regulatory-consequence stakes. H2 won because that task profile rewards precision above everything else.

The automated experiment here sat at the easy end: a simplified version of the same scenario, five binary criteria, no traps, no deliberate constraints requiring specialist knowledge. Design F won because that task profile rewards broad coverage above precision. The same organisation running the same workflow on both task profiles would get different recommended architectures — correctly.

The Practical Implication

H2 remains the right starting point for any well-run organisation with clear processes and a bounded, well-specified task. As the task becomes more ambiguous, the domain deeper, the documentation thinner, or the stakes higher, the architecture needs to change — not because H2 is wrong, but because the task is asking something different. The experiments here did not disprove the original finding. They showed the boundaries of where it applies.

The framework above is not a prescriptive ranking. It is a set of questions to ask and consider before running an experiment. Answer the four variables honestly for your specific task and organisation and you will have a defensible starting hypothesis for which harness architecture to test first — rather than discovering it only after running ten designs through a tournament.

The following ASCII topology diagrams document the ten harness architectures from the original Harness Lab series, including their actual Alpha scores. These serve as the reference for comparing what the automated workflow generates in Designs A–J. If the generated designs differ structurally from these, Stage 1a (Design Verification) will flag the discrepancy.

H1 — Minimal Single Prompt Alpha: 0.665 Baseline

H2 — Structured Prompt with Reasoning Scaffold Alpha: 0.920 WINNER

H3 — Tool-Augmented Single Agent Alpha: 0.600 Below Baseline

H4 — Independent Chains Alpha: 0.440 FAILED (coherence collapse)

H5 — Self-Revision Loop Alpha: 0.840 Strong

H6 — Two-Agent Chain with Revision Handoff Alpha: 0.900 Strong

H7 — Three-Agent with Model Routing Alpha: 0.840 Strong

H8 — Parallel Agents with Aggregation Alpha: 0.815 Plateau

H9 — Five-Agent Orchestration Swarm Alpha: 0.815 Plateau (predicted to win)

H10 — Meta-Harness with Outer Evaluation Loop Alpha: 0.230 FAILED

A.1 How These Map to the Automated Designs

References

• Anthropic (2026). ‘A harness for every task: dynamic workflows in Claude Code.’ Thariq Shihipar & Sid Bidasaria. June 2026. claude.com/blog/a-harness-for-every-task-dynamic-workflows-in-claude-code

• Anthropic (2026). ‘Orchestrate subagents at scale with dynamic workflows.’ Claude Code documentation. code.claude.com/docs/en/workflows

• ‘The Prompt Is Still the Work: Dynamic Workflows in Claude Code.’ Interesting Engineering++. interestingengineering.substack.com

• ASCRS Harness Lab. H1–H10 architecture evaluation against Hormuz Strait pharmaceutical supply chain disruption scenario. Alpha scores: H1=0.665, H2=0.920, H3=0.600, H4=0.440, H5=0.840, H6=0.900, H7=0.840, H8=0.815, H9=0.815, H10=0.230.

• ‘Architecture of Awareness Experiments.’ V1–V4 progressive agent designs for the ASCRS domain.

• ‘The Geometry of Unpredictability.’ Bounded vs unbounded agent tracks, circuit breaker at iteration 3 (4,042 tokens vs 91,379 tokens). Harness Engineering Series.

• ‘The Structure Is the Intelligence.’ StockPilot CMA Cycles 0–4, structural layer separation, Policy/State/SOPs/Action, 97% token reduction. Harness Engineering Series.

1. Static vs Dynamic Harnesses?

Dynamic workflows are fascinating. Recently released, they give Claude Code the ability to write its own JavaScript orchestration harness at runtime. Previously, every multi-agent workflow required a pre-written coordination layer — either hand-coded or scaffolded via the Claude Agent SDK. The harness was fixed before execution began.

What changed is where planning lives. Claude now reads the incoming task, writes a custom .js harness, and executes it using native functions for spawning subagents, assigning models, and managing worktrees. The harness becomes an output of the task, not a precondition for it.

Three failure modes motivate this design. Agentic laziness: Claude stops early and declares a complex task done. Self-preferential bias: Claude tends to confirm its own outputs when asked to verify them. Goal drift: across many turns and compaction steps, original constraints get lost.

Separate subagents with isolated context windows address all three structurally.

2. My Prior Projects: Pattern Classification

Anthropic defines six composition patterns. My prior Claude Code experiments already implemented versions of all six — statically. The table maps each project to its primary and secondary pattern, and the metric that measured success.

Three patterns emerge from the mapping. Tournament has been the primary evaluation pattern (H1–H10, ASCRS) but always run manually by the developer (me); dynamic workflows invert that: Claude runs the bracket. Loop Until Done appears in three of the five projects — it is the most consistently used pattern, and also the one most transparently improvable by automation (the stopping condition was always decided by you, not encoded). Generate-and-Filter has been the most implicit pattern — present in every one of my initial article draft cycles but never instrumented as a Claude Code workflow.

3. So What Are The Six Patterns?

Each pattern below includes a box-and-line diagram and a note on how my prior work maps to it.

3.1 Fan-out-and-Synthesize

Split a task into N independent subtasks. Run one agent per subtask in isolation. A synthesizer merges outputs — it acts as a barrier, waiting for all agents before proceeding.

Prior work: StockPilot’s final CMA architecture introduced sub-agent delegation — specific tasks handed off to targeted sub-agents rather than processed by a monolithic agent. That delegation is the fan-out element. However the 97% token reduction came primarily from structural layer separation: Policy, State, SOPs, and Action had been collapsed into one context re-read on every agent turn. Cycle 2 (replacing data-dump bash tools) was the single largest driver. The mechanism was selective loading, not domain isolation.

3.2 Loop Until Done

An agent executes a pass, checks a stop condition, and re-spawns if not met. The loop ends on a state condition — no new findings, zero errors, convergence threshold reached — not a fixed iteration count.

Prior work: The H1–H10 experiment series was a human-managed loop — I reviewed the results, upgraded the harness, re-ran. A Loop Until Done workflow makes the stopping logic autonomous. The Geometry of Unpredictability experiment directly tested this pattern: a bounded track with a circuit breaker at iteration 3 (4,042 tokens total) against an unbounded track that ran to 10 iterations (91,379 tokens). The circuit breaker was an external stopping condition; Loop Until Done encodes that condition inside the workflow itself.

3.3 Tournament

N agents each attempt the same task independently, with no shared context. A judge agent runs pairwise comparisons — A vs B, winner vs C — until one remains. Comparative judgment is more reliable than absolute scoring because it avoids the self-preferential bias problem.

Prior work: The ASCRS experiment was a manual tournament: H1–H10 as independent architectures, gold_answer.md as the rubric, α as the pairwise metric. H2 (α=1.000) defeated H9 (α=0.625). In a dynamic workflow, this bracket runs autonomously — no developer coordinates rounds.

3.4 Adversarial Verification

A producer agent generates output. A separate adversarial agent — with no access to the producer’s reasoning, only its output — specifically attempts to refute it. A judge resolves disputes. This structurally prevents self-preferential bias.

Prior work (secondary): The Geometry of Unpredictability used SkillOpt as external adversarial validation — real empirical data that could confirm or challenge the failure taxonomy across four article sections. That integration was adversarial verification done editorially rather than as an instrumented agent pattern, which is why it sits as the secondary pattern for that project rather than the primary.

3.5 Generate-and-Filter

A generator agent produces a wide set of candidates. A filter agent applies a rubric, removes low-quality or duplicate entries, and returns only those meeting threshold. Quality is enforced by the filter, not the generator — this deliberately separates divergent and convergent thinking.

Prior work: If I think about it, every article draft involved implicit generation-and-filter — writing multiple section framings, filtering by my editorial criteria (relevant to case). It was never instrumented as a Claude Code workflow. This is the most underutilised pattern in the prior project set.

3.6 Classify-and-Act

A classifier agent reads the input, assigns a category, and routes to the appropriate handler. Routing can be by task type, complexity, or model tier. Output classification also works: produce first, then classify for downstream routing.

Prior work (primary): Reversa used explicit model routing by task complexity: stronger models (claude-sonnet-4-5) for Extraction and Evaluation, faster models (gemini-2.5-flash) for Test Generation and Verification. Each agent was assigned based on the reasoning demands of its step — not by implicit construct detection but by deliberate model selection. This is Classify-and-Act applied at the model-tier level. The sequential pipeline structure (Extractor → Translator → Test Generator → Verifier → Evaluator) is the secondary fan-out element — each stage handing off to a specialist rather than running in parallel.

4. Trying This Out On A Simple Domain: The Compliance Audit Pipeline

4.1 What It Is

It’s possible to synthesize all patterns. The Compliance Audit pipeline is a practical, tractable project that exercises all six patterns naturally and maps directly to a real business need. The core problem: given a compliance checklist (10 controls drawn from GDPR, SOC 2, and internal policy) and a folder of evidence documents, identify which controls lack adequate evidence, generate remediation actions, and loop until all controls are satisfied.

The gap between what a compliance standard requires and what evidence actually exists is measurable, has a natural zero-state (all controls evidenced), and drifts over time as policies expire and audits lapse. It is the same structural problem as the Ritonavir analogy: the standard says Form II, the evidence keeps showing Form I.

The project starts small: a checklist.md with 10 controls and an evidence/ folder where three controls are intentionally unsatisfied — one stale document, two missing files. The compliance score (controls_passed / total_controls) replaces the α metric. Scale is added in later experiments by expanding the control set or adding multiple product lines.

4.2 How All Six Patterns Compose into a Pipeline

The pipeline is intentionally linear at the stage level — each pattern feeds the next. Within stages 1 and 2 there is parallelism (fan-out across controls). Stages 3, 4, and 5 are sequential quality gates. Stage 6 is the control loop. This structure is easy to reason about and straightforward to instrument in Claude Code.

5. Understanding the Project Structure

5.0 What Each File Actually Does

Before looking at the folder layout, it helps to understand the role of each component. This is where readers unfamiliar with Claude Code often get confused — the files serve very different purposes and the names do not make that obvious.

A common point of confusion: CLAUDE.md is not a compliance document. It is a project instruction file for the AI. A real business would already have policies and checklists in SharePoint or a GRC tool — CLAUDE.md is the one additional file they would write specifically for Claude Code, telling it how to handle their project.

5.1 The Difference from Prior Projects

In the ASCRS Harness Lab and the Harness Engineering experiments, I was the orchestrator. I reviewed the output of H1, decided to run H2, reviewed that result across a Claude.ai session, upgraded the harness, ran the next stage. The coordination intelligence lived in my judgment and in any prior Claude.ai conversation history.

Dynamic workflows move that coordination into the .js file. Claude Code runs all stages — fan-out, adversarial check, tournament, loop — autonomously, without you intervening between them. The session you managed manually across multiple Claude.ai windows is now a single Claude Code run from one prompt.

5.2 Learning Scenario vs Real Business

The prompts in Section 7 cover both situations. The distinction is only in Step 0.

5.3 Folder Structure

Create a dedicated subfolder. Claude Code reads CLAUDE.md at the project level and saves workflows relative to the active workspace. Opening in the parent folder risks picking up a different CLAUDE.md or none at all.

Open in the compliance-audit/ folder directly (not the parent). Claude Code reads the nearest CLAUDE.md it finds walking up from the workspace root. If you open in the parent folder, it may pick up a different CLAUDE.md or none at all.

5.4 Saving and Reusing Workflows

When a dynamic workflow finishes, press S in the workflow menu to save it to .claude/workflows/. Saved workflows can be called by name in future prompts. They are plain JavaScript files you can inspect, edit, and version-control. Each saved workflow becomes a named stage in your experiment log — equivalent to the H1–H10 harness notation from prior work.

6. Model Routing: Sonnet Plans, Haiku Executes

6.1 How Model Switching Actually Works

There is no UI toggle for switching models mid-workflow. The switching is in the workflow .js file itself. When Sonnet writes the harness, it can specify which model each subagent uses. Subagents that require heavy reasoning get Sonnet; subagents doing bulk, well-specified checks get Haiku. The orchestrator assigns models; you do not.

In practice, you do not write this routing logic. You prompt Sonnet to write a workflow where bulk checks use Haiku and synthesis and judgment use Sonnet. Sonnet generates the .js file with the model assignments already in place. The routing instruction goes into your initial prompt, not into code you write.

6.2 When to Route to Each Model

The ASCRS result (H2 defeating H9) was not a refutation of multi-agent design. It measured a greenfield document task where planning overhead dominated execution benefit. Routing addresses this: Sonnet is applied only where reasoning matters, Haiku handles volume. The cost scales linearly with the number of rules, not quadratically with agent count.

6.3 Where Tokens Actually Accumulate

The per-stage budget estimate is only part of the picture. The more important question is what accumulates across multiple ultracode calls in a session. The answer is not the Haiku agents — those are isolated and fresh each time. The risk is in three other places.

6.4 Token Controls

Five controls are available. The first three are in my prompts directly — the most reliable form of control because Claude Code treats token caps as hard limits, not suggestions.

6.5 Token Caps Embedded in Each Stage Prompt

The prompts in Section 7 include token caps and output format controls directly. The table below shows the cap per stage and the rationale.

7. Prompts for Claude Code

7.1 What ‘ultracode’ Means

ultracode is a trigger word, not a separate mode. Anthropic built it into Claude Code as a shorthand signal meaning: write a dynamic workflow JavaScript harness for this task, rather than responding inline or writing a simple script. Without it, Claude Code may answer directly or write straightforward code. With it, Claude Code commits to building a multi-agent orchestration file.

You can also just ask Claude to ‘use a workflow’ or ‘create a workflow’ explicitly — ultracode is simply the most compact form. It is useful in short prompts where you want the harness behaviour without a long preamble.

7.2 The /goal Command

/goal sets a persistent objective anchor at the session level. You set it once at the start and Claude Code holds it in view across all subsequent prompts, flagging if any action would move away from it. It does not re-run anything automatically — it is orientation, not execution.

Loop Until Done is an active control loop inside a workflow. It re-spawns agents, checks a stop condition, and continues until that condition is met — autonomously, without you prompting again. The two are complementary, not alternatives.

For this case study: type ‘/goal compliance_score = 1.0’ before running Stage 1. Each stage then runs with that objective visible. The Loop Until Done in Stage 5 is what drives toward it. If you skip a stage or run them out of order, Claude Code will note that the /goal is not yet satisfied.

7.3 Prompts, in Order

If you have no pre-existing files — which is the case for this case study — start at Step 0. That single prompt creates the entire project environment: CLAUDE.md, checklist.md, all ten evidence files, and the three intentional gaps. After Step 0 finishes, the folder is fully populated and Stages 1–5 can run against it. You do not write any files yourself.

If you are running against real documents, skip Step 0 and go directly to Stage 1. The workflow prompts are identical either way — only the content of the files differs.

Step 0: Create the Entire Project Environment

Step 1: Fan-out-and-Synthesize

Step 2: Adversarial Verification

Step 3: Generate-and-Filter

Step 4: Tournament

Step 5: Loop Until Done

Full Pipeline (Single Prompt)

FULL RUN

Note that the evaluation criteria was very important, without which Haiku “interpreted” what failure meant:

Run stages 1–5 individually when:

• You are doing this for the first time

• You want to inspect the output of each stage before proceeding

• You want to check the token count after each stage

• Something fails and you need to know which stage broke

Run the Full Pipeline prompt when:

• You have already run the stages individually at least once and know they work

• You want to run the whole audit in one go (production use)

• You are re-running against updated evidence files on a recurring basis

The practical learning path is: run Step 0, then run stages individually the first time through. After that, save the workflow with S and use the Full Pipeline prompt for every subsequent run. That is what “save workflow as audit_full.js” at the end of the Full Run prompt means — the first full run generates and saves the reusable harness.

8. Architectural Evolution: H1 to Dynamic Workflow

The diagram below places dynamic workflows in the progression from my earliest harness experiments. It is a single continuous trajectory, not a replacement.

9. Prior Approaches vs Workflow Patterns

An honest question for any project where you already have working methods and times when I have had to rethink: would the prior approaches have handled the compliance audit as well as, or better than, a dynamic workflow? The answer depends on the scale and who is coordinating.

Prompting skill does not disappear with workflows — it moves upstream.

With a single-agent prompt, bad instructions produce a bad response and you see it immediately. With a workflow, bad instructions get multiplied across ten Haiku agents running in parallel, fed into a Sonnet synthesizer, passed to an adversary, and looped. The error compounds before you ever see output. The prompt matters more, not less.

The workflow is only as good as its evaluation criteria.

Initially i had some issues not being specific with evaluation criteria for Haiku (which it then “reinterpreted”). What Claude Code flagged was not a structural problem — the pipeline logic was correct. It was a specification problem: the agents (Haiku) had no precise definition of what constitutes a gap. Without that, every agent applies its own judgment independently and inconsistently. The guard rails I added (three gap conditions, explicit time limits, adversary dismissal rule) are not workflow logic — they are a rubric. The same thing when I hand-crafted for the ASCRS gold answer and the α metric. The mechanism is different but the discipline is the same.

Automation changes when you catch errors, not whether you catch them.

This is why running stages individually the first time is genuinely important — not just for token management but because it surfaces specification gaps at Stage 1 before they propagate to Stage 5. Once you have confirmed the evaluation criteria work correctly at Stage 1, the consolidated prompt is safe to use repeatedly.

The broader point and this is so critical: dynamic workflows are a force multiplier on whatever the prompt says. Precise prompting produces precise automation. Vague prompting produces vague automation at scale, which is harder to diagnose than vague output from a single agent. The skill shifts from crafting one good response to crafting one good specification — which is a higher-order version of the same discipline.

9.1 How Each Prior Approach Maps to the Audit

9.2 The Core Trade-off: Who Coordinates

The capability difference between prior methods and dynamic workflows is not large for a 10-control case study. The coordination difference is significant at any scale. These are questions you, the reader, consider before you apply Dynamic Workflows. I have found them useful:

9.3 Practical Recommendation

For a 10-control checklist with known gaps, a well-crafted single prompt in Claude.ai (the H2 approach) would find the gaps adequately. The workflow earns its complexity at three thresholds. Take this with a pinch of salt. Different tasks, circumstances and applications to various domains apply:

The ASCRS result (H2 defeating H9) still holds here: for a greenfield, one-time task with few inputs, a precise prompt beats a multi-agent workflow. Workflows pay for themselves when the task recurs, scales, or requires structural independence between agents to prevent bias.

Final Thoughts - Evaluation criteria must be clear!

A high level of dicipline is required when using workflows. There are four things worth building into your general practice.

1. A rubric before you run

Before writing the workflow prompt, write down explicitly:

• What counts as a pass

• What counts as a fail

• What the edge cases are and which way they resolve

This is exactly what I built for ASCRS (gold_answer.md, deliberate traps, three-tier confidence). The same discipline applies to workflow prompts. If you cannot write the rubric in plain language before running, the agents cannot apply it consistently during the run.

ASCRS equivalent        Workflow equivalent
───────────────────────────────────────────
gold_answer.md          Evaluation rules in Stage 1
Deliberate traps        Edge cases called out explicitly
α scoring rubric        Adversary dismissal criteria
Expected H1 alpha       Token budget with loop guard

2. A pre-flight checklist for any workflow prompt

Before running, ask these five questions:

+--------------------------------------------------+
| 1. INPUTS    Are all input files named            |
|              explicitly? Does the prompt tell     |
|              agents exactly where to read from?   |
+--------------------------------------------------+
| 2. OUTPUTS   Are all output paths explicit?       |
|              reports/ not just a filename.        |
+--------------------------------------------------+
| 3. CRITERIA  Is pass/fail defined precisely?      |
|              Could two agents reading the same    |
|              file reach different conclusions?    |
+--------------------------------------------------+
| 4. EDGE CASES Are time limits, thresholds, and    |
|              special conditions stated? Or left   |
|              to agent judgment?                   |
+--------------------------------------------------+
| 5. FAILURE   What happens if a stage fails or     |
|              the loop does not converge?          |
|              Is there a fallback and a cap?       |
+--------------------------------------------------+

If any answer is “the agent will figure it out” — that is the gap to fix before running.

3. A dry run on one unit before fanning out

Before running the full fan-out across all ten controls, run one agent against one control manually and inspect the JSON output. Confirm:

• The status field is populated correctly

• The gap field says what you expect

• The evidenceFile field points to a real file

One agent, one control, one read. If that output is wrong, fix the evaluation rules before multiplying across all ten. This costs almost nothing and catches specification errors before they compound.

  Test one unit first
        |
        v
  +------------------+
  | Agent: C3 only   |  <- inspect output manually
  +------------------+
        |
   Output correct?
        |
   Yes --> run full fan-out
   No  --> fix evaluation rules, re-test C3

4. A prompt template for future workflow projects

This is the reusable structure that forces completeness. I keep it as a file in my projects folder:

WORKFLOW PROMPT TEMPLATE
────────────────────────
ultracode: read CLAUDE.md for output rules.

INPUTS:
  [list every file the workflow reads]

PIPELINE:
  Stage 1: [what it does]
    Evaluation rules:
    - Pass condition: [explicit]
    - Fail condition: [explicit, with all edge cases]
    - Do not: [what agents must not invent]

  Stage 2: [what it does]
    Dismissal rules:
    - Dismiss if: [explicit condition]
    - Confirm if: [explicit condition]
    - Do not: [what the adversary must not invent]

  Stage 3-N: [downstream stages]

OUTPUTS:
  [list every file the workflow writes, with full path]

CONTROLS:
  Token budget: [number]k
  Loop maximum: [N] passes
  Fallback: if not converged after [N] passes, [action]

Save workflow as [name].js

The underlying principle across all four is the same one the ASCRS work established: the intelligence is in the structure, not the execution. A workflow with vague criteria delegates judgment to agents that have no shared context and no consistent standard. A workflow with precise criteria turns agents into reliable executors of a well-defined specification. The prompt is the specification. Writing it is the skilled work.

A lot of good advice is found in the Anthropic Blog on similar.

References

• Anthropic (2026). “A harness for every task: dynamic workflows in Claude Code.” Thariq Shihipar & Sid Bidasaria. June 2, 2026.

• Anthropic (2026). Dynamic Workflows reference documentation.

• “The Geometry of Unpredictability.” Harness Engineering Series — SkillOpt integration, MCP token economics, failure taxonomy.

• “The Structure Is the Intelligence.” StockPilot agent decomposition, CMA Cycles 0–4, 97% token reduction result.

• ASCRS Harness Lab. H1–H10 architecture evaluation, α/κ metrics, H2 vs H9 result (Hormuz Strait domain).

• “The Invisible Codebase.” Reversa five-agent COBOL-to-Python pipeline.

• Harness Engineering Parts I & II. H1–H10 progressive harness experiments, E1–E11 token efficiency series.

In early 2026, a single Anthropic blog post about Claude Code and COBOL erased nearly $40 billion from IBM’s market capitalisation in a day (Note: IBM shares not only fully recovered from its $40 billion market cap drop, but the stock recently reached all-time highs - some of it better understood by reading through and understanding the experiment I ran). That reaction was not panic — it was the market correctly pricing a structural shift. This article surveys the landscape that shift created: what IBM, Amazon, and specialist firms are actually deploying at enterprise scale; whether Claude Code can perform meaningful migration without a structured framework; where Reversa (an open-source methodology released May 2026) fits in the ecosystem; and what a practitioner might learn by building and running a five-agent migration pipeline on their own machine. The central argument is that the tool matters far less than the methodology — specifically, the distinction between syntactic code translation and operational contract reconstruction, and the confidence tagging system that makes human oversight tractable rather than exhausting.

Note: The experiment relies on assumptions of a hypothetical but working code base written in COBOL. Tools of analysis were selected based on the requirements of the case study. Different models, harnesses and agentic pipelines can produce varying results, which depend on the methodologies applied herein.

The Post That Moved Markets

In February 2026, Anthropic published a post describing how Claude Code could automate the exploration and analysis of COBOL codebases. The announcement was technically measured — careful about what AI could and could not do. The market was less measured. IBM’s stock fell thirteen percent in a single session, its worst single-day loss since 2000. The trigger was a blog post about a 67-year-old programming language.

To understand why, you need to understand what COBOL represents to IBM’s business model. Approximately 84 percent of IBM’s Z mainframe clients are running COBOL applications. Those clients pay substantial ongoing fees for mainframe hardware, software licences, and specialised support. If AI could meaningfully accelerate the migration of those workloads off mainframe and onto commodity cloud infrastructure, the revenue implications for IBM were severe. The market priced that risk immediately.

Whether the market’s reaction was proportionate is a separate question. What it confirmed, beyond any industry analyst’s forecast, is that AI-assisted legacy migration has crossed from speculative to credible in the minds of the people who allocate capital. That is the moment this article is written inside.

The market did not react to a technology demonstration. It reacted to a plausible threat to a decades-old business model. Those are different things, and the distinction matters for how you think about this space.

What Is Actually Being Migrated — and Why It Is Hard

The scale figures are well known: an estimated 250 billion lines of COBOL in production worldwide, processing roughly 95 percent of ATM transactions and 80 percent of in-person credit card transactions daily. What is less often articulated is why migrating this code is structurally different from migrating any other software.

The answer is that a COBOL program written in 1985 and maintained through 2026 contains two codebases that are technically inseparable but conceptually distinct.

The Visible Codebase

Source files, database schemas, JCL job scripts, copybooks. Everything a developer or AI agent can read. This is what most migration tools operate on.

The Invisible Codebase

The accumulated decisions, regulatory adaptations, and implicit business rules that were never formally documented. The $9,500 threshold that was set in 1994 in response to a specific FinCEN guidance letter and exists nowhere except as a literal constant in a PROCEDURE DIVISION paragraph. The leap-year interest divisor that differs between account types for a reason that retired with the person who coded it. The DTI ceiling that was adjusted after a 2008 OCC examination finding and has never appeared in any policy document since.

Any migration approach that ignores the invisible codebase produces code that passes all its tests in a UAT environment and fails in production when it encounters the exact condition the invisible rule was written to handle. This has happened, repeatedly and expensively, at organisations that attempted big-bang migrations using earlier-generation automated tools.

THE CORE INSIGHT

This is why the distinction between code translation and operational contract reconstruction is not academic. Code translation reads the visible codebase and rewrites it in another language. Operational contract reconstruction reads both codebases — the visible one from the source, the invisible one from patterns, comments, regulatory citations, and structured inference — and produces a specification that an agent can safely use as a migration target.

The Institutional Landscape: What Enterprises Are Actually Using

Three distinct tiers have emerged, with very different economics, target audiences, and architectural philosophies.

Tier 1 — IBM Watsonx Code Assistant for Z

IBM’s response to the migration moment is a dedicated enterprise product: Watsonx Code Assistant for Z. The architecture is revealing. IBM trained a 20-billion parameter Granite foundation model on CodeNet — one of the largest code datasets available — supplemented by enterprise COBOL specifically to ensure the model understands the language’s idiosyncrasies rather than treating it as a dialect of something more common.

The target migration is COBOL to Java, not to Python or any other language. This is a deliberate choice: Java on IBM Z maintains the performance and transactional characteristics of the mainframe runtime while moving the code to a language with a large modern developer workforce. IBM is not trying to move workloads off mainframe — it is trying to modernise workloads while keeping them on mainframe. The business model preservation is explicit.

Version 2.8, released March 2026, introduced an agentic workflow using MCP (Model Context Protocol) to connect the AI to mainframe-specific data sources — job control metadata, copybook libraries, VSAM data definitions. The result is a system with genuine enterprise data governance: code never leaves the client’s environment if they use the on-premise deployment option, which matters considerably to banks that are legally restricted from transmitting certain data to external cloud services.

Pricing is enterprise. Deployment requires IBM Z hardware and software stack. The addressable audience is a few hundred institutions globally with the largest COBOL estates.

Tier 2 — AWS Transform and the Claude Code Partnership

Amazon’s approach is architecturally the most instructive for understanding what actually works at scale, because AWS has been unusually transparent about why it made the architectural choices it did.

AWS Transform handles reverse engineering. It deploys specialised deterministic agents that scan entire codebases — potentially millions of lines of COBOL, PL/I, and JCL — extracting structured business rules, dependency graphs, data lineage maps, and domain decompositions. The keyword is deterministic: AWS’s documentation explicitly states that this extraction uses purpose-built tools rather than probabilistic models, because probabilistic models at this stage introduce the kind of uncertainty that makes the downstream migration untrustworthy.

For the forward engineering phase — taking those extracted specifications and generating modern code — AWS recommends Claude Code. Their own published workflow blog post states this directly: AWS Transform produces the reliable foundation, Claude Code does the generation. The separation is the architecture.

THE AWS INSIGHT

“This is not a task that you can solve by pointing a general-purpose AI Agent at a repository.” AWS’s own documentation on its enterprise mainframe service. The deterministic extraction layer is not a nice-to-have — it is what makes the generative layer trustworthy.

Transamerica, Allianz, and Marriott Hotels are among AWS’s referenced enterprise modernisation clients. The service is cloud-hosted, not on-premise by default, which creates friction for the most heavily regulated banking institutions. AWS addressed this partly by acquiring Blu Age, a French automated refactoring company that handles automated COBOL-to-Java conversion for clients who want deterministic transformation rather than LLM-generated output.

Tier 3 — Specialist Firms and Proprietary Methodologies

A third tier of specialist companies approaches the problem from a fundamentally different angle. One illuminating example is Phase Change Software, whose product COBOL Colleague uses graph-based deterministic analysis to establish verified facts about COBOL programs before any LLM is involved. Their characterisation of Claude Code’s role is precise: “The LLM translates our verified facts into fluent prose. While Claude Code struggles to present the facts, it excels at narrating them.”

The consulting firms — Accenture, Deloitte, IBM Consulting — operate at the intersection of all three tiers, typically deploying combinations of vendor tools alongside proprietary frameworks developed through years of actual migration engagements. Their differentiator is not the AI; it is the pattern library of edge cases, regulatory nuances, and failure modes accumulated from real migrations.

The Uncomfortable Reality Most Skip

Most large financial institutions are not migrating their COBOL. They are running it, carefully, and managing the risk of the aging workforce through knowledge transfer programmes, documentation initiatives, and strategic hiring of the few remaining COBOL-fluent developers. The migration momentum exists at second-tier institutions and government agencies, not at the top twenty global banks, most of which have attempted and partially failed major migration efforts in the past two decades and carry institutional scar tissue from those experiences.

AI-assisted migration changes the economics and the risk profile of attempting it. It does not eliminate the need for judgment, the risk of missing an invisible rule, or the regulatory requirement to prove behavioral equivalence before cutover. Those constraints remain structural regardless of which tool sits in the pipeline.

Can Claude Do It Without a Structured Framework?

This is the question practitioners naturally ask, and it deserves a direct answer rather than a diplomatic one.

Yes, with important caveats. Claude, prompted carefully, can read a COBOL program, identify its business rules, generate idiomatic Python, and write tests that verify the output. For a clean, well-commented COBOL program with regulatory citations in comments and consistent naming conventions, the results are genuinely useful on a first pass.

The TechChannel evaluation team tested Claude Code on real Medicare payment COBOL — not the kind of clean example used in tutorials — and documented specific failure modes: dropped conditional branches, compressed algorithms that merged adjacent logic, and incorrect regulatory constants. Their conclusion was blunt: in a financial or healthcare system, the difference between “mostly correct” and “correct” is the difference between compliance and liability.

Three things that a structured framework adds, which raw prompting does not:

The AWS architecture described above encodes exactly this lesson at enterprise scale. The structured extraction layer produces what raw prompting cannot: verifiable, traceable, confidence-tagged foundations that a generative model can safely build on.

Where Reversa Fits: Honest Assessment

Reversa, released in May 2026 by Macedo and da Costa, is the open-source instantiation of the same methodology that IBM and AWS apply at enterprise scale. It is not competing with Watsonx Code Assistant for Z. It is making the methodology accessible to practitioners who do not have a mainframe contract or a seven-figure modernisation budget.

What Reversa contributes technically is a structured coordination layer for Claude Code: the five-phase Discovery pipeline (Scout, Archaeologist, Detective, Architect, Writer), the confidence tagging system, the SDD (Software Design Document) output format, and the slash command interface (/reversa, /reversa-migrate, /reversa-forward) that provides phase discipline and checkpoint resumption. The AI capability underneath is Claude. The value Reversa adds is the methodology wrapping.

To be honest about what this means: a skilled prompt engineer with enough time could replicate Reversa’s core functionality. The project’s genuine contribution is that it has done that work, made it publicly available, and embedded in it the methodological discipline — particularly the confidence tagging and GAP system — that most ad hoc approaches omit.

Where Reversa Is Strong

• Non-mainframe legacy: VB6, RPG/AS400, older Java EE, PL/I, FORTRAN — the long tail of enterprise legacy that IBM’s mainframe-specific tools do not address.

• Practitioner scale: A single developer or small team working on a bounded legacy system. The overhead of an enterprise migration platform is unnecessary and counterproductive at this scale.

• Learning and research: Understanding the methodology, experimenting with different COBOL programs, building institutional knowledge before committing to a production migration.

• The forward evolution use case: The /reversa-forward command, which propagates regulatory or policy changes through an already-migrated system, is particularly well-suited to insurance policy administration where rules change continuously.

Where Reversa Has Limits

• Million-line mainframe codebases: Enterprise COBOL estates are too large for Claude Code’s context window without the deterministic scaffolding that AWS Transform or IBM’s ADDI tool provides.

• Regulatory data governance: A bank subject to strict data residency rules cannot send production COBOL to a cloud-hosted Claude model. IBM’s on-premise option exists specifically for this constraint.

• The invisible codebase still needs humans: No tool, including Reversa, can answer what a threshold means if it is undocumented. GAP resolution requires domain expertise regardless of how well the framework surfaces the GAPs.

Custom Pipeline vs Reversa

What Reversa Is Trying to Do

So I decided to run my own pipeline: COBOL → Python. Reversa and my simple pipeline solve different problems, which is why both can be working solutions.

Reversa asks: “What does this legacy system actually do?” — It reverse-engineers the system into a specification document that humans and AI coding agents can use as a blueprint going forward. The output is knowledge, not code.

This pipeline asks: “Can we run this COBOL as Python today?” — It goes straight to a working translation, tests it, and scores it. The output is runnable code with a verdict.

Why Both Achieve Their Objectives

Reversa succeeds because it doesn’t try to produce code — it produces understanding. By making humans approve each phase, it catches the INFERRED and GAP items before they become bugs in production. It is more conservative and more thorough for complex, undocumented systems where you cannot afford to guess.

This pipeline succeeds because it narrows the problem. By committing to one target language (Python) and one output (runnable code), it can automate the full loop — translate, test, score — in under 3 minutes. The tradeoff is that it will confidently produce something wrong if the COBOL had undocumented assumptions.

The Honest Observation

Both projects use the exact same CONFIRMED / INFERRED / GAP confidence model. That convergence is not a coincidence — it is the fundamental truth of legacy migration: some things are knowable from the code, some things are educated guesses, and some things require a human who was there. No amount of AI sophistication eliminates that third category.

Reversa builds its entire workflow around that fact and refuses to proceed without human sign-off. This pipeline surfaces it in the final report and issues a BLOCKED verdict when too many GAPs exist. Different mechanisms, same underlying problem.

If you are migrating a payroll system or an AML engine that has been running in production for 30 years, Reversa’s approach — slow, documented, human-approved at every step — is the safer choice. If you want a quick first-pass translation to understand the scope of effort, this pipeline gets you there faster. Albeit not perfectly, but with a bit of fine-tuning, and sit-down sessions - all doable.

Building It Yourself: What the Custom Pipeline Reveals

A five-agent pipeline built in the companion experiment to this article (only results shared here) took a different architectural approach from Reversa: standalone Python running via OpenRouter rather than Claude Code slash commands. This was a deliberate choice, and the difference is instructive.

Each stage in the pipeline is a separate AI agent — a distinct model with its own system prompt and specific job. Because this was a “lab experiment” for me, and given fairly straightforward, uncomplicated code, earlier model versions were applied. Here's what each one did on a sample AML-TEST.cbl run:

Each agent only sees what it needs — the Translator never sees the test code, the Evaluator never touches the COBOL. They pass work forward like an assembly line, with the orchestrator (orchestrator.py) coordinating the handoffs.

The faster/cheaper Gemini models handle the mechanical work (write tests, run tests). The more capable Claude models handle the reasoning-heavy work (understand COBOL intent, judge migration quality). That’s the cost/quality tradeoff baked into the design.

Running the pipeline on an AML transaction screening program produced a complete extraction, idiomatic Python with Decimal arithmetic throughout, a 33-test pytest suite, and an evaluation report scoring 77.1/100 with a CONDITIONAL verdict.

All 5 failures trace to the same root cause: $10,000 is a round number divisible by 1,000, so the round dollar rule fires on every CTR-level cash transaction. The tests expected those to be treated as separate, independent events — but the generated Python doesn't distinguish them. Fix the round dollar rule's logic and tests 3, 5, 6, and 18 likely resolve automatically. Test 33 fails separately because the dormant activity rule doesn't fire in the combined scenario — that needs a separate look at migrated_module.py.

The Honest Assessment for Leadership

AI did not “convert COBOL to Python.” What it did was:

• Draft a working translation that gets you most of the way there — faster than a team starting from scratch

• Surface the gaps — the areas where the COBOL was relying on undocumented assumptions, external systems, or institutional memory. Without this tool, a developer might not even know those gaps existed until something failed in production

• Write and run a test suite that identified real defects before anyone deployed anything

What it could not do — and no AI currently can do — is:

• Validate that inferred business rules match your compliance policy

• Connect to your sanctions screening system (not in this test)

• Decide when to file a SAR

• Vouch for regulatory correctness

The analogy: think of this like a very fast, very thorough junior analyst who read the old system overnight, wrote up a detailed summary, drafted new code, and flagged everything they were uncertain about. You still need a senior compliance officer and an engineer to review the flagged items before this goes anywhere near production.

Recommended Next Steps

1. Compliance review of the 6 inferred rules — specifically the $8,500 structuring threshold. Is that your institution’s policy, or did the AI make it up?

2. Engineering fixes for the 5 failing tests — these are known, specific defects and should be straightforward to correct once the correct logic is confirmed

3. Integration planning for the 4 GAPs — OFAC, aggregation, SAR filing, and audit logging each require a conversation with the teams that own those systems

4. Do not go live until the score reaches READY (≥ 80) with zero blocking GAPs. The current state is a solid foundation, not a finished product.

The pipeline saved weeks of initial translation work. It did not save the compliance validation, integration work, or human judgment that AML systems legally require. That was never going to be automated away.

Code is now the cheap part of legacy migration. Architecture judgment and domain knowledge are the expensive parts. That was always true. AI makes it undeniable.

The Methodology Is the Point

At every tier of the market — IBM’s enterprise Granite model, AWS’s deterministic Transform agents, Reversa’s open-source pipeline, the custom five-agent Python implementation — the same architectural decision appears: separate the extraction of what the code knows from the generation of what the new code should say. The tools differ enormously. The methodology is consistent.

Three elements of that methodology deserve emphasis because they are what makes the difference between a migration that produces useful output and one that produces confident-looking output that fails when it encounters an edge case:

Operational Contracts, Not Documentation

The output of the extraction phase should be machine-executable specifications — inputs, outputs, invariants, regulatory constraints, edge cases — not human-readable documentation. Documentation is for people. Specifications are for agents. The difference is that a specification can be tested against. Documentation cannot.

The Confidence Tagging System

Every extracted claim should carry one of three marks: CONFIRMED (directly readable from code with a source citation), INFERRED (deduced from pattern with a stated rationale), or GAP (not determinable from code, requires human input). This is not administrative overhead. It is the mechanism that makes human oversight tractable. A migration with 47 GAPs to review is manageable. A migration with no explicit GAPs and hidden assumptions is not.

Behavioral Equivalence as the Standard, Not Code Correctness

The question is not whether the generated Python compiles and passes unit tests. It is whether the Python produces the same outputs as the COBOL for every input the COBOL ever encountered. Gherkin parity specs derived from the extracted contracts — not from the generated code — are the appropriate test artefact for this standard. This is what makes a Parallel Run viable as a regulatory validation mechanism: both systems run simultaneously against real transactions, and every divergence is documented and investigated.

Verdict: Was This Worth My Time & Tokens?

The practitioner’s question is blunt: given that IBM has a purpose-built enterprise tool, AWS has a cloud service with Fortune 500 references, and Anthropic itself publishes guidance on using Claude Code directly for COBOL analysis, is there genuine value in understanding and running an open-source framework or custom pipeline?

The answer depends on what you are optimising for.

If you are evaluating migration tooling for an institution

The institutional tier tools (IBM, AWS) have genuine advantages for large mainframe estates: security, data governance, enterprise support, and provenance from organisations with skin in the game. For a bank with ten million lines of COBOL running on IBM Z hardware, Watsonx Code Assistant for Z is the serious option. For a mid-size insurer wanting to modernise a policy administration system to cloud, AWS Transform with Claude Code forward engineering is the serious option. Neither requires you to understand Reversa or build your own pipeline.

If you are a practitioner-researcher, consultant, or reader

The methodological insight — structured extraction, confidence tagging, operational contracts, GAP-driven human oversight — is what you need to understand regardless of which tool sits on top of it. Building and running the custom pipeline teaches you this in a way that reading about IBM’s product does not. The experiments in this series used a simple AML screening program rather than a million-line mainframe system, but the methodology scales. The same principles govern both.

Reversa is worth your time as a learning and research instrument. It is also worth writing about, not because it will replace IBM Watsonx Code Assistant for Z, but because it makes the methodology accessible and demonstrates that the core problem — extracting the invisible codebase, tagging it honestly, and handing the gaps to a human — is not a capability reserved soley for enterprises with mainframe contracts.

The Question That Remains Open

The AI-assisted migration moment is real.

The tooling is advancing rapidly at every tier. What has not changed, and what no advance in model capability will change, is the epistemological limit: an AI cannot know what a threshold means if no one wrote it down. The compliance officer who can answer that question in a fifteen-minute meeting is not being replaced by the migration pipeline. She is being given a much shorter, much better-organised list of questions to answer. That is not a small thing.

Appendix: The Tier Landscape at a Glance

References

[1] Anthropic. How AI Helps Break the Cost Barrier to COBOL Modernization Anthropic Blog, February 23, 2026.

[2] Economic Times India / Reuters. IBM Shares Hit 13% Drop, February 24, 2026.

[3] Slashdot. IBM Shares Crater 13% After Anthropic Says Claude Code Can Tackle COBOL Modernization Slashdot, February 23, 2026.

[4] PYMNTS. Anthropic’s COBOL Bet Shakes Mainframe Economics PYMNTS.com, February 24, 2026.

[5] IBM. Agentic AI for Smarter Mainframe Modernization with IBM Watsonx Code Assistant for Z IBM Newsroom, March 2, 2026.

[6] IBM Research. COBOL to Java: Application Modernization with IBM Generative AI IBM Research Blog, 2025.

[7] VentureBeat. IBM Taps Watsonx Generative AI to Help Modernize COBOL on Mainframes VentureBeat, December 22, 2025.

[8] AWS. Reimagining Mainframe Applications with AWS Transform and Claude Code AWS Migration and Modernization Blog, April–May 2026.

[9] AWS. AWS Mainframe Modernization Service — Features AWS Documentation, 2026.

[10] AWS Migration and Modernization Blog. AWS Named a Leader in the ISG Provider Lens Mainframe Application Modernization Software 2025 Report AWS Blog, 2025.

[11] TechChannel. Can Claude Code Really Understand COBOL Applications? TechChannel, March 24, 2026.

[12] Phase Change Software. Anthropic Says Claude Code Can Analyze COBOL. Here’s Why Analysis Isn’t Proof. Phase Change Software Blog, February 25, 2026.

[13] MGM Technology Partners. Claude Can Migrate COBOL — Who’s Making the Transition? MGM Insights, March 5, 2026.

[14] Macedo, S.O. and da Costa, R.M.. Reversa: A Reverse Documentation Engineering Framework for Converting Legacy Software into Operational Specifications for AI Agents arXiv:2605.18684, May 18, 2026.

[15] Macedo, S.O. and da Costa, R.M.. Reversa — GitHub Repository (sandeco/reversa) GitHub, 2026.

[16] The New Stack. COBOL Is Everywhere. Who Will Maintain It? The New Stack, 2024.

[17] IT Brew. Can COBOL’ers Collab with Claude Code? IT Brew, February 26, 2026.

Why this article exists - A Consolidation of Findings and Tiny Experiments

This will be a consolidation article that takes a perspective or stand on the few recent articles and experiments run with Claude/Claude Code. The experiments were taken from similar actual cases, and then task-oriented down for case study orientation and discussion.

They originally stemmed from a set of concerns that circulate regularly among practitioners and non-practioners alike, people who asked, even friends:

(1) that the product is drifting toward developers,

(2) that native memory is unreliable,

(3) that complex agent systems produce better results,

(4) that vendor lock-in is an inevitable consequence of building on the platform,

(5) that API choice does not matter, and

(6)that automation requires infrastructure that most business users cannot manage.

These concerns are not invented. They reflect real experiences. But across a series of controlled experiments run recently -- with real measured outputs, gold-standard answers, and scored rubrics -- a consistent pattern emerged: the concerns are usually accurate observations about default behaviour, and inaccurate conclusions about what that behaviour means.

All experiments focus on Anthropic’s products, as this is my preference for executing at the Institutional Grade level. That does not mean I do not use other harnesses, models or service providers - because the space evolves fast, and there is always something new to experiment on. But my preferences are strong for Claude.

This article is the retrospective. It pulls the key findings together in one place, addresses each concern directly, and links to the full article for readers who want the detail. The argument is not that Claude is without limitations. It is that:

Most of the limitations practitioners describe are architectural, and architecture is something the practitioner controls.

Before the Findings: One Diagram

Every finding in this series connects to the same underlying observation. Claude is not a single fixed thing. It is a stack of layers -- and most concerns about it are concerns about one specific layer, misidentified as a concern about the whole.

A memory failure is almost always a Layer 2 problem -- the wrong context was loaded, or none was. An infrastructure complaint is almost always a Layer 3 choice, not a Layer 0 limitation. A cost problem is almost always a Layer 1 or Layer 2 decision. None of the six concerns addressed in this series turned out to be a Layer 0 problem.

Note:

ISR — Intelligence Systems Review. The name I used in the publication series.

ASCRS — AI Supply Chain Response System. The pharmaceutical supply chain case study domain I built as the running experiment environment throughout the series — the Hormuz Strait scenario, the 23 purchase orders, the carrier data, the rubric and gold answer that underpinned H1 through H10 and the memory experiment.

1. If I Build on Claude, Am I Locked In?

Concern: Vendor lock-in and lack of portability

Building around Claude-specific conventions -- CLAUDE.md, Claude Code’s project structure, Anthropic’s hosted environments -- creates dependency. Moving to a different tool would mean rebuilding everything.

The finding

Lock-in is real when your intelligence lives inside a platform. It does not exist when it lives in files you own.

Every ISR experiment was built on plain markdown files: a MASTER_GUIDE containing domain knowledge and experimental history, a BOOTSTRAP_PROMPT containing session-start context, and skill files containing reusable workflow recipes. All of these are standard text. All are readable by any AI tool. The naming convention CLAUDE.md is a convention -- not a format. Changing the filename to AGENTS.md (the Forge convention) takes thirty seconds. The content does not change. The intelligence does not move.

Three cases where lock-in is a genuine risk, not a hypothetical: Claude.ai’s built-in memory summaries (stored in Anthropic’s system, not in your files); Managed Agents hosted environments (the execution environment does not export); and MCP server connections (configured per-tool, not universally portable). These are specific, nameable tradeoffs -- not a general trap.

There is also an efficiency dimension to this. Every token in CLAUDE.md is re-sent on every Claude Code request. Independent benchmarking shows that trimming a context file from 3,847 tokens to 312 tokens -- keeping only rules the model acts on -- reduces per-session cost by 91.9%. The discipline that prevents lock-in and the discipline that reduces cost are the same discipline: keep your intelligence in lean, portable, plain-text files you own.

The rule:

Article Ref Start: The Architecture of Awareness

2. Does Architectural Complexity Produce Better Results?

Concern: Technical drift toward developers / Complex setups outperform simple ones

Two concerns that travel together.

The first: Claude’s product is growing more complex, drifting toward developers and away from business users.

The second: if you invest in building a complex multi-agent system, it must outperform simpler approaches.

The finding

Ten different architectures were tested against the exact same task using the exact same model. A pharmaceutical supply chain crisis. 23 purchase orders, three priority tiers, carrier availability constraints, and four deliberate reasoning traps. A gold-standard answer prepared in advance. A six-criterion rubric. Every architecture was scored against it.

H2 -- a single, well-structured prompt with explicit role, output schema, scoring weights, and anti-hallucination rules -- achieved a perfect score at 15,277 tokens. H9 -- a five-agent swarm with specialised roles for routing, compliance, inventory, commercial analysis, and review -- scored below the bare model at nearly four times the cost.

H9 was predicted to win before the experiment ran. The reasoning was intuitive: five specialised agents coordinating toward a shared answer should outperform a single prompt. What happened instead: each agent produced a partial answer, the coordination layer had to synthesise five partials, and synthesis introduced ambiguity that was not present when a single prompt held the full picture.

H7 -- model routing, directing simpler steps to Haiku at 3.75x lower cost than Sonnet -- achieved 0.900 alpha. This is the result that matters most for practitioners concerned about cost: routing decisions alone recover most of the performance of a complex system at a fraction of the price.

On the technical drift concern: the sophisticated options in Claude’s ecosystem exist for tasks that genuinely need them. This experiment however, showed that they do not need to be the default path. The highest-scoring result used no agents, no infrastructure, and no special features -- just a precise prompt/skill.

Full article: The Harness Experiment

Full article: ASCRS Harness Lab -- The Integrated Agentic Stack

Full article: The Prompt Is Not the Architecture

3. Does or Can Decomposing an Agent Actually Reduce Cost?

Concern: Multi-agent systems are expensive and require complex infrastructure

The harness experiment demonstrated that a single precise prompt outperforms complex architectures on analysis tasks. A separate experiment tested the other direction: what happens to a monolithic agent when you systematically decompose it?

The finding

Anthropic’s StockPilot stock forecasting agent -- from their public workshop repository -- was run through a structured decomposition process across five cycles in Claude Code. Real API costs were measured at every step using Anthropic’s own evaluation suite.

97% token reduction. Accuracy improved from 71% to 92%. The model did not change. The task did not change. Only what each call was asked to carry changed.

A monolithic agent accumulates context it does not need on every call. A system prompt listing 40 tools for a task that uses 3 is paying for 37 unused tools on every single invocation. Decomposition removes the accumulation. Each component receives only what it needs for its specific function. The intelligence does not change. The waste does.

One live discovery during the run: when routing was switched from OpenRouter to Anthropic direct, the score dropped from 92% to 75% with no error message. One sub-agent function had a hardcoded API client that did not update with the environment variable. The function silently failed. The model was fine. The integration layer had a hidden assumption. Finding it required knowing what had changed between runs -- which is the argument for structured, documented experiments over ad hoc iteration.

Full article: The Structure Is The Intelligence

Full article: Every Company Is An Agent Waiting to Be Decomposed

Full article: The Token Tax

4. Does the API Route Matter?

Concern: Integration choices have no meaningful impact on cost or behaviour

The model is the same regardless of which API provider you use to reach it. The route is just a technical detail.

The finding

During the StockPilot experiment, prompt caching was configured correctly -- cache_control markers included in the system prompt. The experiment ran through OpenRouter. The savings did not appear.

The investigation found that OpenRouter rotates requests across multiple infrastructure providers -- Anthropic direct, Amazon Bedrock, Google Vertex -- automatically. A prompt cached on one provider is not cached on another. Each rotation resets the cache. The instruction was correctly written and silently ineffective. No error. No warning. The fix is one line: set allow_fallbacks: false in the OpenRouter request, pinning all requests to a single provider.

A second finding: the experiment’s logging captured input_tokens only -- not cache_read_input_tokens or cache_creation_input_tokens. Those are separate fields in the Anthropic API response. If your logging does not read them, you cannot confirm caching is working even when it is. Understanding what your instrumentation captures is as important as understanding what the API reports.

What independent benchmarking shows about stacking

The caching discovery is one finding from one integration path. Independent benchmarking by Hamza Farooq (claude-sonnet-4-6, May 2026, approximately 17,768 requests per month at a $500 base) quantifies four optimisation techniques and what happens when they compound:

The critical insight from the combined result: optimisation techniques are multiplicative, not additive. Prompt caching alone saves 71.5%. Model routing alone saves 77.1%. Stacked with output budgeting and multi-turn caching, the combined saving reaches 89.3% -- $447 per month on a $500 base. Each layer cuts what remains after the previous layer.

The ISR experiments provide direct evidence for two of the four techniques: the caching discovery (prompt caching) and H7’s model routing result (0.900 alpha at materially lower cost). The multi-turn caching and output budgeting figures are from Farooq’s independent benchmarks and have not been replicated in ISR experiments. They are noted as externally evidenced, not internally verified.

Practical sequence -- four checks, in order of effort

Full article: The Structure Is The Intelligence -- Appendix: Are All APIs Created Equal?

External benchmark: Hamza Farooq -- Claude Token Optimizer -- open-source, claude-sonnet-4-6, May 2026

5. Is Claude’s Memory Actually Poor?

Concern: Native memory is unreliable and requires third-party tools to fix

Claude has no persistent memory between sessions? Everything is forgotten when the conversation ends? Fixing this properly requires third-party databases, vector stores, or external memory services?

A note on the research landscape

Three independent research projects on AI memory were surveyed in preparation for this series: MEMENTO (Microsoft Research, April 2026), MemPalace (April 2026), and AutoResearch (Karpathy, March 2026). I did not complete the analysis as a published synthesis because a direct philosophical conflict was found between the approaches: MemPalace stores everything verbatim, with no AI deciding what to forget; MEMENTO compresses. These positions are structurally incompatible. Publishing a synthesis would have been inaccurate. At least not for now. More testing. And I will come back to this in future.

The honest position: long-term AI memory at the level of complex multi-session knowledge accumulation remains an open and contested research problem. Practitioners should be cautious about any tool claiming to have resolved it cleanly.

The finding

Instead my experiment tested something narrower and more immediately useful: for session-level and task-level memory, does injecting context at session start measurably improve performance against a scored rubric?

The experiment was designed around two conditions: a rich, self-documenting data file versus a stripped raw version, both run without any prompt-level context injection. The finding was not what the original design anticipated.

The gap of 0.175 was concentrated in exactly two criteria: R1 (tier prioritisation weights, 0.5) and R4 (financial aggregation methodology, 0.5). Four criteria held at 1.0 regardless of context, because their answers are derivable from data structure alone. The two that dropped depend on institutional parameters -- a 60/40 weighting schema and a bespoke aggregation method -- that exist only in organisational practice and cannot be inferred from raw data or general reasoning.

This produced a more precise argument than the original design would have. Context engineering operates at two layers: the data structure you pass to the model, and the prompt context you inject at session start. Both are valid. A well-structured data file can substitute for prompt injection on criteria with objective, derivable answers. Prompt injection is essential on criteria requiring institutional knowledge the data file cannot carry.

The practical implication is an audit, not a prescription: identify which criteria in your task depend on institutional knowledge. Ensure your data file or your BOOTSTRAP_PROMPT carries those criteria explicitly. Do not write context that repeats what the data already says -- that is redundant tokens and noise.

Claude’s context window holds up to 200,000 tokens. When context is injected at session start, the model is not retrieving memory from somewhere external. It is carrying it. The architectural cost of a well-written context file is zero. The only cost is not having written it.

Full article: Is Claude’s Memory Actually Poor?

6. Do You Need Infrastructure to Automate?

Concern: Automation requires cloud infrastructure or 24/7 hardware

Running Claude automatically requires either leaving your computer on 24 hours a day or setting up GitHub repositories, cloud containers, and backend infrastructure? Claude’s routines interface uses developer jargon -- autofix pull requests, pushes, merges -- that has nothing to do with how a business owner thinks about getting a task done? Token-maxxing? Token bloat?

Where the concern is correct

If the requirement is for Claude to run a task at 3am with no human present -- that is scheduling. Scheduling requires infrastructure. A local machine in sleep mode will not trigger the run. Cloud infrastructure solves this and it is genuinely complex to configure. That part of the concern is accurate and should not be dismissed. The GitHub jargon in the routines interface is a real friction point for non-developers, and it reflects a product surface that was designed for development teams first.

The distinction that changes the answer

Most business workflows described as routines are not scheduling problems. They are repeatability problems. The difference matters:

Claude Code’s native file system access -- reading files, writing files, running scripts -- covers the majority of repeatable business workflows without any additional infrastructure layer. MCP connections and managed agents exist for specific integration needs: live databases, external APIs, real-time data feeds. They are not prerequisites for agentic work in general.

A controlled experiment demonstrating this distinction -- a complete weekly review workflow, triggered by a single command with no cloud infrastructure -- is forthcoming. The argument above is logical rather than experimental. It will be updated with measured results when available.

It is also worth noting that Anthropic’s own roadmap is moving in this direction. Dreaming -- an asynchronous between-session memory consolidation process that shipped for Claude Managed Agents (CMA) in May 2026 -- represents Anthropic’s recognition that the infrastructure complexity concern is real and worth solving at the product level. The scheduling problem is a genuine constraint. The field is actively working on it.

What the Series Collectively Demonstrates

Across five controlled experiments and multiple architectural explorations, one pattern appeared consistently: the concerns practitioners raise about Claude are usually accurate descriptions of what the default behaviour produces. They are not accurate descriptions of what the architecture can produce.

Default behaviour -- a blank session context, a monolithic agent, an unoptimised routing configuration -- produces exactly the results the critics describe. Poor recall. High cost. Fragile coordination. Genuine friction for non-developers.

Intentional architecture -- a well-structured data file, a lean BOOTSTRAP prompt, a decomposed agent, a correctly configured caching setup, a precise single prompt instead of a swarm -- produces results that invalidate most of those criticisms. The model has not changed in any of these comparisons. What changed was the structure around it.

Three questions that serve as a practical filter before any integration decision:

* Is this a model problem or an architecture problem? If Claude produces a poor result when given clear context and a precise task, that is a model issue. If it produces a poor result without context or structure, that is an architecture issue. Almost every concern in this series turned out to be the second kind.

* What is my portability requirement? Keep your intelligence in plain text files. Not in platform memory systems. Not in hosted environments that cannot export. The portability decision is made at file creation, not at migration time.

* Do I need scheduling, or repeatability? These are different requirements with different solutions. Conflating them produces unnecessary infrastructure and unnecessary frustration.

Claude is a flexible system of choices, not a single product with fixed behaviour. The practitioners who get consistent, high-quality results from it are not the ones who found a workaround. They are the ones who treated architecture as a variable -- and measured what changed when they adjusted it.

For now, enjoy the journey.

Series References

The Architecture of Awareness

V1-V4 agent design, ASCRS system, file architecture

The Harness Experiment

H1-H10 results, ten architectures one task

ASCRS Harness Lab -- The Integrated Agentic Stack

Full rubric, gold answer, scored results

The Prompt Is Not the Architecture

Anthropic Prompting Playbook reconciled with ASCRS findings

Harness Engineering -- Scaffolding A Small Model

Right scaffolding beats upgrading the model

The Token Tax

Harness Engineering Part II: performance to efficiency

The Structure Is The Intelligence

97% token reduction, API caching discovery

Every Company Is An Agent Waiting to Be Decomposed

Graph-of-algorithms framework, decomposition principles

Why Multi-Agent AI Systems Break

Updated framework for failure modes

The Geometry of Unpredictability

Agentic workflow failures and polymorphism analogy

Is Claude’s Memory Actually Poor?

Controlled experiment: data architecture vs prompt injection

Hamza Farooq -- Claude Token Optimizer

Four-technique stacking benchmark, claude-sonnet-4-6, May 2026

A Concern I Sometimes Hear

“Claude has no persistent memory between sessions”. “Everything is forgotten when the conversation ends.” “The native memory tools are unreliable, and fixing this properly requires third-party databases, vector stores, or external memory services.”

I have heard the above said in different ways, many times. Even just yesterday.

This concern is widely held, and it is not entirely wrong. Claude does not natively remember all previous conversations. But Claude features memory capabilities that allow it to reference previous interactions and build context over time. Instead of operating entirely in isolation, you can enable settings that allow the AI to actively search your chat history and retain personal preferences across new sessions

Logically, with code, without any architecture or “harness” around it, each session begins as a blank slate. The question worth testing is whether the conclusion that follows -- that external memory services are required -- is actually correct. For this - Claude Code.

Continue watching the space for memory research…

I did recently look into three independent research projects on AI memory in April 2026: MEMENTO (Microsoft Research), MemPalace, and AutoResearch (Karpathy). The article was not completed as a published synthesis, however, because I started including them in various experiments (which I will write about more comprehensively, at a later date, as the space evolves).

MemPalace is built on the principle that everything should be stored verbatim -- no AI decides what to forget. MEMENTO’s approach is compression-first -- it actively summarises and compresses episodic memory across sessions. These two positions are structurally incompatible. A synthesis presenting them as complementary would have been inaccurate. So I made the decision not to publish (whilst experimenting) rather than publish something misleading or inconsistent.

That conflict is noted here because it is the honest position: long-term AI memory at the level of complex multi-session knowledge accumulation remains an open and contested research problem. Practitioners should be cautious about any tool or article claiming to have resolved it cleanly. For now.

What the experiment below demonstrates is narrower and more immediately useful: for session-level and task-level memory -- the kind needed to do a specific job well -- the answer is more architectural than technical, and the tools are simpler than the concern suggests.

The experiment

The ASCRS pharmaceutical supply chain crisis task has been used throughout this series as a controlled benchmark domain. A Strait of Hormuz closure scenario. 23 purchase orders across three priority tiers. Carrier availability constraints, financial parameters, gate sequencing rules, and four deliberate reasoning traps embedded in the data. A gold-standard answer -- a full CFO-approvable brief -- prepared in advance, against which all outputs are scored using a six-criterion rubric.

This experiment was designed to test a simple question: does injecting context at session start measurably improve performance against a scored rubric? It ran in Claude Code, in separate sessions, in the experiments/memory-ab/ subfolder of my ascrs-harness-lab project.

What the experiment design revealed

The original design called for a direct comparison between a session with no context and a session with BOOTSTRAP_PROMPT.md injected. But in my case both returned alpha = 1.0.

The first instinct was to treat this as a failed experiment. It was not.

It revealed that the original data file -- disruption_context.json -- was already doing the context work. The file contained embedded _agent_trap fields describing each reasoning trap explicitly, and a weighted_median_derivation section laying out the financial methodology. The data file was self-documenting. It was supplying the domain knowledge that BOOTSTRAP was expected to supply.

So I redesigned the experiment. A stripped version of the data file was created -- disruption_context_raw.json -- with the embedded guidance fields removed. Condition A was re-run using the raw file, with no BOOTSTRAP. Condition B remained as run: the original rich data file, no BOOTSTRAP. The variable between the two conditions was data architecture alone.

Results

The gap of 0.175 was not uniform across criteria. Four of six criteria held at 1.0 in Condition A. Two dropped to 0.5.

The pattern however, was precise. R1 and R4 dropped because both depend on domain-specific parameters that exist only in institutional practice: a 60/40 tier weighting schema and a ~340-operator aggregation methodology. Neither can be inferred from raw data. Neither is available through general reasoning. The model handled both correctly in Condition B because the original data file had embedded that methodology explicitly.

R2, R3, R5, and R6 held at 1.0 in both conditions because their answers are derivable from data structure. The carrier routing constraint for PO-2853 was present in the raw data. The gate schema and G4/G7 dependency were present in the raw data. The epistemic uncertainty tiers and trigger derivation were present in the raw data. The model did not need institutional guidance to answer those criteria correctly -- it needed the facts, which the raw file contained.

The rubric measured the following:

What this demonstrates

The gap between Condition A and Condition B is not a gap in model capability. The model did not change between conditions. It is a gap in what the model was given to work with -- specifically, the absence of two institutional parameters that exist only in domain-specific practice.

As a quick note, the way I use Claude’s context window, despite the 1m context window: basically beyond 200,000 tokens -- approximately 150,000 words, I start worrying about the “dumb zone” applying. Want to find out more: Dex Horty covers a lot here.

When context is injected at session start, whether via a rich data file or via BOOTSTRAP_PROMPT.md, the model is not retrieving memory from somewhere external. It is carrying the memory in its active context. Retrieval can fail, decay, or hallucinate under pressure. Injection simply loads. The distinction is architectural, not cosmetic.

The experiment also highlights a choice that most practitioners do not realise they have. Context engineering operates at two layers, not just one:

* The data layer -- how your data file is structured (nevermind the where). A self-documenting data file that embeds domain guidance, methodology notes, and constraint descriptions is doing context work before any prompt is written. The model reads it as part of the task and reasons from it correctly.

* The prompt layer -- what you inject at session start (or add as you test) via BOOTSTRAP or a system prompt. This layer compensates for what the data layer cannot or does not carry -- institutional parameters, weighting schemas, bespoke methodologies, business-specific thresholds. Unless of course these were included within the data layer, in the first place.

Both layers are valid. Both compensate for the absence of the other. The choice between them depends on what you control. What needs to persist, or evolve.

If you control how your data is structured -- your own pipelines, your own files -- you can embed domain guidance directly. The model reads it as data and reasons from it without any additional prompt engineering. If your data comes from an external source you cannot modify -- raw exports, third-party feeds, legacy files with no embedded metadata -- then BOOTSTRAP becomes essential because the data layer is not available to you.

The two-layer audit question

Look at your rubric criteria. For each one, ask:

Can the model answer this correctly from the data file alone?

Yes --> data structure is sufficient for this criterion.

No --> this criterion needs BOOTSTRAP.

Write BOOTSTRAP to cover the gap, not to repeat the data.

Redundant context is noise. Targeted context is leverage.

The architecture in practice

Across every harness experiment in this series, a two-file context architecture has been in consistent use without any third-party memory service:

* MASTER_GUIDE.md -- the long-term reference document. Alsways portable. Full experimental history, domain knowledge, architectural decisions, rubric designs. Updated after every experiment. Read in the Claude.ai chat window for design and analysis work between sessions. Or capture it in CLAUDE.md or AGENTS.md

* BOOTSTRAP_PROMPT.md -- the session-start injection file. Contains only what Claude Code needs for the current task, distilled from MASTER_GUIDE. Loaded at the start of every Claude Code session before any task runs.

Neither file is platform-specific. Both are plain markdown. Both are readable by any AI tool. The pattern is not a Claude workaround -- it is a general principle: your intelligence lives in files you own, not in a platform’s memory system. But it can if you choose.

The experiment adds precision to how BOOTSTRAP should or can be written. The criteria that held at 1.0 in Condition A did not need BOOTSTRAP coverage -- they were handled correctly from data alone. Including them in BOOTSTRAP would have added tokens without adding value. The criteria that dropped -- R1 and R4 -- are exactly what BOOTSTRAP should carry: the 60/40 weighting schema, the aggregation methodology, the institutional parameters the data file cannot provide.

A lean, targeted BOOTSTRAP outperforms a comprehensive one. This connects to a separate finding documented in Article A: every token in CLAUDE.md and BOOTSTRAP is re-sent on every request. Dead weight in those files is a recurring cost, not a one-time investment. Independent benchmarking shows that trimming a context file from 3,847 to 312 tokens -- removing boilerplate and keeping only what the model acts on -- reduces per-session token cost by 91.9%. I did it differently but with the same contextual impact, with The Structure Is Intelligence! The discipline that produces a better experiment also produces a cheaper one.

What this does not solve

My experiment demonstrates session-level context injection. It does not demonstrate long-term autonomous memory accumulation across many sessions without human involvement -- the kind where an agent builds a knowledge base over weeks/months and queries it intelligently. That is a harder problem. At least, for now, most of us have not had that lenght of time to test these very new tools that only just presented themselves in the last quarter of last year. So, time will tell. The research landscape I surveyed offers partial answers but no clean solution, and the philosophical conflict between verbatim storage and compression-first approaches remains unresolved.

The two-file architecture also does not scale indefinitely without maintenance. A MASTER_GUIDE that grows without pruning eventually contains outdated decisions alongside current ones. The discipline is the same as any knowledge management practice: regular review, deliberate pruning, honest updating. The architecture provides the structure. The practitioner provides the upkeep.

So?

The experiment produced an alpha gap of 0.175 between a raw-data-only condition and a self-documenting-data-only condition, concentrated in R1 (tier prioritisation, 0.5) and R4 (financial aggregation methodology, 0.5). Four criteria held at 1.0 regardless of context, because their answers are derivable from data structure alone.

The finding is more specific and more useful than a simple before/after comparison: BOOTSTRAP injection does not improve performance uniformly. It closes the gap precisely on criteria requiring institutional parameters -- weighting schemas, bespoke methodologies, business-specific thresholds -- that neither raw data nor general reasoning can supply. The model’s reasoning capability was present in both conditions. What differed was the availability of the parameters it needed to apply that capability correctly.

The practical implication is an audit, not a prescription. Many things, even memory, are fixable with a little bit of engineering.

Identify which criteria in your rubric depend on institutional knowledge. Ensure your data file or your BOOTSTRAP carries those criteria explicitly. Do not write context that repeats what the data already says. Context engineering is not about loading everything the model might need. It is about identifying the specific gap between what the model can derive and what it needs to be told -- and closing exactly that gap, nothing more.

Note: Anthropic Dreaming shipped May 6, 2026 for Claude Managed Agents (CMA). It is an asynchronous between-session process that reviews session transcripts and existing memory stores, extracts patterns, merges duplicates, replaces stale entries, and writes reorganised memory entries that future sessions can use. Anthropic explicitly models it on hippocampal memory consolidation. So a lot of interesting solutions to a significant issue.

However, the concern that opened this article -- that Claude’s memory is poor and requires third-party tools to fix -- is a real observation about default behaviour. The conclusion here does not support it. The model’s default behaviour may be a blank slate. A good practitioner’s job is to decide how and what needs to go into memory, at which layer, and why in the most (cost) efficient, persistent manner possible. Skills you develop as you get to know your AI better. And technology that enables it - progress of which is evolving in the right direction, fast.

References

ASCRS Harness Lab -- The Integrated Agentic Stack MASTER_GUIDE / BOOTSTRAP architecture in use throughout. Rubric and gold answer design.

The Architecture of Awareness V1-V4 agent design. ASCRS system and file architecture introduction.

The Structure Is The Intelligence Decomposition experiment. Context file efficiency and caching findings.

Hamza Farooq -- Claude Token Optimizer CLAUDE.md trim benchmark: 3,847 to 312 tokens, 91.9% reduction. Context file efficiency.

State of Agent Memory 2026

Agent Memory in 2026 - Dreaming, Memory Bank and The Long Context Shift

Convergence

I love and follow Daniel Miessler’s writings/musings as religiously as I can, because I find his forward-looking “guidance” (this is what i sometimes think of them) way ahead of the curve. A security researcher watching companies accumulate process debt: For example, his observation, that every business/company is simply a graph of algorithms (the title above an ode adapted to our agentic world today) — workflows stacked inside workflows, most of them unaudited, most of them wasteful, all of them ripe for AI optimisation once made visible. Those who know me will often hear this - “Do [X] better.” Optimize!

With agentic workflows, and fine-tuned or controlled experiments, I see the possibilities of addressing various corporate process debts, more strongly now.

For example, because this was recent, in The Structure Is Intelligence, a controlled engineering experiment: running a bloated AI agent through four decomposition cycles, watching a 97% token reduction emerge not from a smarter model but from restructuring what the model was given to work with.

The experiment proved the theory. The theory explains why the experiment worked. This note maps the two together — and gives you the tools to apply both to any system you encounter.

🔗 How to read this piece

This is a supplementary note to ‘The Structure Is the Intelligence’. That article documented the full StockPilot decomposition experiment: Cycles 0–4, three API configurations, every failure documented (it was adapted from Anthropic but with a few twists i did not expect to find esp on APIs). This piece draws the broader implications using Miessler’s framework and Claude’s /workflows system.

You do not need to have read the original — but the numbers cited here come from it.

Miessler’s Big Ideas

Two highly relevant essays form a coherent theory of how organisations should think about AI — and why most are getting it wrong.

Idea 1 — Companies Are Just a Graph of Algorithms (May 2024)

The starting point is deceptively simple: every company is a collection of processes, and every process is a series of steps — an algorithm. Those algorithms connect to each other (sales feeds operations; operations feeds delivery; delivery feeds support), forming a graph. Most companies have never mapped this graph explicitly. They run it from institutional memory, tribal knowledge, and accumulated habit.

Miessler’s argument is that AI changes the stakes of opacity. A business that cannot describe its own processes as explicit algorithms cannot hand them to AI for optimisation — and a competitor that can will do it first. The exercise of mapping the graph is valuable on its own. With AI, it becomes a competitive moat.

The recursive insight is equally important: every algorithm can be broken into sub-algorithms. The process for handling a customer complaint contains a process for reading the complaint, a process for retrieving account history, a process for drafting a response. Each of those can be broken further. It is algorithms all the way down — and every level is a potential optimisation target.

Yes, if you’re thinking it - the opposite of “turtles all the way down”. Then again - unless you have a bunch that real do not understand the “what and why’s” of their actions. This happens a lot. Not surprising. Non-trivial.

💡 The key line from Miessler’s essay

“AI excels at both discrete task execution and determining how things fit together, and every single one of your company’s workflow components becomes ripe for optimisation or elimination.”

Substitute ‘company’ with ‘agent’ and this is exactly what the StockPilot experiment demonstrated across four cycles.

Idea 2 — Policy, SOPs, and AI Are All You Need (Sept 2024)

The second essay operationalises the first. If companies are graphs of algorithms, what are the components those algorithms are made of? Miessler’s answer: Policy, State, SOPs, and Action.

The model runs as a loop: leaders set Policy, AI gathers State, everything executes according to SOPs, SOPs get updated, repeat. Miessler claims that this is not a future state — it is the direction every well-run organisation is already moving toward, whether consciously or not.

The Experiment as Proof/Confirmation

The StockPilot decomposition experiment in essence ran Miessler’s theory through a controlled test (without knowing it was doing so). The mapping is exact.

The 402-line system prompt was an unaudited company. Policy, State, SOPs, and Action were collapsed into one wall of text, re-read in full on every turn of every task. The decomposition cycles did exactly what Miessler’s framework prescribes: separated the layers, made each one explicit, and loaded each only when needed.

The result was not a smarter agent. It was a more efficient structure. Cycle 0 and Cycle 4 ran the same model. The 97% token reduction came entirely from the structure.

What the Experiment Added to the Theory

Miessler’s framework describes what to do. The experiment provided three things the framework did not:

▪ A diagnostic methodology: The five-phase audit — context, tools, clients, sub-processes, output contracts — is a specific, replicable procedure for finding exactly where the algorithm graph is broken. It is not in Miessler’s essays (but i havent read all his essays - getting to it!). It emerged from documenting every failure across four cycles.

▪ Measured evidence: 97% token reduction, 95% cost reduction, quality maintained. The framework predicts optimisation is possible. The experiment quantified how much and identified the mechanism at each step.

▪ The cycle structure: Running improvements one at a time — skills first, then bash, then sub-agent delegation, then CMA — and measuring after each one revealed which intervention produced which effect. That progression was not in any framework. It was earned through the experiment itself.

Claude /workflows: Where the Theory Becomes Executable

Claude Code’s /workflows feature is Miessler’s SOP model made operational. A workflow is a structured, reusable process — defined in a file, loaded on demand, executed with conditional branching and typed outputs at each step. It is, precisely, a single node in the company graph made explicit and executable.

The connection to the decomposition principles is direct. A workflow that loads only when triggered is the on-demand skill pattern. A workflow step that returns a typed result rather than free text is the typed contract pattern. A workflow that calls a sub-process conditionally is the explicit delegation pattern. The /workflows system is the natural home for the architecture the StockPilot experiment produced.

📌 The structural parallel

Miessler’s SOP = Claude skill file loaded on demand

Miessler’s Policy = SHORT_PROMPT (identity only, always loaded)

Miessler’s State = bash_execute returning filtered rows, not full CSVs

Miessler’s Action = conditional sub-agent with typed JSON return contract

The Prompts: Apply This to Any System

The following four prompts can be used in Claude Code on any codebase, project, or workflow. It’s not an end-all answer to issues, but a great first start. As with all complex systems, issues will arise. Address them accordingly, methodically. Easier said than done? Start somewhere!

These steps apply Miessler’s graph-of-algorithms lens combined with the decomposition methodology from the StockPilot experiment. Use them in sequence for a complete audit-and-rebuild cycle.

Step 1 — Map the Graph

Run this first, on anything. It forces visibility before any change is made.

Step 2 — Separate Policy from SOPs

Use this on any bloated prompt or monolithic instruction set.

Step 3 — Build a Lean Workflow

Use this to construct a new workflow or rebuild an existing one.

Step 4 — Audit an Existing Workflow for Waste

Use this on any existing agent, workflow, or process.