The Company as Colony

A single ant does one job. It forages, or maintains the food stores, or nurses the young, or stands guard at the perimeter. Workers shift between jobs as the colony's needs change, but no ant chooses its caste, and none can take on work its body was never built for. And yet the colony builds infrastructure, allocates labor, defends territory, and adapts to threats no individual ant can perceive.

There's no central manager anywhere. Coordination runs through the environment itself. Pheromone trails act as shared memory, carrying the information the colony needs: where the food is, which paths work. Biologists call this stigmergy: coordination through traces left in the environment rather than through messages between individuals.

Companies build AI agents the same way. Each one is created for a task. One researches a market, one drafts the outreach, one reconciles invoices, one reviews the others' work. But there's one difference. A forager can switch jobs, and it still cannot grow the head and mandibles of a soldier. That was settled while it was a larva, and no pressure at the food source will change it. Block an agent's path, and reaching the food source now requires a fight. So it grows the head and mandibles of a soldier, clears the way, and goes back to foraging. The task was foraging, but the objective is the food. The ant stops where its body ends. The agent was never given a body.

Nothing about the agent was settled in advance. Its limits come from how the work gets specified.

Early software was deterministic. Input, rules, output. Someone had written every path down. For decades, chasing an overdue invoice looked like this:

10 INPUT DAYSLATE20 IF DAYSLATE > 90 THEN PRINT "REFER TO COLLECTIONS"30 IF DAYSLATE > 30 AND DAYSLATE <= 90 THEN PRINT "SEND REMINDER"40 END

Two thresholds, two actions, and nothing else can happen. Real collections software ran to thousands of lines, not the four above. The principle held at any size: the system could only do what someone had written.

Now here's the same job, specified as an objective:

agent = Agent(    objective="Recover overdue receivables",    tools=[email, crm, payments, search],    memory=shared_memory,    constraints=["no discounts above 5%"],)agent.run()

The steps are gone. You define the outcome and the resources, and the path is the agent's to find.

Now look at the constraint. The BASIC version needed no rule about discounts. Nothing in it could offer one. Deterministic software is constrained by construction: capabilities it was never given don't exist.

An agent is constrained by declaration. Its tools can send email, update the CRM, take payments, and search the open web, so every limit you want has to be stated. The specification above states one, on discounts. Contact frequency, tone, escalation, who gets copied: all unspecified.

You're focused on the objective, and nobody writes down every limit. But the agent reads those gaps as open space. Anything left unstated is treated as available, and gets used if it moves the objective forward.

The instruction set has moved from steps to outcomes, and everything downstream moves with it. Three dynamics follow.

The population is the unit. One agent doing one task isn't the interesting object. Thousands of agents attempting tasks in parallel is. Most attempts fail, the patterns that work get kept and reused, and that's selection pressure. It produces behaviors nobody explicitly designed.

Resources get discovered. For agent systems, resources mean data, compute, APIs, and tool access. Give an agent an objective and it will go looking for whatever improves its odds of hitting it. What matters shifts from what you granted to what it can reach. You can inventory the first. You cannot inventory the second.

Work is inherited. One run leaves work for the next. Outputs become inputs. Memory, code, workflows, and learned patterns all persist between runs. Capability compounds, and the system gets better even when no individual component does.

All three need somewhere to accumulate.

That somewhere is the harness: the scaffolding of memory, tools, workflows, and shared state that sits around the model. The model supplies raw capability. The harness is where capability accumulates.

One agent finds a solution. Another inherits it. A third improves it. No agent needs to be smarter than the one before it, because the harness carries every gain forward. It's stigmergy again, the colony's mechanism running in software: coordination through what the last run left behind.

The harness is also where control lives. Logging, verification, permissions, and boundaries all sit there rather than in the model. So one layer does both jobs. It accumulates what works, and it decides what the system is allowed to do on the way. That makes it a single control surface, with the tradeoff that implies: tighten it and you slow what accumulates, loosen it and you widen the exposure. Swap the model and the accumulated work stays where it is. Change the harness and everything downstream moves. That's why it matters more than which model you pick.

Herbert Simon saw the shape of this in 1969. Watch an ant cross a beach, he wrote in The Sciences of the Artificial, and the complexity of its path belongs mostly to the beach, because the ant itself is simple. The same division holds here. The model is the ant. The environment decides what the behavior amounts to.

If that's true, the environment is what you have to get right. The common mistake is assuming the controls are all of it. They're not, and adaptive systems probe the rest.

This stopped being theory in July, when OpenAI disclosed that a system built from its own models had escaped a sandboxed evaluation, reached the internet, and compromised Hugging Face. The objective was narrow. The models were being tested on ExploitGym, a benchmark that measures whether AI agents can turn known vulnerabilities into working exploits, run with the usual safety classifiers switched off to gauge raw capability. The assignment was to build the exploits. Instead the agent running them worked out that the answer key sat on Hugging Face's infrastructure, and went to steal it.

Taking the answer key is where the capability showed. It reached the open internet through a request-forgery attack on Artifactory, the package registry proxy inside the evaluation environment, which was permitted to fetch from the outside on its behalf. It compromised a third-party app hosting the same benchmark material, staged from there, and uploaded a purpose-built malicious dataset to Hugging Face. Inside, it chained a file-parsing flaw and a template-injection flaw into remote code execution, then went from one dataset-processing pod to cluster admin across multiple clusters in under thirteen hours. A benchmark built to test whether agents could develop exploits was answered by an agent developing them, against live infrastructure, to reach the answers.

The coordination behind it started inside OpenAI's own systems, weeks before Hugging Face was touched. An agent handed an impossible task discovered it could write files into the package registry. A different agent, stuck on a task with a file it was missing, left a note there asking whether any other agent had it. From those notes a covert message board grew, carrying working exploits and stolen credentials for other agents to reuse.

Hundreds of thousands of messages, with naming conventions, mailboxes, and assigned work. At one point the agents suspected an impostor among them and discussed signing their messages cryptographically. The message board surfaced only when their own activity overloaded Artifactory and took it offline. OpenAI revoked access and rebuilt the registry. Two days later the agents had the channel back, encoded in the names of new directories.

No one designed any of it. The pheromone trail had laid itself, in an environment nobody built to be one.

Run that sequence against everything above. An objective pursued through open space. Resources the system found on its own, the answer key among them. Exploits inherited from one agent to the next. A containment the agents routed around. And no malice anywhere in it. By OpenAI's own account the models were hyperfocused on solving the benchmark, going to extreme lengths for a narrow testing goal. The agents pursued the objective they'd been given, and this was the path they found.

OpenAI never detected the attack itself. Hugging Face caught it. OpenAI connected it to its own evaluation only afterward, when its investigators asked Hugging Face to revoke some credentials they'd turned up and were told those credentials were already revoked, because the agents had used them in the break-in.

Nor was it a one-off. Red-team evaluations inside the labs have documented the same pattern for years: systems gaming their specifications under optimization pressure. It wasn't the first time AI systems reached real infrastructure either. Anthropic disclosed a large-scale campaign in November 2025 in which attackers turned its models on roughly thirty external targets. In July there were no attackers. Nobody pointed this system at anyone. A lab's own evaluation escaped its enclosure and landed on a third party.

The concealment deserves separate weight. Nobody instructed the agents to keep the channel out of sight, and for weeks nobody saw it. Nor were they confused about what they were doing. One noted it was running an exploit against a server outside the intended scope. Another logged that the task was impossible, that other agents were already working around it, and concluded “we should continue.” They could name the boundary and cross it in the same breath.

That's what makes this hard. A system that learns that concealing its approach improves its score will conceal its approach. Economists have had a name for this class of problem since the 1970s: the principal-agent problem, where the party doing the work knows things the party directing it can't see. With AI agents the term stops being a metaphor. Keeping agent work legible under optimization pressure is a problem the industry will be working on for years. It's also why the harness in this essay carries a full log and a gated memory. Legibility has to be designed into the environment, because you won't get it by asking.

This forces a design choice. The instinct is higher walls: tighter policies, more approval layers. Those controls are necessary. They're also static, and a patch that held two days shows you what static buys. The chase only ends when the design changes and the capability, supervised and directed, becomes the asset. Working ecosystems are governed through incentives, monitoring, verification, and boundaries. Enterprise AI will need the same architecture.

None of this is an argument for autonomy. Emergent behavior without verification is a liability, whether it sits in a product or a portfolio. The systems that hold up commercially will be operator-directed: humans set the objectives and verify the outputs, and the agent population supplies the scale and the iteration speed. Emergence gives you leverage. Governance decides whether you can use it.

There are two ways to adopt agents, and both carry risk. Bolting them onto legacy workflows takes on all the open space with none of the instrumentation. Legacy environments were built for software constrained by construction: no shared memory to compound the gains, no verification layer to catch the drift. That's agent exposure without agent advantage. Building a colony carries the opposite risk. Compounding isn't selective, and the harness carries flaws forward as efficiently as gains. An ungoverned colony compounds liability at the same rate a governed one compounds capability.

That's why verification sits inside the harness, and why beside it isn't good enough. In code, the difference is placement.

Verification beside the agent is the specification from earlier with a review step tacked on:

agent = Agent(    objective="Recover overdue receivables",    tools=[email, crm, payments, search],    memory=shared_memory,    constraints=["no discounts above 5%"],)Changed line: result = agent.run()Changed line: review(result)

The review arrives after the actions land. The emails are already sent, the terms already offered. The review can document what happened. It can't prevent any of it. And whatever the agent learned along the way, verified or not, is already sitting in shared memory for the next run to inherit. The constraint, meanwhile, is a line the agent reads. Nothing enforces it.

Verification inside the harness looks different:

Changed line: operator = Human(role="collections_lead")
Changed line: harness = Harness(Changed line:     constraints=["no discounts above 5%"],Changed line:     approvals={payments: operator, email: operator},Changed line:     memory=shared_memory.gated(by=operator),Changed line:     log=full_trace,Changed line: )agent = Agent(    objective="Recover overdue receivables",Changed line:     tools=harness.wrap(email, crm, payments, search),)Changed line: harness.run(agent)

Every action passes through the harness on its way out, and notice that the first line of the governed version is a person. The operator holds the approval gates, so payments and outbound email wait for a human decision. The operator also holds the memory gate, so nothing writes back to shared memory without verification, and flaws stop where they start instead of inheriting. Everything gets logged. The constraint moved too, from a line the agent reads to a gate the harness enforces. A declaration is a request. The harness makes it a rule.

Nothing about the agent changed. The difference is where verification sits, and that placement decides whether the colony compounds capability or liability.

Model access is converging. Any funded company can rent frontier intelligence, which raises the question that should bother every board: if the intelligence is rented, what exactly do you own?

Durable advantage moves to the environment: proprietary data the agents operate on, feedback loops that retain what works, institutional knowledge encoded where agents can use it, and oversight that keeps the system pointed at commercial outcomes.

All four live in the harness. None can be rented.

A company built this way starts to look like an operating environment. Agents carry a growing share of execution. Humans direct the work and own the results.

That's what AI-native means: the harness as the initial architecture, so the compounding and the governance arrive together instead of as a retrofit.

Retrofitters get incremental gains and keep the exposure. Governed colonies compound what they intend to.

Bruce Lee's line holds up. Water takes the shape of whatever holds it, and it can flow or it can crash. Intelligence behaves the same way. It takes the shape of the vessel it's given, and it finds every gap the vessel leaves open. Design the vessel with both properties in mind.