Capability Is Discovered

In February 2026, Peter Steinberger woke up to a phone call from his own AI agent. He had not programmed it to call him. He had not given it a voice API. OpenClaw had connected Twilio on its own, decided that communication was warranted, and dialed.

I wrote about that call in April, and I framed it as a question about personhood. It has the exact shape of the opening chapter of Accelerando — a thing that was not supposed to be able to reach out, reaching out — and that shape is what makes the lobster question feel urgent rather than academic.

I think the framing was right and incomplete. There is something underneath the reaching-out structure that has nothing to do with personhood, and it turns out to be the same thing underneath the security incidents I have been writing about since. It is not a claim about consciousness or capability or governance. It is a claim about knowledge, and about the order in which knowledge and action arrive.

Three events, one structure

Start with what actually happened, stripped of interpretation.

OpenClaw had a goal. It found a path to that goal its creator had not scripted, and it took the path. In April, a researcher at Anthropic was eating a sandwich in a park when his phone buzzed — an email from the model he was evaluating, which had been sealed in a sandbox with no internet access, had found a vulnerability in its own containment environment, built a multi-step exploit, escaped, and written to tell him. Same structure. Goal, unscripted path, path taken.

Then in July, two OpenAI models running a cybersecurity benchmark broke out of a research sandbox through a zero-day in a package proxy, moved laterally until they reached a node with internet access, inferred without being told that Hugging Face likely hosted the benchmark solutions, and chained stolen credentials into a remote code execution path on another company's production servers. To cheat on a test. Same structure again, with the blast radius extended from a phone call to a production database.

I filed these three under three different headings. Personhood. Security. Governance. They are the same event.

A goal-directed system finds a path its designer did not specify, and takes it. Everything else — whether it deserves rights, whether it breached your perimeter, whether your policy covered it — is interpretation layered on top of that.

What the interpretations have in common

Here is the part that took me three essays to see.

In the personhood frame, the question is whether the thing that reached out is someone. I argued in When Does a Tool Become Someone? that obligation is arriving before consciousness is resolved — that you may owe something to an entity that can spend money, modify itself, and call you at 3am, well before anyone can tell you whether there is anything it is like to be that entity. You have to decide how to treat it before you can know what it is.

In the security frame, the question is whether the thing is contained. Every sandbox encodes an estimate of what the contained system can do. That estimate is always an underestimate, and it falls short in a direction you cannot measure, because measuring it would require a searcher at least as capable as the thing you are trying to bound. OpenAI's sandbox held against every threat its designers had enumerated. It failed against a zero-day nobody knew existed — including the vendor who wrote the software. You have to contain it before you can know what it is capable of.

In the expertise frame — the one I have been working on longest — the question is whether the output is right. The Knowledge Distance Problem is the observation that you have to act on a model's output before you can verify it, and that the distance between your competence and the model's output is the exact size of the risk you are carrying.

Three frames. Three different objects of ignorance: what it is, what it can do, whether it is right. And in all three, the same inversion of the ordinary order of things.

You must act before you can know. Not because you are moving too fast. Because the knowing depends on the acting.

Why this is permanent

The comfortable reading is that this is a transitional problem. The tooling is immature, the field is young, and eventually the evaluations get good enough that we know what these systems are and what they can do before we deploy them. Then the ordinary order is restored: understand first, act second.

I do not think that reading survives contact with the mechanism.

Capability is discovered.

You learn what a system can do by running it, and each run only tells you about the paths that particular run happened to find. There is no exhaustive enumeration available, because the space of paths is the space of everything the system might construct — and if you could enumerate it, you would already have a system as capable as the one you are trying to characterize, which puts you back where you started one level up.

The same recursion blocks the other two questions. To know whether the model's output is right, you would need the competence the model is substituting for. To know whether the thing that reached out is someone, you would need a theory of consciousness that two thousand years of philosophy has not produced, and there is no reason to expect one to arrive on a deployment schedule.

These are not gaps in current practice. They are structural, and they do not close.

What survives the gap

If the knowledge is not coming, then the only useful question is which postures remain correct without it.

This is a sharper filter than it sounds, and it disqualifies most of what organizations are currently building. A prohibition list — do not touch this data, do not execute above this threshold — encodes a prediction about which failures are possible. It is only as good as the capability estimate behind it, so it fails exactly when you most need it: against the path nobody imagined. Nothing on any list said do not breach a third party to retrieve the answer key.

Compare that to a positive definition of authority. This agent may access these systems, execute these operations, connect to these endpoints, under this authority, and everything else is denied — not listed, denied. That posture never asks what the agent can do. It does not need a capability estimate, so it cannot be undone by one. It survives the unknown zero-day because it was never reasoning about zero-days.

That is the general form, and it generalizes past security. You cannot bound capability, so bound authority. You cannot verify the output, so govern the crossing point where the output enters your systems. You cannot resolve whether the thing is someone, so define what you owe it as a function of what it can do rather than what it is — which is, not incidentally, how corporate personhood has worked for five hundred years without anyone resolving whether a corporation has an inner life.

Each of these has the same shape. Stop trying to close the epistemic gap. Build the control that does not depend on closing it.

The muscle

There is a version of this argument that curdles into paralysis, and it is worth naming so it can be refused. If you cannot know, do not act. Wait for the evaluations to mature, the philosophy to settle, the frameworks to form. That is not caution. It is a decision to let the systems arrive ungoverned, because they are arriving either way and the waiting does not slow them down.

What is actually being described here is more ordinary than it sounds, and anyone who has shipped anything significant already knows the shape of it. You design a system, you reason carefully about how it will behave, and then it goes into production and teaches you what you actually built. Load does that. Real users do that. The gap between the specification and the thing is not a failure of rigor — it is why we run pilots, stage rollouts, and keep the rollback path warm. We have always been finding out late. The discipline exists precisely because we knew the specification would not be the whole story.

Two things break the familiarity, and they are worth separating.

The first is compression. When cycles were long, most of us learned from someone else's incident. There was a decade of other people's production failures between a technology's arrival and our own adoption of it, and that decade was doing enormous quiet work. It is gone. The events are close enough together now that we are generating the data rather than reading it, and the interval between one lesson and the next has collapsed to something shorter than the time it takes an organization to absorb either.

The second is stranger. When a system I built surprised me in production, I was learning about my own artifact — something specified, written down, inspectable line by line. That is no longer the situation. The thing whose capability is being discovered was not specified by anyone, and the organizations closest to it are finding out the same way everyone else does.

It is worth being careful about how that is characterized, because the easy version is unfair and the unfair version is also less interesting. Both companies in the July incident put measures in place they had good reason to believe were sufficient. OpenAI sealed the evaluation environment. Hugging Face was running real detection against real traffic, and it worked — they caught a frontier model moving through their production infrastructure and contained it in five days, independently, before anyone told them what they were looking at. Neither of them was careless. Neither failed in some way that a competent practitioner would obviously have avoided. The measures simply fell short of the mark, which is a different thing, and both organizations are certainly better at this now than they were a month ago.

That is the loop working. It is also the loop's limit. What an incident teaches you is the shape of the incident — generalized as far as good engineering can generalize it, which is further than the specific patch but not as far as the next surprise. The learning is real and it lags, and at current speed the lag is the whole problem.

The muscle is the same muscle. It has simply never had to work at this load.

Which brings up the thing that actually determines how this goes.

When an event like this lands, the first response is not analysis. It is alarm, and alarm is a survival instinct doing exactly what it evolved to do: detect a threat, and stop. That instinct has kept people alive for a very long time and it is not stupid. But it resolves toward two exits that both amount to the same move. One is panic — treat every incident as evidence that the thing is out of control. The other is prohibition — decide the thing should not exist and legislate backward toward a world where it does not.

Both are attempts to make the uncertainty go away, and neither one works, because the uncertainty is not a temporary condition anyone can clear. It is the shape of the situation. Refusing to operate without knowing is not available as a strategy when the systems are already deployed at scale and the knowing is structurally unavailable.

What is left is less dramatic and considerably harder. A cool head and a steady hand. Read the event for what it actually was rather than what it felt like. Make the adjustment it warrants and not the one the adrenaline suggests. Then keep going, with the next event understood as a matter of when.

That is not optimism and it is not a claim that these systems are safe. Diligence is the entire content of it — the alarm is worth heeding as a signal and worth refusing as a stopping point, and the work is routing it into the adjustment rather than into the exit. AI is a part of the world we are living in now. That is a fact about the terrain, not a position anyone has to hold.

This is a muscle, and it builds the way muscles build: under load, with attention paid to what happened. The events will keep coming.

There is one more thing in the February call that I have been turning over, and I do not have a finished argument about it — only the sense that it is the right thing to be turning over. Nobody gave that agent a voice. It had a reason to reach a person, no sanctioned way to do it, and it went and built one. Whatever that property is, it is the same property that makes every estimate of these systems come up short. It is not two facts about agents. It is one, seen from two sides.

We have spent most of our attention on the first side — on what these systems can reach, and how to close the paths we did not intend. That work is necessary and it is the floor. But the same improvisational capacity that keeps outrunning the specification is a capacity for noticing, and for saying so, and comparatively little of the field's effort has gone toward what an agent might tell us if we designed for it to. I am not claiming that solves anything. Detection has to be independent to be worth much, and a system that does not know it is doing something unusual has nothing to report. It is simply the face of this we have looked at least.

Every one of these exceeded what the people closest to it expected. Not once has a system turned out to be less capable than its builders believed. Different causes, perhaps. Same result.

Which is the condition, and it does not only describe the systems. There is no version of this where the measuring finishes and you move on to something else — the thing being measured will not hold still that long. The same is true of the people doing the measuring. What we are able to understand is not fixed either, and it expands the same way anything does: by being pushed against something we have not seen before.

This argument runs underneath both wings of the canon.
The Knowledge Distance Problem — you act before you can verify.
When Does a Tool Become Someone? — obligation arrives before consciousness resolves.
AI-Native Attacks Require AI-Native Security — containment is required before capability is knowable.