Most conversations about AI safety begin with a question about intent: will a powerful system do what we want?
It is the right question. I am no longer convinced it is the first one.
Before any intelligent system can change the world, it has to cross a boundary. It has to access compute, move money, operate a machine, alter infrastructure, issue a command or persuade a human to do one of those things for it.
That boundary is where safety becomes enforceable.
We have spent years thinking about how machines should reason. We have spent far less time asking under which verifiable conditions they should be allowed to act.
Intelligence is not authority
A model can be brilliant and still have no legitimate authority over a consequential system. A less capable model can be dangerous if it is connected to the wrong interfaces.
This distinction sounds obvious. In practice, we blur it constantly.
Modern digital systems usually grant authority through accounts, credentials, API keys, network position and institutional process. These controls were designed for a world in which human time, expertise and coordination imposed natural limits on an attacker.
AI weakens those limits. It can search more paths, iterate faster and combine small weaknesses at a scale that human defenders cannot manually match. A control that depends on an attacker overlooking something is not a durable control.
The uncomfortable conclusion is that policy alone cannot carry the full burden. Neither can a permission screen, an identity claim or a software gate that the same system can eventually learn to route around.
If a decision matters, the ability to execute it should depend on evidence that can be checked at the point of action.
The missing layer is enforceability
Alignment asks whether a system chooses the right action. Enforceability asks whether it can perform a high-consequence action without satisfying conditions outside its discretion.
We need both.
An aligned system operating through weak controls can be misused, copied or redirected. A misaligned system facing no hard boundary can turn a reasoning failure into a physical consequence. The difference between a bad output and a catastrophic outcome is often access.
This is why I think verification should be treated as a control primitive, not as a feature added at the end.
Consequential authority should be conditional.
A system may propose, plan and optimize freely. But when it attempts to cross into privileged execution, it should have to prove that the action meets defined conditions.
That does not mean asking a human to click “approve” every few seconds. Human review does not scale to machine speed. It means designing the boundary itself so that authority is limited, inspectable and revocable.
Partial coverage creates false comfort
There is a second problem that security discussions often understate: a control is only as useful as its coverage.
Imagine that nine out of ten doors in a facility require strong verification. If the tenth leads to the same room, the system is not ninety percent safe. It has a bypass.
Digital infrastructure behaves the same way. Protecting the obvious interface while leaving an old administrative path, a supplier connection or a human workaround untouched can preserve the critical failure mode.
This is one reason I am skeptical of safety mechanisms that work only inside a single platform or under ideal operating conditions. The most capable adversary will not attack the control where it is strongest. It will search for the remaining path where authority can still be obtained without equivalent proof.
The relevant goal is not to add more security objects. It is to reduce the number of consequential state changes that can happen without verification.
Safety infrastructure has to solve problems today
There is an institutional trap in frontier-risk work. We design systems that become valuable only when the future threat arrives, then expect organizations to deploy them early, at cost, across enormous infrastructure.
That is unlikely to work.
Infrastructure reaches scale when it earns its place before the crisis. The same underlying verification layer needed for high-capability AI can create value now: reducing counterfeit goods, strengthening authentication, improving provenance and making access to physical systems harder to fake.
That commercial path matters. It gives deployment an economic engine. Each useful integration increases coverage, and coverage built for present-day problems can become the foundation for future controls.
The best safety architecture may therefore look less like an emergency switch and more like ordinary infrastructure: valuable enough to adopt, quiet enough to disappear into the workflow, and robust enough to remain meaningful when the adversary becomes far more capable.
Optimism needs constraints it can trust
I am optimistic about AI because I think automation can create extraordinary abundance. Intelligence becoming cheaper and more widely available can accelerate science, reduce the cost of essential services and let small teams build at a scale that once required institutions.
But abundance and control are not opposing visions. The more agency we give machines, the more important it becomes to separate what a system can understand from what it is permitted to change.
That is the first-principles version of the problem:
Intelligence will become abundant.
Authority should remain conditional.
Consequential action should be verifiable.
At 4SI, this is the systems question we are working on. I develop the broader argument in the public technical report “Verification as a Control Primitive for Frontier AI”. The report sets out the risk model and a mechanism-neutral reference architecture; protected mechanisms and implementation details remain deliberately outside the public version.
AI safety needs better models, better governance and better alignment research. It also needs a boundary that still holds when all three are imperfect.
Because in the end, safety is not only what a system intends to do.
It is what the world allows it to execute.