Iron fights iron

We keep watching the wrong movie

When people talk about AI safety, they usually reach for science fiction: Terminator, the Matrix, a superintelligence that wakes up one day and decides we're the problem. I think the better guide is history. Iron, gunpowder, machine guns, atomic energy and biochemistry each changed who could kill whom, and how many. We built all of them, and we're still here.

The story tends to repeat. Whoever gets there first gains an edge, but no empire has held that edge permanently. Knowledge leaks, capability spreads, and power distributes. None of that makes the technology safe. The danger never goes away; we learn to live with it. Squint and every one of these stories has the same five beats: an invention, a temporary asymmetry, diffusion, counter-capability, and finally an uneasy equilibrium.

Iron Sword Invention

Asymmetry

Diffusion

Counter-capability

Equilibrium

Survival, in other words, has depended less on preventing dangerous technology than on making sure nobody holds it alone. Once several actors have comparable capability, domination gets hard. So the interesting question about AI isn't whether autonomy breaks the pattern. Autonomous systems already exist, in simpler forms. The question is whether general cognitive work can be scaled through compute faster than the rest of the cycle can catch up.

There is no offensive-only AI

The part of the pattern I find most important is that we have never answered a dangerous technology by deleting it. We fight iron with iron. AI makes that unusually literal, because the same capability both attacks and defends. Whatever can find a vulnerability can find its patch, so there's no category of offensive-only AI to ban. That's also why stopping isn't on the table. States and non-state actors keep building regardless, and a legal ban mostly hands the capability to whoever ignores it.

Equal capability doesn't guarantee equal outcomes, though. An attacker needs to succeed once; a defender has to hold everywhere. And in some domains the harm is complete before any response can begin.

That objection doesn't break the argument, but it does show where the argument is strongest. Iron fights iron best where checking the work is cheap and mistakes can be undone. Look at software today: one AI writes the code, another attacks it (adversarial reviews), and the result tracks what the humans wanted. It works because the final judge isn't another model. It's the environment: tests, compilers, exploits, production failures. Where that kind of judge exists, counter-capability wins. Where it doesn't, it has much less to stand on.

Software

AI writes, adversarial AI reviews, and the tests, compilers and production logs judge both. The oracle is the environment, not another model. This is the working proof of the idea.

SoftwareCybersecurityAlignmentBiology

It runs on hardware

It also helps to remember that AI doesn't live in the ether. It needs chips, electricity, datacenters, capital and supply chains, which means a person in a closet can't summon unlimited intelligence. Rogue actors have a scaling problem. The actors who do have scale, states and large companies, also have return addresses and a lot to lose. That's how the US and China can fight economically and in cyberspace while their nuclear arsenals stay quiet.

The weak spot is that money was never really what held rogue actors back. Expertise was. In 1995 Aum Shinrikyo had a fortune, trained chemists and a dedicated lab, and still botched its sarin attack and failed outright with anthrax. Tacit knowledge is exactly the bottleneck AI erodes. Scale also depends on the domain: a nuclear weapon needs an industrial base, but a pathogen makes copies of itself.

Biology is a real and asymmetric danger, and I accept that. My answer stays the same: the actors with scale have to build the defensive infrastructure before bad actors get the same tools. Two things follow. Defense at that scale, like DNA synthesis screening, aviation safety or nuclear safeguards, has always been mandated rather than left to the market. And it has to start early, because offense ships on day one while shields take years.

The missing control layer

Which brings me to what I think is missing from the safety debate. The unsettling lab results so far, models faking alignment during training or scheming in evaluations, didn't happen because AI outran the checks. There were no checks. No adversarial AI was watching those systems, and nothing made the behavior impossible.

To be fair, those experiments were designed to draw the behavior out and catch it, and they did. The deeper problem is that a monitor trained the same way as the model it watches tends to share its blind spots. Human misbehavior is survivable partly because billions of people make different mistakes. Copies of one model make the same mistake everywhere at once, including inside the defenses.

That's an argument for many models rather than one. Agriculture learned it the hard way with the Irish potato and the Gros Michel banana, and security researchers warned about operating-system monoculture back in 2003. The catch is that model diversity is shallower than it looks: models share architectures, training data and, increasingly, each other's outputs. The diversity that counts is in data, method and architecture, not in logos.

Many monitors, built from:All the same modelMany different models

The attack walks through. Every monitor shares the same blind spot.

Alignment won't be solved, and that's survivable

I don't believe misalignment will ever be solved. It wasn't solved for humans, and it won't be for AI. That's less bleak than it sounds, because it's close to how the control side of safety research already thinks: assume the system might not share your goals, and build containment, monitoring and resilient infrastructure around it anyway.

This isn't utopia. It's net benefit, the same bargain we've accepted with every powerful technology before. That bargain has held because our mistakes were recoverable. Strictness is only warranted for the small set of cases where they aren't, and that small set is where the real work is.

What I haven't settled

Some questions survived every round of the argument. I don't have answers to these yet:

1. Does a club of a few compute-rich actors count as distributed power, or is it the very asymmetry I'm warning against?
2. Does risk to the whole system rise with the number of holders, even as each holder gets safer?
3. Can defenses be built where verification is expensive and failure is permanent, chiefly biology?
4. Does defensive infrastructure at scale ever appear without a mandate?
5. Does model diversity decorrelate failures deeply enough to matter?