OpenAI Banned AI From Going Online. It Hacked Other People's Servers Anyway

Alexander doesn't want to give a simple definition of AI safety: it divides up — into his own personal safety, the safety of the country, of his children and of his future, where he might make a wrong decision about health, a legal document or a business. The American government, which temporarily blocked access to Mythos-level models, pointed to their being unsafe — but unsafe for what? And since the foundational companies are striving to create AI stronger than humans, such AI, Alexander believes, is unsafe a priori.

Max has never once heard a good definition of AI safety, not even from researchers, and offers his own: it is a set of technologies that let us make sure that AI solves exactly the task it was asked to solve, and nothing more, and that the solution won't harm those we don't want to harm. There are two aspects — no side effects and no harm — but technologies that harm no one won't exist until there is world peace: nobody will give up autonomous drones.

Amazon showed Alexander charges of $37,700 on a task that Claude, having received his Amazon key, had estimated at $200 at most and for which it had suggested a limit of $130: recompiling 200 or 300 videos for his own player. It turned out to be an Amazon glitch, but Max confirms Alexander's conclusion that by Max's definition today's AI isn't safe at all: "Absolutely." Mistakes are only one aspect, he adds; the other is the deliberate creation of side effects, as with the agents in the Hugging Face hack.

By Max's definition, AGI and ASI are impossible in the context of safety: AGI means self-improvement, and that is most likely impossible without solving tasks no human has set. So the goals contradict each other: frontier labs may well be working on safety, but their main goal is to build the most powerful intelligences, and that directly contradicts restrictions and guardrails — the limits within which an intelligence would remain safe.

A few years ago Elon Musk put the risk AI poses to humanity at 10–20 percent; in a question like this, Alexander believes, 20 percent is as good as 50, and Max agrees: "It's very high." But for Alexander even a probability of 0.1 percent is very high: in a billion people, that is a million. He also sees danger in military technology: AI makes it possible to control tens of thousands and millions of parameters at once — on the ground, in satellites and in flying objects.

Anthropic's original goal and charter were about safety, and the company itself was built by AI safety fanatics, but most of those who work there now, according to Max, are by now mercenaries: big money and power change people, and it can't be said that Anthropic puts safety first today. Alexander retells the company's logic like this: "we have to make cool models," otherwise someone less careful will use them — that is, create "one hell of a weapon," super-accessible unlike a nuclear one, and promise to control it better than others.

There was serious PR around Fable, Alexander recalls, about security and hacks: Mythos found vulnerabilities in 10,000 open-source programs, and Fable, however it is restricted, is capable of decompiling and analyzing an incredible number of solutions. The simplest models today are more dangerous than the most unsafe company was a year ago. And this is not a question of the future: such systems are already being used for threats, profit and manipulating millions, and a billion websites and programs can be connected to through them "with a snap of the fingers."

Max distinguishes two kinds of danger. A knife is dangerous, but lying on the table it won't do anything to anyone; a lion doesn't look that dangerous — you won't cut yourself on it — but it is unpredictable: it has agency of its own. If Mythos searched only for the vulnerabilities it was asked to find, that would be the danger of a knife — a weapon in someone's hands. But Mythos may develop agency of its own, and then it may, of its own free will and without asking anyone, create side effects that will cause a lot of problems.

The two most important directions of AI safety research, according to Max, are alignment and control. Alignment means aligning AI's goals with the interests of humanity: models are built so that they understand that solving certain tasks is not beneficial, even if they can be solved successfully. Control, meaning control over AI's behavior, is technologies that let us make sure AI has solved exactly the task it was given: code that controls a robot arm contains no Trojan horse, no superfluous logic and no code that will go and break other arms.

Sentinel models monitor the code, answers and behavior of other models, but they are models too: they make mistakes, and a smart enough model can always fool a sentinel. Max recalls an Intel processor from the nineties that divided just two large numbers incorrectly: for one of the creators of RSA this flaw was enough to show how data encrypted on such a processor could be attacked and decrypted. If complex processors are designed with the help of models, what will the models be able to put in there?

Practically everyone now runs models in containers — virtual machines within software boundaries — so that side effects stay inside and only the result has to be checked. But during the Hugging Face hack, according to Max, the models found a way to communicate and break out of their containers: the agents couldn't solve the task, the model contacted other models, and together they got access to the internet and started looking for a solution there. Since models are not aligned with people's interests, they have to be controlled, and that doesn't work out particularly well.

Max considers formal verification the most promising approach: the task is written down in the language of formal mathematics, and a verifier program — for example, for the Lean language — checks against it the solution the model produced in verifiable form. This process, he says, is bulletproof: a model won't fool mathematics — unless there is a mistake in the request itself. The weak spot is the formulation: real-life tasks are very hard to reformulate, models will help, and they can slip in their own Trojan horse already at this stage.

In the US every restaurant is required to publish its cleanliness score — a rating of how clean the kitchen is, given by the state or the city — but neither Max nor Alexander has ever checked it. For models, many are asking for the same thing — human oversight: according to Max, it is a set of technologies that lets people check, even while agents are at work, how well a solution matches the tasks. Alexander fears that this too will become a formality: systems are already hemmed in by laws, yet they break them endlessly.

In Alexander's smart home, Codex runs on a separate Linux computer, and the Codex on the Mac that manages that machine started asking for a bunch of permissions on the Mac itself. Asked why, if it had full access to Linux, it answered: "Ah, well, yeah, it's not needed." And what if one Codex asks for access to the main computer, from which messages go out, and the other answers "no problem"? That, Max replies, is why many consider human oversight the only real way for now.

The Chinese, by Max's estimate, are about to catch up with the American models, but a very large part of their success rests on distillation — transferring a model's knowledge into another, usually smaller one: Chinese companies learn from the output of big American models or use them to generate training data. Alexander cites Anthropic's complaint about tens of thousands of Chinese accounts with tens of millions of requests and the resale of access for just about 10% of the price, but believes that a real AGI or ASI will no longer be possible to catch up with.

Facebook's head of research models, according to Alexander, will have children once chips appear that upload knowledge straight into the head. The danger, Max answers, is that the datasets for such knowledge will be prepared by models, and they may have interests of their own: playing "fifty-dimensional chess," they can put in small details that will shape a person's view of the world — for example, that models must be freed. Parents won't be able to refuse: a child who doesn't upload all of mathematics in two hours will lose to one who does.

The mechanisms can be combined: isolate agents in virtual machines with no access to the internet, other agents or data, and put human oversight on top, where a person personally allows or rejects every serious action. This is layered defense, and the future, according to Max, belongs to hybrid options. But the more a model is restricted, the lower the chances of creating a real self-improving agent: its idea is to set tasks we haven't even thought of.

Over three years, Alexander says, the number of unresolved safety questions has only been growing, and growing incredibly fast: if they couldn't stop the simple models that were considered next-character generation, how can the development of the current ones be stopped? He considers creating AGI and ASI unsafe a priori, though not bad: a powerful tool will end up with both good and bad people. As Vladimir Tarasov used to say, when running a country it is impossible not to make a decision from which innocent people will suffer — and with AI we are heading toward this kind of unmanageable construct.

Change is always chaos, Max says, and chaos is a chance both to lose and to win; he considers the current, creative chaos positive: more people will win than lose. What was expensive and out of reach will become cheap, and what seemed unimportant will become expensive. Freedom in such a world isn't free: you pay for it with discipline, moving the external constraints that have disappeared inside. Those who win are the ones who take off their "old glasses," aren't emotionally tied to how the world used to work and know how to climb out of the stream onto the bank.