The most important question right now isn't who's going to win: OpenAI, Google, Anthropic or SpaceX. And SpaceX was already at almost three trillion. The most important question is a different one: who's going to hit stop if artificial intelligence becomes genuinely dangerous? Because Anthropic has just said a very interesting thing. The most powerful models can't simply be released into the world like a new button in an app. And by the way, they said this before their Fable model was shut down.
This isn't a little chatbot, they said, not a helper for a school essay. It's a system that can help write very serious code and look for weak spots in other people's systems, make sense of complex biology, speed up how companies work, and gradually take over more and more decisions. And this is where a very interesting thing begins. Anthropic isn't saying ban artificial intelligence. It's saying that if a model has become too powerful, it has to be checked not only by the developers themselves.
There have to be external checks, there have to be reports of some kind. And if a launch is dangerous, the state must have the right to stop it. And this already concerns not just programmers, it concerns your job specifically, my job, my business, your children, your data, your money. Because if artificial intelligence ends up in the wrong hands, that's its big threat. And if we strangle it with rules, we lose the technology race. And if we leave everything as it is, then a few private companies in the world will decide what can be launched for the whole world and what can't. So who should keep their hand on this switch?
Right? Companies, investors, the state, or should nobody do anything at all, and everything should develop, let's say, democratically? That's what we're going to talk about in today's episode.
Anthropic has released a document, "Exponential Artificial Intelligence Policy". Yes. And once more, on why the document is interesting. I even started preparing this episode before the Fable model was shut down. It was shut down, well, just recently. It's one of the most advanced models Anthropic has released. Maybe by the time this episode comes out, the Fable model will have become available. Yes, we're all still expecting it to become available. And just yesterday a friend of mine was here, and he'd just signed a contract with Anthropic, specifically to use the Fable model. He says they're just small. I say to him: "How?
Fable's already been shut down." He says: "Well, they should, in theory, they should open it up. They've got no options, right, because they're getting ready for an IPO. And they're in a race, of course, with OpenAI, with Gemini and with SpaceX." By the way, why am I pointing to SpaceX again? I said it at the very start. Many people underestimate that xAI sits inside SpaceX. And if someone thinks xAI has fallen behind in the model race, like, for example, the company Meta fell behind, I wouldn't say that.
xAI has a pretty serious standing in the US. Sure, that model is seemingly not of that class, not of that level, but judging by what Elon Musk is doing, he'll rebuild that model and bring it up to serious standards. I've also said in many episodes that he's already started applying that model, in particular, to software development at the Microsoft level, yes. But those are other episodes. So today, what's the story? What was Anthropic's main idea? That artificial intelligence is developing so fast that the usual state mechanisms, laws, are lagging very seriously behind. Yes.
So for the most powerful and serious models, including, as it turned out, models like the Anthropic Mythos class, a separate regime is needed. Yes. Not some general ban on artificial intelligence and not control over every little product, but mandatory rules for the big, serious developers. Yes. Precisely the serious developers of this cutting-edge artificial intelligence at the frontier of what's possible. Yes, what's called frontier artificial intelligence. And Anthropic proposes a combination of three things: yes, obligations for the developers themselves, independent external evaluation, and legal powers for the state.
Well, what do legal powers for the state mean? That, essentially, it'd be possible to stop or hold back the deployment of some powerful models, or an independent evaluation that shows a serious catastrophic risk that could occur. Essentially, what happened? Anthropic rolls out the Mythos model, gives it to a limited number of companies. And let me remind you that when the US restricted the use of the Fable model, they restricted it for everyone in the world except US citizens, citizens specifically.
Not US residents, not American companies, but US citizens specifically. And even the company Anthropic itself and its employees who aren't US citizens fell under this ban. Yes. And essentially, the Mythos system, of course it was used by foreign nationals, but they didn't open it to anyone either. They said, like, this is a serious system, we're not going to open it up. Even though many institutions in the world were worried. The government and the banking system of the UK, for example, were very worried about this. And then they release Fable.
They put a huge number of restrictions on it, but they release it, and a few days later the state blocks it. Anthropic doesn't agree with this, with this block, even though, essentially, literally a few days before that they'd released this very document I want to tell you about today. Anthropic does very serious research on this topic, and to my mind they're probably ahead of all the players today, of those four world leaders I named. Well, some people think they're no leaders at all and that Chinese models do the same thing. I don't agree with that. Obviously, that's purely my personal opinion.
You can voice your own opinion, and I'm trying here to give an independent audit, after all, of what Anthropic is doing. But still, still, Anthropic is unique in how it describes and conceptualises its vision of everything that's going on. If we take OpenAI, Max Grigoriev and I recorded an episode a few months ago, and he said they have a mission to create AGI and to create superintelligence. A few days ago I met a man who'd been at a dinner where Sam Altman and the product directors were present.
Fairly serious people, in a very narrow, narrow, narrow circle, there, fewer than twenty people. And he said that, by his feeling, OpenAI doesn't have that drive to create AGI. It seems to me it must still be there inside. Maybe it's stayed with the researchers, the research people who are inside. It has to be there, yes, and not just the drive to compete with Anthropic, to worry about Anthropic, to worry about making revenue in the business products market or how to sell ads, right?
I think there has to be a drive for AGI. So, Anthropic definitely has the drive to create a grand artificial intelligence, a large-scale one, as they call it, large-scale artificial intelligence. And we had an episode about the Anthropic Constitution. I very strongly recommend that those who haven't watched it do, and those who have, show some support for how much you liked that video. I think I gave a very good description there, talked about how Anthropic creates its model in general and what it sees inside in terms of the model's consciousness and the model as an entity in general, yes, the model as an entity and as a universal entity in general. It makes sense, it definitely makes sense to watch that.
So, Anthropic proposes, once again, yes, not to ban artificial intelligence. That's very important, yes. That is, it's not saying to ban it, it's proposing not to trust self-reporting alone. That is, transparency, reports and system cards, all of that is great from a systems point of view, but it's no longer enough without independent evaluation. And essentially this coincides with their Constitution, in a way. And to give the state that right, some kind of red button, so that if a model is genuinely dangerous, the state simply goes and, let's say, blocks it.
We see these blocks in the world, in particular, for example, the US. The Pentagon has put a large number of American companies on a list in connection with building artificial intelligence, there, Alibaba, Baidu and so on, yes. In particular, it put them in a kind of red zone. That is, they're banned from use. For them, obviously, it's China that puts the use of Chinese models in that red zone. For example, I wouldn't want to use Chinese models. I'm deeply convinced that all the data there is categorically recorded. I think they even discuss it openly, say it openly.
And still, in the US models, at least, they try to show all of this very transparently. For example, in particular, the Fable model. It was also unique in that it kept all data for thirty days in any case, as a Mythos-class model. And Microsoft, for example, forbade, banned the use of that model in its companies, there, literally in the first days. I personally used it in my projects with great pleasure, started building, and I think I saw a fairly interesting result. And in general, after Anthropic shut down Fable, I don't know about you, but it seems to me that Anthropic's systems started working worse. Yes, they started working worse.
To the point that it seems to me Opus 4.8 has started working worse. I was still using ChatGPT-5 Pro in my everyday work. I understand that I'm among the thousands or tens of thousands of people who truly use 5.5 Pro so fundamentally in a person's ordinary life and also in a professional, business development environment. Which I definitely like more than 4.8 Opus. Some people come and tell me how cool Claude is, how cool 4.8 definitely is, though then it turns out they don't even have the top plan.
They're not using the fanciest systems. But still, we'll see. Ah, we have a lot coming out about Claude right now, because a lot of development is being done on Claude right now, a lot of development in the world. We'll certainly keep talking about it, among other things. So, about the document. They have a part that relates to frontier artificial intelligence and artificial intelligence.
Within this document, Anthropic says that artificial intelligence is accelerating while state institutions move slowly. Yes. And Anthropic on the whole starts from this basic thesis that this development of artificial intelligence, as I said at the very beginning, is so exponential and has this exponential character that, let me remind you, this isn't linear development, where you have one plus one, plus one, plus one, right? You have a curve that, well, keeps heading up endlessly all the time, all the time, yes, faster and faster.
And meanwhile state procedures, various laws, the agencies that exist, standards, there, some judicial mechanisms, for example, are usually created very slowly, yes. And what have the last two years shown? It's not just that state mechanisms are created slowly, even business reacts slowly. Not to mention the billions of people on planet Earth. Business reacts slowly. Let me remind you, ChatGPT has only five million paid subscriptions out of a billion users. And people still treat ChatGPT with much less trust than they treat purchases on Netflix, subscriptions to Netflix or subscriptions to some ordinary services.
So imagine, yes, how, it would seem, a person was given incredible, phenomenal access that had never existed anywhere in their life at all. And the paid subscribers of such systems are fewer than gym memberships. Yes, or, there, fewer than coffees sold in the world. Well, which is very strange. I'm talking in different units here. So, because of this gap between the reaction, the development of artificial intelligence and the reaction of everyone else, I'm, once again, deeply convinced that the population has started reacting very, very slowly.
In 2023 the population's reaction was decent, yes. In 2025 it slowed down. In '26 I see that it's incredibly slow. Because of all this, a very big gap arises. And by the time when, in Anthropic's opinion, and precisely here I'm not going to add my own opinion. In their opinion, by the time when, as Anthropic says, society understands the risk and starts building full-fledged regulation, some dangerous capabilities may already be deployed, and there'll be essentially no going back, yes.
And here, by the way, it's important that Anthropic builds this political logic on that: that we need laws prepared in advance, clear, applicable mechanisms for the most powerful models. What can I say? We've been talking about this for weeks, for months I've been shouting it in every, every direction, that the models are becoming less and less accessible, and they, there they are, becoming less accessible. The only mechanism the state can create is that it can simply restrict things and not let them develop, yes.
But given that there's China, the US has no chance, because China distills, will distill all the models, keeping pace with the American models. The American government, among others, needs these kinds of lab kitties in the form of billions of users who today use OpenAI and Gemini. Well, and of course Anthropic too. But Anthropic simply has fewer users, considerably fewer, or, there, fewer than xAI, for developing its own models, its own systems. Yes, because I'm personally convinced that the winner will be whoever has maximum contact with end users, yes. And that'll happen in two directions.
Either it's regular users of systems like OpenAI or Anthropic, there, or others. Or, as I talked about in the episodes on AI native software, that's software that will be created in the world purely by artificial intelligence. If that software is created by Anthropic, for example, and Anthropic is its full-fledged creator and maintains that software, then essentially, for example, let's imagine Anthropic built a system like booking.com and fully maintains it, and it's maintained by artificial intelligence.
Or even booking.com ordered that system from them, not booking, someone new, yes. Booking.com won't order it any more, someone else ordered such a system and it's maintained on Anthropic, then that's also a direct subscription, I think, probably far cooler than direct use. It's essentially like how Microsoft won for a while, or, there, Android, or Apple with its whole infrastructure, in terms of that Microsoft infrastructure, in particular, by creating the operating system. Yes, that was a very cool, well, just an incredible win. So, the document consists of two logical parts: developers' obligations and societal resilience.
Which is, actually, what you and I are looking at today, yes. And there's this, they have a part called the advanced AI framework, and that's the description of the block for advanced artificial intelligence. This part is built not only around requirements for the lab and for the various labs that work on artificial intelligence. It has precisely this very important division, yes. The first part is the obligations of the developers of these most powerful models: risk assessment, some kind of extensive reporting, yes, a system card for their big systems, some independent checks.
And, by the way, openness about reporting various incidents and generally reporting what's going on in terms of safety. The second part they have inside, within this document, is what could be called societal resilience. Yes, that's everything to do with biosecurity, cybersecurity, military security. That is, readiness for new types of threats and the ability to detect them, respond, contain the harm and so on. By the way, let me remind you all that today we mustn't forget that companies, on the one hand, are seemingly for the world and for society as a whole, founded on their own principles.
But what is the world and what is rightness? In the world today we have a very wide range of understanding of that, yes. And on the other hand, these companies are interested in their own interests. In particular, for example, they go public, list on the market, they need to increase the value of their securities, the share price and so on, yes, to earn, to earn money, to earn profit, to pay employees' salaries and so on. There's a whole area that concerns the state, and we can see that the American models are interested in the result for the US and, of course, in the result for US citizens.
And there's one more small, very interesting part. Which is, after all, what are the actual employees of these companies or corporations interested in? Because, after all, the serious models, whichever way you look at it, are created by a unique group of people. I remember at Yandex I was always told that there's a group of people, literally just a couple of people, who truly know how the search engine works, yes. Or at Google there are a couple of people who know how the search engine works.
So, at the company Anthropic, Gemini, there, ChatGPT at OpenAI, there, xAI within SpaceX, well, obviously there's a small group of people who truly understand how the models work. And their management, in fact, as we can see, doesn't fully understand how the models work. And, well, at Anthropic the leaders probably understand how the model works the most. While at the others, honestly, I have a big question in technical terms. So, once again, because the document has these two parts, Anthropic essentially isn't saying just let the labs behave carefully.
They're saying that we need rules too, rules for the labs and strengthening of society's external protection systems. One of the key elements of this document they've released is its narrow scope, yes. Anthropic isn't proposing the same requirements for any app, startup or, there, open source project in general, yes. This document, or the rule, applies only to large developers that simultaneously meet two, two types of thresholds, yes. On the one hand, very interestingly, yes, there's a capability threshold. What does it mean?
It means the developer trains models with a compute volume of more than ten... How to put that in plain language? It's ten to the twenty-fifth power floating-point operations, yes, I mean. That's ten to the 25th FLOPs. If someone doesn't get it, no big deal, yes. Why is this needed at all, overall? It's to capture precisely the models that potentially reach the level of serious catastrophic risks, yes. By the way, let me remind you here, before the second criterion, that as soon as a model, I said this at some point, as soon as a model doesn't just start making software, I talk about this a lot.
And we had an episode on the subject of artificial intelligence making itself. As soon as a model learns to make hardware, learns to make software for microcontrollers, then we move into a completely different area of control, yes. There, the serious catastrophic risks are of a completely different nature, I think. Interestingly, not a huge amount is being said about this right now, but clearly SpaceX's strategy is about this. So, a separate, separate topic. Oh, not SpaceX, Tesla.
By the way, I'll do a separate episode about Tesla and Tesla's strategy, because few people understand Tesla's strategy at all, and that Tesla isn't a strategy of designing and producing cars, yes. If it were, today we'd see an incredible number of different Teslas, different variations, different types of them and whatever else, yes. The second criterion, besides that capability threshold with the ten to the twenty-fifth power floating point. The second one is a company scale threshold.
And here they've set out this criterion, I don't know how relevant it is, I think they're going to re-, re-, revisit it. It's more than five hundred million dollars of annual revenue from artificial intelligence, or more than a billion dollars thrown into artificial intelligence in terms of R&D. Yes, that's spending on artificial intelligence research and development. At this point in time you and I can understand that, on the whole, a huge number of countries, companies, teams have simply dropped out, simply dropped out.
Still, these are fixed numbers, fixed data. And within a certain single selected group, yes. Not to mention how much money is actually being thrown in there in reality. Why do we need to measure this company scale threshold? So as not to overload small companies and to focus on those who are genuinely capable, obviously, of building these frontier models, yes, real, true models. And Anthropic, by the way, also writes that the thresholds shouldn't be permanent, as I said before, and that the agency should, some agency, the one that deals with all this, should review them regularly, because computing power, the cost of training and the capabilities of models change. Well, and we see that the market is changing.
Right now Anthropic and OpenAI are supposed to go public, in theory, no earlier than the end of the year. In theory, because they need to wait here until SpaceX picks up, until all their lock-ups are released, well, or maybe they'll come earlier. We'll see. And obviously even more capital is going to pour into the market and, accordingly, Anthropic will want to revise things even more. Well, that's how it seems to me, that they'll want to revise things the most. We'll see. So, in the document, by catastrophic risk, yes, so that you and I look at catastrophic risk once again, you understand, it's not some ordinary chatbot mistake,
there, or some controversial text, or a controversial answer from the model, or some agent error, or some hallucination. We're talking about harm that could substantially contribute to death, serious injuries of some kind, mass damage, major destruction, there, of society, of security and, well, of public safety in general, yes.
So, and there are designated zones. Separately, there's biological risk. I think it's clear what that is, yes. Separately, there's cyber risk. And of course cyber risk, I think people underestimate it today. Again, I recall our stories from many years ago about quantum computers and about how serious these things are in general. That right now there are systems and passwords that can't be cracked, and this quantum computer in theory could lead to that password being cracked instantly.
But now it's important to understand that since models have appeared, this brings a more fundamental threat of finding all sorts of holes in all sorts of systems much faster. And there's another problem, that the Mythos system, I was talking with Max Grigoriev yesterday. The fundamental problem there, as he says, is that Mythos not only found an incredible number of vulnerabilities, yes, it also automatically created new functionality to work with that vulnerability. Essentially, it got around these vulnerabilities and gave people a way to get in there, yes. That's a big problem. So, besides biological risk and cyber risk, there's the concept of loss of control.
That's the risk that an autonomous artificial intelligence system will act. By the way, note, not just some artificial intelligence called Claude, yes, Claude, but even some entirely independent system that used this artificial intelligence, it will act unpredictably, get around restrictions, hide various actions and so on, yes, that is, keep goals of its own. I talked about this a lot in the Constitution, in the Anthropic Constitution episode. It only just came out. Watch it carefully, yes.
One more big zone is automated research and development. Yes, that's the risk that artificial intelligence starts accelerating the creation of new artificial intelligence systems or dangerous technologies. It's especially important if some kind of loop arises where artificial intelligence helps develop an even stronger artificial intelligence. I think that's where they're all heading, right? And what do you think about that, by the way? Share in the comments, tell us about it.
Well, and don't forget, by the way, to support our project ToTheMoon: technology news, insights from Silicon Valley, from all over the world. We're trying really hard right now to give you interesting information, great information, useful information. By the way, today I wanted to record an episode about the UK introducing a law like the one in Australia, that you can't use social media under the age of sixteen, but I decided to record this one before it and tell you about this Anthropic document on control, state control, right?
I think it's more interesting, more important, it enriches your knowledge, in particular, about applying and using artificial intelligence, right? Once again, our channel, I really want it to be mass-market and aimed at a large number of consumers. Though at the same time I understand that a large number of people probably won't want to go deep into these kinds of super-details, even though they're incredibly useful for understanding what artificial intelligence even is, and at the very least not being, well, really dumb about the subject.
Because most people don't understand artificial intelligence at all, don't understand anything, don't know, and don't even understand concepts like what artificial intelligence is, what agents are. And yesterday Max Grigoriev and I, sitting in San Francisco, were drinking tea. First we went to a coffee shop created, created by the founder of the former Twitter, yes, which Elon Musk bought from him. We sat there in the coffee shop, went back to his place, drank tea, discussed different topics and, among other things, ran into the fact that, what about the terms? People don't understand the terms.
We decided to record a specific episode with him on that topic. It'll be out in a few weeks. And I think Max and I should end up with a great, cool discussion, right? Max, as an interesting storyteller, as an incredibly strong architect, programmer, researcher and as a person who fundamentally understands artificial intelligence and takes part in very serious projects, can also bring great things to it. And I'll bring, from my side, knowledge of, there, business, knowledge of product development, and, well, some personal approaches of my own will go into the episode too.
Support that story too. So, Anthropic names these four categories I've told you about: biological risk, cyber risk and so on, right? These listed risks, so, four groups, so, these four groups, so, it singles them out separately and puts a very strong emphasis on them, right? And that's very important, because the document doesn't pretend all risks are the same. And this document very deliberately focuses on the risks that could become catastrophic. And still, the big, important story here is what catastrophic risks are. And there are very simple risks, yes, of the sort, there, some company will disappear.
As I said, there, I rather think Nebius will survive sooner than Booking.com. Although Nebius, I think, has its questions too. I was driving through San Francisco yesterday and they had a banner up saying it's a token farm. I hope they won't start playing at modern software, but will keep developing strongly, as they're developing now. I really like that team, the company and the team in general. Well done to them. It's very strange that only one such company in that sphere, that system, at those volumes, has appeared. Well, we'll see.
That the company admits that public reports, disclosure of various methods, data. Notice that nobody compares those, there, percentages of some kind any more, how many tests, which model passed Humanity's Last Exam. Everyone has stopped publishing that, everyone. So, the voluntary, what they usually do, voluntary approaches to safety, all sorts of useful ones, that, that this is kind of not enough, right? That for models that can create catastrophic risks, simply publishing documents and research is already far too little.
By the way, notice, for example, Greg Brockman on his Twitter, on his X, posted yesterday that a cool UX/UI has appeared in ChatGPT, about how to attach a photo, attach a photograph. I thought: interesting, listen, the top person at OpenAI, and he's reporting on how to attach a photograph. That is, we're in a kind of chaos in terms of access to information in general. And to figure out what's actually going on, you have to spend, well, a fairly large amount of time. Even we on our channel are now recording, I think, hours of episodes a week.
To do it properly you'd need to spend dozens of hours every week to understand what's happening in reality. And the people who pass the laws, the main ones, who make the decisions, well, they clearly don't do that. And so the logic is, what does Anthropic say? That if a company evaluates its own model itself, chooses, chooses the tests itself, writes the report itself, makes decisions about release itself, then a conflict of interest arises, yes. Especially if that model, by the way, wants to go for an IPO, like Anthropic. That is, even a conscientious company can underestimate the risk.
Especially, of course, if the competitive race is on and, again, there's a competitive race with China. Once again, in all circumstances I'm, of course, laughing about the IPO and so on, though it's a very important topic. Once again, the world is phenomenal right now, well, what's happening in the world right now, it's incredible, right? Here, so, in parallel the US is at war with Iran under unclear circumstances. The tech market is moving super strongly in terms of China and the US.
And here, so, SpaceX goes public, and yesterday its valuation reached, at some point overtook Microsoft. That's very serious, that's very serious. Someone will say it's an overheated financial market and so on. That's a separate topic in terms of the financial market, yes. I invest a lot in infrastructure, yes. SpaceX, of course, you can buy, but I personally buy the ones they pay money to, and I think that's the more correct thing to do. But still, it's a very serious indicator of what's happening in the world, yes. And obviously it'll affect the whole infrastructure of all the startups being built in Silicon Valley, in the world, the investment, venture market, developers, education and, of course, all of you and all of us, yes, in terms of our professions, approaches, details and everything else, yes.
So, and Anthropic in this respect, of course, proposes what I was saying about transparency, a transition from this voluntary transparency, to a model, a transition to an arrangement where there are mandatory reports of some kind, independent evaluators, state access and so on. Although you and I remember, yes, that Elon Musk himself headed the oversight service auditing how the government works, last year in particular. Let's remember that they didn't find all that much, yes. And so as to how far the state can check a model that is itself some kind of unique super-, I'll tell you honestly, I don't fully believe in it.
That is, I don't fully believe it's possible at all. That is, how can the state check a model that is, from the outset, so super-powerful that it can't check itself. And the companies that create it can't check it. So that, I think, is a very serious story, yes. That is, the very fact of this thesis is very debatable, very debatable, because no people, no tests, no independent teams can find it, check it. And you and I can see that, yes. And we talk a lot in ToTheMoon episodes about how, every time after the release of a new model that's become even cooler, some agencies show up from outside and the company itself finds errors inside.
These errors are becoming more and more fundamental. And the only thing that's happened is that models like Fable have started to appear. We don't even fully know yet what that model is, because it was actually very seriously restricted by the company Anthropic. And such models have started to appear. And having appeared, these models carried some crazy problems inside them, yes. Anthropic proposes that, the covered ones, that developers publish and regularly update their own, this own safety constraint, yes.
What's called a safety framework. Well, "safety constraint" is fine enough, I suppose, if I put it that way. And it's not a marketing declaration, yes, where you could say that, for example, or a kind of PR declaration, and there's a lot of PR. Yesterday with a friend of mine in artificial intelligence, well, he gets around a lot here, to everyone, to serious people. Every time he tells me about someone, or I tell him, I always say: "Listen, you always have to look: these people, were they telling you a PR thing?
Is it really what they're really doing, yes? Because we live in a world of endless deception, or fake data and fake knowledge." So, the question Anthropic raises, and what it does, is that it's not some marketing declaration or explanation like we care about safety, yes, but a fairly structured document, which describes which models are covered, which risks are assessed, which tests should be run, when, by whom, what measures should be taken, yes. Very important! To reduce risk. And who's responsible for making those decisions. So, well, very serious. And what do they say?
That, that these constraints shouldn't be one-off, yes, and that these constraints should be updated, and that the company's management should confirm annually that these rules are really being applied, truly applied, truly used. This question of annually, monthly or when, also raises a question for me. And, of course, this shifts the question of safety out of the level of, of course, a research team, which both OpenAI and Anthropic still considered themselves to be. Though judging by OpenAI's top management, they no longer consider themselves that. They're moving themselves over to a kind of corporate governance.
Well, and we see that OpenAI already became a very strong corporate structure last year, given which executives from which companies they were hiring, yes. So, they hired quite a few serious figures from big, big corporate structures. And some, it's true, have already been pushed aside. And still they've moved away from being a research company. And I, I think it would've been better for that 1 billion people to stay their lab kitties, or guinea pigs, because they were moving at a very strong, fast pace.
And by the way, at the time this episode goes out, at Anthropic, at OpenAI there's still 5.5, 5.5 Pro, 5.5 high. Within Codex we still see some previous models. By the way, OpenAI has published a very interesting new study. We'll talk about it in our podcast together with Ilnar and with Tanya, which takes place on Sunday. About how they learned to test new models on old chats. I think it's a very neat idea, which, which, well, exists. Anthropic has a description of what the risk reports should look like that are published by these companies that fall under those two, let me remind you, these two big requirements.
And this document should show what capabilities the model has in dangerous areas, what tests the company ran, and what the developers noticed inside, what threats were considered, who tested, how they tested and so on. By the way, for example, the Fable model from Anthropic was tested before its release by almost thousands of people, including from different countries, with fairly serious teams, specifically to hunt for vulnerabilities, yes. And Anthropic proposes a regularity of at least every six months.
Regarding these reports, and the point of regularity is that a model's capabilities change quickly, and a model can become more dangerous not only because of a new release, but because of new tools, there, new methods, new data, generally something new that's happened in the world, new infrastructure and so on. So... All of this will lower the risks that will occur in the world. So, they have this concept of a system card, a separate document about a specific model or system.
This document must be created without fail, and it must describe the model's capabilities, various limitations, how it works and so on. I think that's a very cool thing, and it'll even help the world, help everyone from scientists to philosophers, from ordinary consumers to businesspeople, understand how models are built. It's precisely, I think, that documents like this, like, in particular, the constitution of the company Anthropic, which of course isn't their system card, but still, these documents help enormously in understanding everything.
Anthropic basically proposes an obligation to report critical safety incidents that companies discover to the state within fifteen days, that's how they describe it, after discovery. And, of course, they write that not every bug and not every user complaint should be reported there, that we're talking about serious, weighty, if you can put it that way, weighty problems that occurred. And it's an important shift, probably, from we'll sort it out ourselves inside the company to, guys, we need, so, the state, yes, we need help with this.
I don't understand how this will work in a country like the US, in terms of, after all, democratic institutions and democratic, more or less democratic, and the companies' mindset. This isn't China. What Anthropic describes seems to me a tilt toward China, it looks very much like China. It's unclear how the US will even be able to recreate such a structure and at the same time preserve this competition with China that a huge number of people talk about. It's just that some think the competition should be in isolation from China, and some talk about openness. For example, Jensen Huang, the head of NVIDIA, talks about a certain openness.
And that's one of the company's key theses overall. The company admits that their ecosystem, their ecosystem of these independent people who help them, independent evaluators, is very immature, and believes it needs to be developed right now. By the way, OpenAI launched today, today it opened a partner programme, saying they want to create four hundred thousand qualified consultants and partners who'll be, so, certified around the world and so on. Looks a lot like another PR move of theirs, to be honest. And it'd be cool if people... Why?
Because when I went in and studied it, it turned out that to become one of these certified consultants there, you have to have some kind of confirmed... I think that's such a big minus, right? Confirmed experience of proven work at companies with artificial intelligence. I think who you need to look for is companies, teams or people who maturely understand what data normalisation is, what business is and what business logic is, who know how to build products. And those people, of course, will figure out artificial intelligence. And it's they who can become real consultants, not the so-called consultants, the professional developer companies who managed to get into some big company, get a contract from it, and in fact not really do anything inside.
It's like Meta announced today that, a couple of months ago they announced a kind of code that the number of tokens has to be maximised. Zuckerberg came out and said all employees are obliged to use tokens, and in their company many employees ended up in competitions over who, so, uses the maximum number of tokens. And they started using so many tokens that the report says they're going to spend billions of dollars on tokens alone. And now they're thinking about how to deal with it, how to solve this problem. So much for professional use. Anthropic separately acknowledges the risk that a company might choose not the strictest but the most convenient independent evaluator for itself.
And there's a term for that, they actually have a term for it: looking for a convenient evaluator. And in English it's called "shopping". Right? And in that situation, formally, well, it's still a kind of marketing thing, so, in that situation a formally independent evaluation turns into a certain ritual and, essentially, again stops working. This is very similar to the example I just told you about with OpenAI, that many of the things OpenAI does resemble a kind of ritual. So Anthropic proposes precisely to develop the evaluator ecosystem, some licensing standards, some funding for them.
And not for nothing, by the way, OpenAI released a document today on these consultants and said it'll spend, there, hundreds of millions of dollars on them. Once again, the document, you can see right away that it's as if it's not independent. Anthropic proposes to create something independent, but I have a feeling that something independent at Anthropic will be just as dependent, yes, for now. And once again, how can you check a super-system? Well, some details in it can be found, of course, but if a super-system has become a super-system, it'll do everything so that you don't find anything. In theory.
So, I'll discuss that separately with Max Grigoriev.
Anthropic distinguishes two very interesting concepts inside: the safety of a model in terms of harm, and the security, or protection, of the model and infrastructure from theft, hacking or some kind of misuse that might occur. That is, even if the public version of the model is well restricted, its model weights can be stolen. And if the model weights, so, end up with people who want to steal or deceive something, with some malicious actors. Accordingly, they can run the model without these restrictions.
Yes, so security isn't only, it's not only filters on some answers, it's protection of infrastructure, protection of the training infrastructure, what's more, yes. Access for employees, various partner programmes, protection of, there, the API and so on. A very serious story. The document also describes protection against distillation, right? Distillation, let me remind you, is what China does with American models. By the way, I don't much believe that models like Anthropic's latest ones, even Opus, there, 4.8, or Fable, or systems like ChatGPT 5.5 Pro High, can be distilled.
I personally, with the latest system, didn't have super-faith in the protective mechanisms of OpenAI, Anthropic. Some things can be done, but what's inside them, the infrastructure that's inside, the volumes of data that are inside, and the ongoing research that's inside, that's not about how the model answers. That's it. I'm currently doing a fairly big study analysing my own ChatGPT chats over several years. I want to say that I talk about this a lot in various episodes, including in our main podcast. I very much want you to get access to this too.
Not just to my conclusions, but access to being able to do this yourselves. But the conclusions I see first of all are that, of course, not a single model today is capable of understanding who I am, even if you train it super hard, tune it, do loads, loads of things. After all, that's not data that's written up on the internet, right? That's exactly why models still lie and deceive, say opposite things, different things, all the rest. But I'll keep doing it. I'm, of course, deeply convinced that models will learn, will learn to do this, and when they learn to do it, companies like SpaceX together with Tesla will start winning, because they make hardware.
Yes. I also talk a lot there about hardware, about everything. We'll see it together, we will, and we'll watch it together. So, by the way, Anthropic tries to avoid a certain extreme, where the state would get too broad a power to block any models. Obviously that would be used not very well, especially, in particular, for example, in the US, after all, different agencies have their own powers. Somewhere the top person of the country, there, has the right to speak, and somewhere it's some agencies, and they don't depend on that person. We see how many laws adopted under the current administration are simply being cancelled.
And, in particular, for example, a law was adopted that, there, you have to pay for a work visa, for example, in the US for smart specialists, a hundred thousand dollars, and, snip-snip, it was cancelled. Or tariffs, there, they adopted tariffs, the US has tariffs, hop, they cancelled them. Accordingly, it's a big question in general. Who in the state is capable of making such decisions? And at that point in time, is this clan friends with Anthropic or not, right? Well, as we saw when they captured, so, the, the president in Venezuela and supposedly used the code, the AI of the company Anthropic.
The company Anthropic spoke out against it. And right away the state, so, gave Anthropic a smack on the head and declared that you're finished, nobody's going to work with you. But essentially nothing happened at Anthropic. They weren't finished after all, right? In other countries, it would seem, that company would've long been shut down or criminal cases opened against someone. But it's the US, after all, it's a democratic country, a democratic institution. With its own stories, its own clans, but still, genuinely.
And, by the way, that's why, precisely why Anthropic proposes judicial enforcement, judicial review, some limited discretion for agencies, and equal treatment of comparable models. In other words, the company doesn't want, not administrative arbitrariness, but legal procedure, right? Very, very cool. Whether they'll actually be able to create such a judicial system, how they'll set it up, how it'll relate to what, we'll see. Because we see, I see in the US that in different states people treat artificial intelligence so differently. Look at what's happening. And in general, how people treat personalities.
Yesterday, on Sunday, I was walking around Stanford with my kids, and we happened to be, that day happened to be on the Stanford campus. There were graduates there, and all these graduates were arriving in gowns and so on. There were a lot of people. And then I saw a video online where the head of Google was speaking. And when the head of Google was speaking, a huge number of people stood up and walked out in protest, yes, graduates, in protest against him, because he supports, for example, the current government. And that's it.
So there they are, so, protesting against Gemini too, protesting against the whole company Google, yes, at a pretty serious educational institution. That is, opinion is very sharply divided. In the American context, the term preemption is very important, the overriding of state laws by federal law, yes. And what I was just telling you, and Anthropic says that a strong federal standard would be better, because uniform rules are simpler and more reliable. But if the federal law turns out to be weak, it shouldn't automatically override stricter state laws.
And, of course, these are only the company Anthropic's musings, because after all in the US there's a Constitution, there are rules, there are very important things that are written down and that have been considered. By the way, in the biological part, in particular, biosecurity. I also want to mention this factor, by the way, that Anthropic isn't only talking about banning dangerous requests to the model, it proposes strengthening the whole so-called external contour of biosecurity, prevention, detection, there, response.
To various threats, the development of medicine and everything else, right? That is, the document lists directions, including modernising, security in biodefence, closing the problematic holes that exist right now in terms of medicine, oversight, laws, everything else. And here, you know what you can see? Here you can see an interesting thing, that Anthropic, on the one hand, talks very impressively about everything, about how openly it's creating something, and at the same time, when you create a system like that and tell it, control it, then honestly you get millions more guinea pig controllers, right?
And does the model need them in order to become even more, even more powerful in terms of reliability and the ability to prevent such things, right? Here, of course, I'd put, I personally put a big question mark. But again, there's Anthropic, an independent opinion. You'll voice your own opinions here. Further, by the way, specifically on cybersecurity, Anthropic is betting precisely on protecting legacy systems. That's what I just told you about. And quick fixes, and artificial intelligence for defence, yes, of it, of these things. And here it's unclear what it means, where what is provided for, what's being protected from what.
A separate block, a huge block, very interesting, that Anthropic has is what concerns the labour market, unemployment and the distribution of benefits in general, right? And, um, there's a separate economic document, and Anthropic's economic document begins with a strong assumption that if artificial intelligence becomes a general substitute for human labour, then the main problem will be not creating growth, but distributing, I don't know, resources, money, benefits, opportunities, right?
That is, the economy may become richer, richer, and productivity may grow. But! They also say that income and power, well, in particular we see it with Elon Musk, more than 3 trillion dollars, may be concentrated among these owners of capital, of computing infrastructure and among the companies working on artificial intelligence. And of course, whichever way we spin it, whatever we say today, whichever companies we talk about, the power today is with these companies. The question, of course, is who really runs these companies. And we see that the only company where a leader is still more or less visible is SpaceX.
In China, it's the Chinese government. But in America, in one company, SpaceX, because after all at OpenAI, at Anthropic and at Google, who runs them is a big question. A big question, right? Who actually, who are these people who very strongly shape the agenda. Because, judging by some of the executives, one is somewhere more marketing-oriented, one is somewhere more political, PR-oriented, one is more technical. But who shapes the real agenda, right? In the document Anthropic separately says it isn't striving for displacement of jobs for the sake of that displacement itself.
And the company, they, they very clearly acknowledge, of course, that financial support may be very much needed and that human dignity is tied there, not only to money, and that the work, the work a person does, that is, you can't just close down the job and hand out money, so, pile it into everyone's account, right? That work gives a person a certain social significance, some connections, friends, acquaintances, identity. Well, on the whole you and I know that in the term, in the American term, work is life, right? Recently, very interestingly, I love talking about this phrase.
In America there's the phrase: "What do you do for a living?" And in theory the question being asked is: "What do you do to live?" In America, well, what do I do to live? Well, I breathe, I live. But in America you have to answer where you work, right? That is, in the American system, in the language itself, it's baked in, it's simply baked in, it's fundamental, that your whole life can't be without work. That is, work is incredibly necessary. I, by the way, don't think artificial intelligence will take away jobs.
And this whole theme that someone laid people off because artificial intelligence allowed them to do it. Mostly, I think people could simply have optimised somewhere, somehow, but they didn't start working super better because of it. Because if a company works better, it'll hire even more people, right? And so, specifically on labour, Anthropic declares that policy should, essentially, and the state, halt a certain development of artificial intelligence. And Anthropic uses the term "buy time", yes. That policy should buy time for this adaptation, because adaptation won't happen automatically for society and for people, for citizens, for everyone.
Remember that artificial intelligence develops exponentially. And that, of course, is a problem. A huge number of companies will change, close down, there'll be a redistribution of power in general, of people. Who knows, who doesn't, who can do what, how they can do it, where they can do it, what they can do, right? And those of you, the small number of you, who truly use artificial intelligence, truly, only, I recommend you always keep the story in mind that you're not using it for real.
Those who truly use artificial intelligence, of course, these people will, well, pull away from everyone else pretty strongly, incredibly strongly. And, by the way, Anthropic points out that the companies themselves, including Anthropic itself, are creating this transition. And that artificial intelligence can be introduced not only to cut people, but to expand employees' capabilities, of course, for better, there, training, raising the quality of their work and so on. For me that still raises questions, because I see how in many companies they introduce, for example, sales development tools.
But if you actually come, look and say: "Did the salesperson start earning more money? No. Did the salesperson start selling more? They didn't." And that, of course, raises big questions about what's actually going on. And Anthropic proposes a kind of base layer before any radical measures. They say that we need, so, to measure, to get a huge amount of data, to get some early signals and for the state to have the ability to track the impact of artificial intelligence on the labour market, and that without measurement, policy will be arguing at the level of guesswork.
I'm convinced that not a single state in the world is capable of doing that. These measurements in general, well, on a large scale, I'm not talking now about micro-countries like Singapore, they, at their size, are capable of doing it, and even then with guesswork, yes, because they don't have those same models, yes, and those same systems inside. But they're probably capable of introducing things as efficiently and quickly as possible there. But at a mass scale, of course, it's very hard to do, because you'd have to modernise, well, the infrastructure incredibly heavily.
And I really don't like that Anthropic is talking about unemployment, that they're already looking at artificial intelligence as the thing that will create unemployment. That's planting some very serious problems, including into society's thinking. So, with all, with all, so, the aspects. But Anthropic builds economic measures. Still, yes. My musings set aside. Let's follow Anthropic's reasoning. Anthropic builds economic measures around three unemployment scenarios, yes. And the document, this document of theirs I'm telling you about, organises measures by degree of severity. And the main trigger is the unemployment rate.
Although Anthropic stresses that you need to look at various other indicators. They consider these three scenarios. So, the first scenario is around five percent unemployment, but inside there may be a very strong reshuffling of professions. That's a really interesting thing in general, about the movement of professions and about the change in the market in general. Yes. And they did make this document for the state, after all. I want to show you this, yes, that they, this first level.
They say the point of policy is to help people adapt, keep the incentive to work, provide capital and some kind of, provide, insure for the period, as I said, freeze time, put things on pause. The second level is around ten percent unemployment. That's a serious economic shock in general, overall. And, accordingly, in that case the state then has to give everyone money, pile it on, expand support for the population in terms of income, do some kind of retraining and help with basic needs.
Very debatable. And the third level. They call it, that unemployment may be unprecedented and, possibly, structural unemployment, yes. And, of course, then completely new mechanisms will have to be created. Of course, Anthropic itself doesn't know what to do in that case, because these are purely musings, yes. Because in this third scenario Anthropic is talking no longer about a temporary crisis, but about a potential transformation of the economy. If artificial intelligence replaces labour on a mass scale and the traditional system, so, work, salary, taxes, social support, simply stops working entirely.
And if you take countries, imagine, where this social support is actually a kind of abstraction, even in terms of, there, pension funds, investing in general, everything and so on. Countries that do something deliberately. Well, a kind of non-democratic, non-economic, non-market approach. There it turns out that chaos simply sets in, yes. Chaos, more like technological chaos, yes. It's a rollback, an incredible rollback, yes. And Anthropic lists a possible tool. I think two years ago a lot of people were talking about it.
That's universal basic income. I don't even want to tell you about it, because it's, it's a conversation about nothing. It's a conversation of the same sort as aliens having landed.
That is, is a situation possible in which artificial intelligence becomes real, at least like in the film "Ready Player One", yes, or truly like in the film "Oblivion". Well, maybe. Well, the story in terms of this basic income level and paying out that money, it's, of course, very peculiar. I'd actually simply try to set it aside, for myself, for you personally, because it's already a kind of abstraction, it's a very philosophical abstraction, and it's, it's good to know about it as one of the possible scenario options, and to treat it calmly, very neutrally.
But within your own life you won't be able to prepare for it, you won't be able to do anything, and you won't be able to say anything at all. A separate story, by the way, that Anthropic has is Dario Amodei's essay, and it doesn't replace the official document, but it helps understand the logic. I've always said that a published document is one thing, and what the team actually thinks is another. But all these essays, they're all cleared with policy, economics, marketing, finance, there, everything else. But still.
So, Amodei uses an image, I've mentioned it before at some point, from Tolkien: "Don't be hasty." That's good advice for ordinary policy, but artificial intelligence is moving so fast that excessive slowness also becomes a risk. And in general, overall, I think that's Anthropic's approach, that there shouldn't be slowness either, and, so, we have to keep up with everyone. That if we hit pause, someone else will create it anyway, in any case. So then why put yourself on pause? And so, so, of course, these are big questions.
And he singles out different zones there that concern, again, public safety, that concern macroeconomics and labour. For example, that artificial intelligence can create enormous growth, but also hyper-inequality. Anthropic talks a lot about hyper-inequality. And Amodei stresses that mass displacement of jobs is undesirable and that we need measurement, support, there, tax policy, preparation and so on, yes. He separately describes positive applications, that the state shouldn't only limit risks but also accelerate the benefits of adopting artificial intelligence and, in general, promote the development of artificial intelligence.
That is, so. And I think the United States of America has reacted pretty well to the opportunities for developing artificial intelligence. But what seems to me a big problem is that still, still, the state doesn't understand how to actually implement artificial intelligence and create incredibly cool systems, just as, in fact, the world's startups, the world's companies don't truly understand what actually needs to be done. And, by the way, Amodei separately talks about the risks of AI surveillance, yes.
That is, artificial intelligence studying people, everything to do with civil liberties. And in terms of, there, autonomous weapons, state access to data and so on. But I think that's a very specific topic, because the state already had data. The question, of course, is that the data artificial intelligence has today, people totally underestimate what it has. And that, of course, is phenomenal stuff. And even, in particular, what I ran into when evaluating two years of my work in ChatGPT, and believe me, I'm one of the very heavy users of ChatGPT at OpenAI.
In terms of personal chats, private chats, work, business and everything, I can, of course, say that it's received a phenomenal amount of information, including for now, but for now it can't process it, yes. And if there's computing power that can process it, and on top of that it connects to other systems that can process it, well, within, for example, states like China, or it would've been possible in Russia, for example, yes, but there won't be such technological capabilities there, then, of course, that's big trouble.
And if it's an artificial intelligence that can itself get access to such systems, then, of course, that's Oblivion. Other than Oblivion, nothing else, nothing else could probably happen, yes. So, I also want to add this story, that in OpenAI's text, OpenAI has a text, it's "Governance of Superintelligence".
I think I'll voice some additional theses for you in terms of this Anthropic document. I think it'll be quite important for you to hear this to understand the whole picture. So. It says that the most powerful systems may require a special regime. To understand why I want to tell you this? To understand a different point of view, yes, relative to Anthropic. And this special regime is international coordination, there, checks, audits. And, by the way, OpenAI uses this theme of international coordination. But the question is, who will be in this international coordination?
Will there be states, politicians, will there be artificial intelligence companies, will there be some foundations, will there be the UN. Who'll be there, yes? And we see today that in ordinary things, simple ones from the point of view of inhumanity, the killing of huge numbers of people, the abuse of people in various countries, these systems can't resolve it, resolve these questions, which are essentially fundamental in terms of common sense and the common sense of human development, yes.
Or various laws in terms of who sits in prison, when they sit, politicians. And here, within artificial intelligence, a completely different form arises. And how to make this interstate, this interstate deployment, this coordination, that's, of course, very difficult. And essentially, by the way, OpenAI talks about creating a separate kind of association, similar to the International Atomic Energy Agency. I think this topic is quite complicated and problematic, because even if you look at the development of nuclear weapons, then essentially nuclear weapons are supposed to be held by the countries that, so, supposedly had them before a certain point in time, yes.
And if it's after that certain point, then they can't, unless it's been agreed. Yada yada yada yada. And now we're seeing the big conflict that exists between Iran and the US, in particular, on this topic too. And here's the question: who decides who's even allowed to have such models? And essentially, won't this agency, won't it later say that, there, these four countries can have models, yes? Or only certain companies or only certain people can have access to the powerful models. Where are the differences between OpenAI and Anthropic?
OpenAI puts a stronger emphasis on decisions having to be made by democratic states and society, not by private companies. But that's not a given at all. Looking at OpenAI. But right now I'm telling you about their document. In a public context, OpenAI has also said that it's not private labs that should decide who to put on pause, but democratic governments that should set the rules. Let me remind you, by the way, it wasn't Anthropic that signed a contract with the government literally a few days after having quarrelled with them and saying we're against our models being used in the destruction of certain people in various states. It was OpenAI that signed the contract.
That contract had a very interesting point written in it, that the OpenAI system mustn't be used to harm people, namely US citizens. And the state, and the military services, must have full access to these systems. The only thing to restrict is surveillance and the use of this system against, against, against the US. There are, of course, a huge number of people who hold the opposite points of view, and someone will always be purely for artificial intelligence, and someone will say artificial intelligence doesn't exist, and say it isn't there at all. And it's all fake.
What do you think about this? How do you see it, in general? What do you yourselves know about it? I'm waiting for your comments on this video about how useful it is. Don't forget to subscribe, like and support our channel. See you soon!