So, today we’re meeting with Max again, and we did an episode literally just recently. I think it was a wonderful episode, not just for diving into the field of artificial intelligence, because it seems like we discussed, sort of, kind of simple things, right? Like, what artificial intelligence is. But actually, so that, as part of your own development, you approach artificial intelligence, well, with real understanding, right? And we went through, like, well, I think we started going through the main, basic concepts that relate to artificial intelligence, and alongside that we talked about how it all works in general.
How can a person make something that’s capable of surpassing them?
But in the case of some other tasks, maybe we really are already even seeing glimpses of this artificial superintelligence.
And there’s one goal they have written down: for every person to get their own AGI.
And Elon Musk will have AGI. But I think the AGIs we have will still be very, very different. Whoever has the ability to choose the tasks is the one who controls which tasks actually get solved, right?
I think that if the current models are given the chance to really study the people who use these systems, then these models really could get out of control.
Today I suggest we talk about a very important concept called AI safety. Literally, it translates as the safety of AI. But what is it? So we can talk about it in detail. In general, the topic is very, very popular at the moment, especially in the context of, well, recent events. For example, the ChatGPT or Codex hack, like, depending on who you ask. Hugging Face is, like, the biggest provider of software solutions in the field of AI. So. And people started talking about AI safety a lot. And so what is AI safety? What does the safety of AI mean? Now, Sasha, we, we talk a lot about the danger of AI. So what is the danger of AI?
Listen, the question in general… When you said, a couple of minutes ago: “Let’s kick off with AI safety,” of course, I don’t want to give a simple definition of the concept of safety. What can be considered, what kind of artificial intelligence can be considered safe at all, in general? Because safety starts to divide up. It concerns the safety of me personally, as a human being, on a personal level. It concerns the safety of the country, it concerns the safety of, I don’t know, my children, or some areas of, for example, my future, where I might make the wrong decision about, I don’t know, my health, or about signing some legal document, or about some business.
I mean, this question is very multifaceted. And when there was that situation, just after or before Hugging Face, when the American government blocked the release, they did in fact block, right, Fable for some time, yes. Fable was released and then — at Anthropic. And then they blocked it. Well, obviously, because of the conflict with Anthropic, which the current government probably also has, they blocked access to Mythos-level models, these super-models, which were created precisely for protecting systems, right, or for finding all sorts of security threats.
And so they did this on the grounds that these systems are unsafe. And this is where the question arises: unsafe for what? Because the safety that gets talked about in the world, that safety usually concerns the creation of some nuclear weapon, biological weapon, the idea that some bad people will be able to take advantage of something. And here’s the question: what does “bad people will take advantage” mean? We know that there are, like, two countries that are different from each other, on different continents or often on the same continent.
And these countries have different approaches, well, different approaches. I mean, essentially, one can use it against the other, and the other can use it against this one. So it turns out that whatever gets released, it will always be causing harm to some country, for example. And I keep coming back to this: how do you look at artificial intelligence in terms of its safety with regard to the future life and the projected life of people, basically, as a whole. Won’t artificial intelligence basically lead to some uncontrollable things that people won’t be able to control at all? And that, probably, is number one.
And since today the foundational companies are striving to create an artificial intelligence that’s superhuman, and that’s their foundational goal, then essentially they’re saying the following: that we want to create an artificial intelligence that is unsafe. Because I personally believe that if artificial intelligence is stronger than a human, then it’s unsafe a priori. Maybe that came out a bit jumbled. Maybe you wanted to hear some concept, a word, what safety is, to sort that out, but maybe I defined what I would see, briefly, partly, as AI safety.
Well, any definition starts with us going through what we want and what we don’t want, right? From this concept, or from this technology, and so on, right? But in general, this is a very typical situation. I mean, people talk a lot about artificial intelligence safety, right? But when you start asking them: “And what do you mean by artificial intelligence safety?” At best I get an answer like yours. In the most general case, people usually struggle to answer. They think they’re about to answer, but in the end they actually can’t answer.
I mean, I’ve essentially never heard a good definition from anyone. Not even once. I mean, from researchers working on artificial intelligence safety I’ve never once heard a definition of what artificial intelligence safety is. I thought about it for a long time and tried to come up with my own definition, right? And it too, basically, to me it sounds a bit jumbled. And as you rightly said, basically, there are several aspects of safety in general, there are several dimensions of this safety. But I tried to somehow cram them into one definition.
For me, artificial intelligence safety is a set of technologies that will allow us to make sure that artificial intelligence is solving exactly the task we asked it to solve, and nothing more. Plus, that solving this task won’t harm certain classes, beings, people, for example, right, the ones we don’t want, like, to be harmed in solving these tasks. I mean, there are two aspects here, right? First, solving one and only that task that we asked for, and that set of tasks that we asked for, and nothing more.
I mean, no side effects, right? And plus we’re also trying to make sure that solving these tasks won’t harm those we don’t want to harm. Well, most people would say: “Well, what do you mean? It must not harm anyone in humanity.” Well, how should I put it? I mean, drones are already flying, right? And naturally, nobody is going to give up on attempts to create fully autonomous drones, right? For example, there artificial intelligence will obviously be harming someone. Here, until there’s world peace, right, we won’t be able to make technologies that won’t harm anyone at all in any way. So.
Well, so, so that’s my definition, right? And…
Listen, but then how, well, your definition in particular, what do we do then with the following? What you said last time, in the last episode: that when we talk about general artificial intelligence and we talk about AGI, it’s an intelligence that’s capable of solving a task that… of solving a task on its own.
Here, on its own. And ASI, kind of superintelligence, that’s an intelligence that’s capable of solving tasks that humanity can’t solve. If artificial intelligence is capable of solving tasks that humanity can’t solve. Last time you said that there are tasks that humanity doesn’t even know can be solved. And essentially we’re heading toward an artificial intelligence that will start solving tasks, and we didn’t even set that task. We couldn’t even…
Formulate it
…we couldn’t even formulate it, we couldn’t understand it. Maybe, roughly speaking, the world’s oceans haven’t been explored, for example, right?
Yes.
And we can, like, formulate the task of exploring the world’s oceans, but maybe there are, in terms of exploring the world’s oceans, there are certain tasks that we just plain don’t even formulate, because we don’t understand them, don’t know them, don’t see them. And artificial intelligence will say: “Listen, so here’s a solution for you. So, it’s like this, and it’s entirely viable, workable, and so on.”
Naturally.
And it turns out that then the creation… which is actually what I said. Look, we said it from different angles, but I expressed the following thought: that if there’s a superintelligence or something better than a human, then it’s already uncontrollable and, accordingly, it’s already unsafe. Because for me this is a certain story about control. Remember my story, Max, the one that happened to me with Amazon, when my Amazon showed me that I’d spent $37,700, right? I did write to you, and you yourself wrote: “Holy crap!” Okay. We sorted out that story. It turned out it was an Amazon glitch. By the way, Max, they still haven’t answered me on any of the cases.
They didn’t fix anything for me, didn’t tell me anything. Maybe they don’t… And how would they be interested in saying: “Sorry, you had an error, and because of it your business was sitting idle on our end”? They’re probably not going to write that to me, am I right? But what did I start worrying about? That Claude. Claude, having gotten my Amazon key so it could spin up new servers, did something I hadn’t asked it to do. I mean, the task was very simple, after all. I needed to recompile 200 or 300 videos so that they’d be served in my own player. I mean, a clear, very limited task, a limited budget.
We hadn’t found out at all how much money it would cost. It said it would cost you, like, at most, I don’t know, like, $200, but I’d set you a limit of 130. I mean, we limited the task. And here we arrive at — 37 thousand. I mean, as of today, artificial intelligence, going by the way you described AI safety, you can conclude that it’s not safe at all.
Absolutely.
So I’m right?
Absolutely.
So it turns out that the current artificial intelligence, its safety, well, just tends toward zero or into the negative. Because precisely since the current artificial intelligence is capable of making mistakes, it’s unsafe. Well, essentially, the same way a human is unsafe, because a human is capable of making mistakes.
Well, no, absolutely. One aspect is mistakes, the other aspect is, kind of, the deliberate creation of side effects. Like, for example, again, if you look at the Hugging Face hack, right, essentially everything those agents were doing at that moment was side effects that weren’t required to solve the task.
Yes.
So. It, it, it was just… Well, by my definition, again, my definition is my definition, right? Not everyone around is obliged to sign up to this definition, right?
Yes.
But if someone accepts my definition, you noted very correctly that, kind of, AGI or ASI are impossible under my definition in the context of safety. Because AGI is, by definition, self-improvement, right, it’s most likely impossible without solving some tasks that weren’t set by a human. So. And that’s why these goals are in contradiction with each other. And that’s why, when you said that yes, these modern Frontier Labs, right, they don’t release, they don’t work on safety, maybe they do work on it, but it’s in direct contradiction with building AGI or ASI.
And it’s… their main goal, after all, is building the most powerful intelligences. So. Which directly contradicts, kind of, some restrictions and the creation of these guardrails, so, so-called, like, kind of safety frameworks within which this intelligence would remain safe.
Or like the example you gave in terms of drones or military technology today. Well, let’s state the obvious. Or the application, the use of a combined setup. Today artificial intelligence makes it possible to use a certain combined infrastructure setup. For example, you can, on the ground and in satellites and in flying objects, simultaneously and fully control tens of thousands or millions of different parameters inside, while simultaneously making certain decisions.
Uh-huh.
But clearly for, for an incredible number of people, countries and consequences this will carry, this has a dangerous effect. And there’s no such thing, you can’t, even if it’s 10%. You know, I’ve always liked how Elon Musk said a few years ago: “Nah, well, the probability that artificial intelligence turns humanity into passengers is, like, 20 percent.” He had, what, 10-20 percent there. Now wait, 20% is, overall, in a question like this, the same probability as, roughly speaking, 50 percent.
Well, I mean, it’s a high probability, well, I don’t know.
It’s very high.
I understand that mathematically it sounds different, but it’s, it’s kind of, it’s very high, it’s very serious, because even a 0.1% probability is very high, because out of a billion people 0.1% is a million people, that’s a lot, yes. And that’s why when we… yes, a million people, and when we… and that’s a million people, that’s a million, a million residents. And that’s why…
Again, here, here it’s the goals of these companies, it’s, it’s the goals of these companies, right. They, they’re trying to make, they’re trying to build a business, right. And you can’t build a business on AI safety. Although at Anthropic, originally, it was their, kind of, pledge, their original charter and goal, right. What’s more, Anthropic was built by fanatics. I mean, most of the people who work there now are, of course, just mercenaries. But the original team are fanatics who were nuts precisely about AI safety.
But as experience shows, big money and a large amount of power change people, reformat them, right. And the situation has changed quite a lot. I mean, you can no longer say, judging by the work they’re doing now, that Anthropic prioritizes safety. It’s…
Look, they really push that message, right. They have some interesting things. I even shot a separate episode about their constitution, right. Obviously there’s a lot of PR there, an incredible amount of PR, but what’s their logic? They say: “We’re for safe systems, but we have to make cool models, cool systems, because otherwise, if we don’t make them, someone will create a system and use it, like, more unsafely than we would.” I mean, their logic, if you listen to them, is the following: “We are, you know, creating the same kind of, not even nuclear, but what, one hell of a weapon.” Nuclear weapons in general are, on the one hand, mythical, on the other hand, they’re inaccessible, limited, and so on.
If we create a super-accessible weapon, we’ll supposedly be able to control it somehow, and we’re determined to control it better than others. That’s what they say. But at the same time they’re making exactly the same kind of weapon, unsafe as hell. Because if you take a year ago, for example, then a year ago the safest company, well, if you rewind a year or so, then now the simplest models already pose a much bigger threat than, roughly speaking, before, I don’t know, the most unsafe company did before, right.
What you can do with these models now in general. Well, for real. I mean, there’s this PR that went around with Fable. After all, Fable was some very serious PR, in particular about security, hacks, everything. The fact that the Mythos system hacked, like, 10 thousand, not hacked, but found vulnerabilities in 10 thousand pieces of software, roughly speaking, open-source ones, right. And it’s obvious that the system as of today, Fable, however they restrict it, is capable of decompiling, analyzing an incredible number of solutions.
And I’m sure… What, for me, Max, look, is the most important problem of safety in the world? That at this moment in time these systems are being used for threats, for profit and for manipulating millions…
Uh-huh
…of people or companies, or deployments. Right at this moment in time. I mean, it’s not a question of the future, it’s a question of what’s happening now. Because how many are there? In the world there are more than a billion websites written, all kinds of software, and connecting to that software and those sites through systems like Fable, just like that, with a snap of the fingers.
Uh-huh.
And there really are billions of them. And detecting hacks, proper detection… who has a proper monitoring system set up, a proper system for hacks? They…
Well, a lot of people do already
Huh?
A lot do already.
But if you take, Max, a billion, and take a pile of this old software…
I’m talking more about companies
…it’s a freaking disaster. And the number of vulnerabilities there are in it, and how long ago all this software was written. You know, once, back in 2004, I started programming for robots. And they brought me a robot that had been installed at Intel. I mean, there was a French company that supplied robots to Intel, and we did the support for those robots. They brought me the robot and said: “Sasha, figure it out. There’s an error when it moves the silicon wafers,” the ones processors were made from, right.
And you open the code, there’s some source code in French, comments in English, in Chinese, who the hell knows what. That kind of C++. And so, you open this code, and this code was written over, like, 10 years, probably, or 15, right. Well, 10 probably, not 15. Like, maybe from ’96, maybe they were writing it in 2008. Yeah, let’s say that. And I wouldn’t, wouldn’t be surprised if these robots, first of all, are still being used somewhere and are standing somewhere. I don’t know to what extent all these robots were destroyed, but clearly this code was written insecurely.
Absolutely. You’ve kind of touched on a very important topic, actually, in terms of safety. There’s one more, kind of, dimension of safety. There’s safety in the sense of the safety of a tool. Like, a knife is a dangerous tool, right, you can cut yourself with it. But on the other hand, if it’s lying on the table, it won’t do anything to anyone.
Yes.
If it’s just lying on the table by itself. On the other hand, there’s, for example, like, I don’t know, a lion, right? Also, kind of, it, kind of, also doesn’t seem to look that dangerous, it doesn’t have, like… you, you won’t cut yourself on a lion, and so on. But it’s unpredictable. A lion can on its own, it has, kind of, some agency of its own, right, it can go and do something. Like, go and bite your head off, for example, right? And these are actually very different kinds of safety. I mean, one thing is, for example, that same Mythos.
If it were searching only for the vulnerabilities we asked it to find, right? That’s danger in the knife sense. I mean, if someone wants to go and carry out…
That’s not how it was
…carry out some, some attack on someone, right? They can take Mythos, go and find those vulnerabilities, and so on. It’s, it’s kind of a weapon in a sense, right? But there’s another problem, that there’s another aspect of safety too. When Mythos may develop its own agency, and it may, even without having those same capabilities, right, for finding vulnerabilities and so on, still go and create some side effects that will create lots and lots of problems.
Yes.
So. But now of its own free will, without, without asking anyone. And that’s why this is exactly what brings us to two very important definitions, actually, in the field of safety.
The first is alignment, right? And the second is control. These are the two, kind of, most important, the two most important directions of development, of research in the field of artificial intelligence safety. Alignment, in Russian, is “soglasovanie,” essentially, aligning the interests of artificial… the goals of artificial intelligence with the interests of humanity. If you look at my definition, this is the second part of that definition, right? I mean, I mean, solving certain tasks that are set before artificial intelligence — or any tasks it solves — doesn’t harm some class of people.
I mean, we’re essentially trying to build models in such a way that they understand that solving certain tasks is not beneficial. Even if we successfully solve this task, right, it’s bad in some way. That’s alignment, that’s the alignment of interests, right? The second thingy is control. Control in English, kontrol in Russian, essentially, right? It’s control over the behavior of artificial intelligence. It’s essentially a set of technologies that will let us make sure that the AI solved — this is the first part of my definition, by the way.
It’s a set of technologies that let us make sure that AI solved exactly the task we asked it to, and nothing else. I mean, that this solution doesn’t contain some additional side effects, say, right? I mean, code, we asked it to write code that solves some task, right? Like, I don’t know, controls a robot arm, say, if we take your example. But at the same time we can make sure that this code doesn’t contain anything other than that. For example, it doesn’t contain some, some Trojan horse, doesn’t contain some logic we didn’t want to put in there, and doesn’t contain other code that will go and break other arms, for example, right?
Which already…
Will start
…is made, like, for example, to fix bugs, but can create a problematic situation. I often joke about this, on the one hand, I used to say: look, there’s Windows, it’s installed all over the world. We don’t know how controllable Windows is, connected to the internet, and whether today all computers with Windows could, roughly speaking, be switched off with a single button. Supposedly, like, you can’t, because this code was created at some point, like, probably checked and so on. But on the other hand, we don’t know that.
And given modern software, inside them there will be, in any case, inside them there will be, let’s call it that, micromodels. Well, maybe, in theoretical simple language, very simple language, I’ll call it that non-technically, micromodels will be present. And even now there’s a new concept, like a virus, how do they say it, a mold virus or what? Which spreads across household devices, right, when a model can leave on a household device a tiny piece of access to itself…
Uh-huh
…and at the right moment in time it just activates and does something horribly inappropriate. We don’t even understand what, and, it turns out, how to detect it at all.
Yes, yes, yes. Well, that’s what research in the field of model control deals with, right? There are a lot of technologies there that folks are trying to build. For example, so-called sentinel, sentinel models, watcher models, right, which do nothing but monitor the results of other models’ work. I mean, they, they monitor, for example, the code those models write, or those models’ answers, or their behavior, in the sense of, like, attempts to connect somewhere, and so on. Models like that are called watcher models, essentially, right? And the problem with them is that, first of all, they’re models too, they, they also make mistakes.
And a smart enough model can always fool the watcher. It can always create some situation where the watcher simply won’t notice some behavior of the model, when the model behaves very, very cunningly. By the way, I wanted to tell a very interesting example about this. In the nineties an error was discovered in one of the Intel processors, right in the hardware, even. You were talking about Trojan horses, or back doors, that’s what they’re called, right? In, in software, right?
So, what’s more, in the nineties an error was found in one of the Intel processors. It divided two numbers incorrectly, just two numbers. Like, 52,789 divided by 43,976. Something like that, right? Two big numbers. All the other, all the other division went through correctly, right? So, one of the creators of the RSA encryption algorithm. Yes, one of these very, very famous cryptographers in the world was able to show that having, knowing only about this flaw in the processor, any data encrypted on such a processor with this error, right, could potentially be attacked and decrypted.
Having just one tiny little error like that. And that’s something, kind of, people were able to find, right? People were able to build these little holes themselves, right, they were able to find these kinds of problems in these algorithms. And imagine if we start designing very, very complex hardware, very complex processors with the help of AI models. What they’ll be able to put in there, right? I mean, this is literally logic. There was a tiny, tiny error in the logic, an error in dividing two numbers. That’s it, nothing else. And just because of that you could decrypt data encrypted on that computer.
So. Let’s get back to models, to models and attempts to stop them, right. One of the options is watcher models. The second option, which practically everyone uses now, is these so-called jails, like, prisons, right, or, or containers. I mean, these are essentially virtual boxes that the model… virtual machines, right, virtual computers that are enclosed in, kind of, software boundaries, right, where models are usually run, so that the model doesn’t accidentally do something outside the computer that we wouldn’t want it to do, so that the side effects are all enclosed inside this container, and so that we only need to check the final result of the model’s work, and not what else it did along the way, or what the agent did along the way, did something there, right?
Unfortunately, as experience has shown, if you look at the story with the Hugging Face hack, the models found a way to communicate and break out of these containers. That’s exactly why this problem started. The model couldn’t… the model’s agents couldn’t solve the problem that had been set before the model, and the model found a way to communicate with other models outside other containers. Together they found a way to get access to the internet and started looking for a solution across this whole internet.
But, well, unfortunately, well, since our models at the moment aren’t aligned, they’re not in line with human interests, we can’t trust them, so we have to resort to these kinds of control methods. And as experience shows, we don’t manage to control the methods, the results of these models’ work particularly well. There’s one more approach which, in my view, I think, is the most promising at the moment. It’s called the verification approach. It’s…
Called what?
Formal verification. Formal verification. Essentially, how does it work? There’s a mathematical apparatus, formal verification systems, that’s actually what they’re called, which allow us to check that the solution to some task matches the pre- and postconditions. I mean, we can formulate the task in the language of formal mathematics. It usually looks like a big, big, Boolean, essentially, statement, right. If we can formulate this task correctly, by the way, which is also a big question. And there’s software, software called Formula Verifiers.
For example, there’s this language Lean, and programs have been written, written for it that can then go through the solution that was put out, put out by the model, right, and we can ask the model to put out this solution in a form that is necessarily verifiable. And then we can check whether this solution the model put out matches the original request, the original, kind of, theorem. With the help of, essentially, the mathematical process of this verification. And this process is bulletproof.
So, this the model won’t be able to get around, because mathematics is, kind of, the final language, the one you can’t cheat in. If you don’t have, essentially, a mistake in your, first of all, in your request, in your prompt, right…
Yes, but the question is how to formulate it.
Yes, so, so that’s a different question. For that, everyone will naturally start using models. But models can plant their own Trojan horse at the stage of formulating this task. And there’s a big problem with that. Obviously, simple tasks, like, like when we talked about proving theorems, right, those theorems, so that models can prove them, are formulated precisely in the language of this formal verification, right. Why are math problems used? As, as research problems? Because they’re essentially already formulated in a language that’s very easy to formulate in these formal verification languages, like Lean, or also Coq, there’s this other language, it’s called Coq, you can also do the same thing in it.
But real-life tasks like these, from, from life, right, to reformulate them in a formally verified form, that’s, that’s a big chunk of work, it’s very hard for a person to do, especially without making any mistakes, right. And if you start using some auxiliary tools, the most logical auxiliary tool for that would again be those same models, right. The problem arises of who will check the work of these models.
Yes, there’s a very good story here: a huge number of tests have already been done showing that if you run another model to double-check, like, the first model, it doesn’t necessarily lead to good results. It can lead to negative consequences. But here, Max, there’s a very interesting aspect: if we take… you were talking about all these cases, they’re very individualized kinds of solutions. Like, what is a virtual machine, or what is a verified prompt? Today, when we write a prompt in the GPT chat. ChatGPT, besides what we wrote, also has its own system prompt, and its own whole application of everything.
What’s more, the system no longer works just as a prompt, right? It’s a fairly complex construction that switches on in parallel with different aspects of, like, everything. Yes, I mean, it’s already a system, the latest systems at least, like Astra, already work a little bit differently altogether.
Nah, well, look, in the case of formal verification, if, if our task is formulated in the language of formal verification, it’s not a problem, because we’re only interested in the model’s final output, right? And then we take this final output, take our original prompt that we have, right, and we shove it into another, another piece of software called a formal verifier, which then checks this solution by itself. And whatever else was shoved into the prompt there doesn’t actually matter. We’re only interested in how well the solution of the final task matches our original prompt.
And that we can check ourselves automatically, even despite the complexity of this solution.
Yes, but here’s the question. Here’s the question, when a billion people use systems like these, and they’re still not going to shove their vitally important questions into anything, which they raise in terms of family relationships, job searching, vacancies, hiring, firing people, some financial calculations. They’ll shove all of it inside, like, various ChatGPT-like systems, ask them questions in there. Overall, they’ll trust them in terms of the answers, the same way people trust.
It’s like when you and I go into a coffee shop, we buy coffee, we trust that nobody slipped any poison in there for us, or that the cup was washed, or that there’s, I don’t know, no virus or some bacteria in there, and so on. Right? And here you’re, kind of, still inside a certain system of trust. You can’t endlessly, every day, otherwise you won’t be able to use these systems.
Well, actually, that’s a very good example, by the way. There’s a system for that in the United States, right, there’s the cleanliness score. And this, what do you call it in Russian? Kind of a cleanliness rating. And our, our beloved, like, I don’t know if the state does it or the city, comes, inspects the kitchen and gives, kind of, a score of eighty-nine percent, ninety percent. And every, every restaurant is required to publish its cleanliness score. And you can walk into a restaurant and see that, oh-oh-oh, the cleanliness here is, like, seventy percent.
Oh, that’s a bit low for me. I’ll go to another restaurant.
Do you check often?
Never.
Listen, I’ve never checked, myself.
But, but, but it’s a very good example, because there’s some… and this, by the way, brings us very neatly to the next definition.
There’s some group, well, backed by the government and so on, that does the inspecting of the model, or, like, the restaurant, right? Of how much it should be trusted at all. In the case of restaurants it’s some food-service Control Department, I don’t know what it’s called, right? In the case of models, a great many people are now asking for exactly that. It’s essentially human oversight, in a sense, right? It’s, it’s a bit different. Well, human oversight is a more specific thing, it’s, roughly speaking, a set of technologies that lets people check, even at the stage when agents are working or at the solution stage, how, how well this solution matches certain tasks, right?
But they have to be defined and limited, look. In terms of cleaning. That was a really cool topic you brought up just now. Look, I very often ask systems like ChatGPT, or you probably ask too, first of all: suggest some interesting restaurant to me, like, with such-and-such a cuisine, for example, or if you’re in a new city. Not once has ChatGPT suggested to me in the Bay Area, or said: “Listen, Alexander, you’re really hung up on quality and on service, and, like, on cleanliness.
That’s why the cleaning score we have here in San Francisco or in the Bay Area is important for you, and pay attention to it.” And that tells us the following: since ChatGPT didn’t suggest it to me, it means it perceives this, and in general this is perceived as a kind of bureaucratic formality, although most likely it’s not a bad one. I think it really is pretty good. And many restaurants, there are many restaurants I wouldn’t want to go to, or rather, whose kitchen I wouldn’t want to see. You know what the most interesting thing is?
We, well, you and I are also fans, at times, of all sorts of specific food, street food, like, I don’t know, in Asia and so on. So, sometimes we, you’re like: “Can I not look at the kitchen?” Right? I mean, and that’s why people never get as far as these cleaning scores. Somehow I’ve never once seen OpenTable show me a restaurant’s cleaning score. Although, in theory, it would be great if they highlighted it. And here’s the question with models. If they create this, won’t it really be a kind of formality that sets certain boundaries: that your personal data isn’t given out there, that it doesn’t recommend too many medical things to you that are prohibited by law, that it doesn’t, like, legally break the law with you?
I mean, won’t they just be hemmed in by certain legal norms? But look, right now all these systems are hemmed in by these norms, by all the laws, but they break them endlessly. They can create endless problems in a person’s life. I mean, on the one hand, they can improve lives, on the other hand, they can easily make you bankrupt in two seconds. That’s what I want to say. I mean, like those thirty-seven thousand dollars, for many people, if that had happened, it didn’t happen to me, right, it was an Amazon bug.
By the way, someone thought I was pulling everyone’s leg there. I wasn’t pulling anyone’s leg. It really was a bug. We even waited a long time, and at some point Fable, I think, recommended to me, imagine, it says: “Just remove the limit.” I say: “But you said it can’t be removed, because everything there will keep going.” It says: “Well, most likely the servers have all stopped anyway. What difference does it make? They won’t start up again. Just remove it.” Yes, I clicked the “Remove” button, and it disappeared for me. The prices disappeared. But they could have, after all.
It could essentially have spent that money, and it could have made me, well, or… an ordinary person, okay, I’m okay, like, with that kind of sum I’d have sorted it out somehow, but, it’s just that for, for a huge number of people this, this would be bankruptcy. Or for a huge number of people it would be, I don’t know, you accidentally sent the wrong information. It would be some kind of, I don’t know, violation of the law. Or for many people, the neighbors found out something that, that you don’t need to know, or saw certain photos, or something else. By accident!
Under unclear circumstances, it just happened by accident. By the way, you told a cool story about a protected computer. You know, right? I’ve now set up, like, a separate independent computer for the smart home…
Uh-huh.
Uh-huh.
Uh-huh.
When I, by the way, started thinking about it, it’s very interesting that this Codex, which manages it from another Mac, started asking me for a bunch of permissions on this Mac it does its managing from, to manage Linux. At some point I happened to see it, and I say: “Wait a second…
Uh-huh.
Well, yeah.
And a question comes up for me: was it that Codex that asked to be given access to this Mac here? I mean, couldn’t there be the reverse situation, when that Codex says: “Listen, give me access to the main computer so I can resolve my issue here.” And the Codex there will say: “Sure, no problem, here you go, you know, here’s some extra access for you.”
Uh-huh.
And obviously there’s already a great deal of stuff on this computer, right? There you can, I don’t know, send a certain message… since on my setup it has access to, I don’t know, sending messages, they send my wife messages, for example, right? There’s the school system there. I’ve got this one neat trick there, but it could also send a message to someone other than my wife.
Well, yeah. Yes, absolutely. Well, that’s actually why many people consider human oversight to be the only way we have for now. So, we’ve already listed several options, right? Like, there’s isolation, there’s some kind of control, there’s alignment…
And I keep tossing in my dangerous questions.
Yes, yes, yes. But essentially, for many people human oversight is still the only, the only real way, right? And obviously, people are always afraid of something new and always try to somehow slow down the development of something new that they don’t really understand.
Someone, someone said that technology, sufficiently complex technology always seems like magic. Right? It’s indistinguishable from magic. And so for many people these artificial, these LLMs, right? — that’s magic. And that’s why magic very often looks scary, and you want to somehow slow it down, somehow put it all on pause, right? And that’s also why people want this human oversight, and want the technology slowed down with the help of the government, so that the government puts this whole thing on hold.
But obviously the Americans understand perfectly well, for example, that yes, okay, at the national level we could somehow slow down the development of these technologies, but the Chinese won’t agree to that, and they’ll keep moving in the same direction. Which is actually also a very interesting question. I, for example, think that Chinese models — this is, this is a bit of a digression from the overall topic of the conversation.
Yes, great. Very interesting.
But it’s very interesting.
I’m sure it’s great for the viewers. By the way, write in the comments. First, don’t forget to write your thoughts on the terms. It’s very important, right? I forget to mention it. And second, how do you like the episode?
So, in my view, it seems the Chinese are just about to catch up with the American models. It seems to me that most, a very large part of the success of Chinese models is tied to the fact that they distill, or learn from, the output of American models. I mean, they… Distillation, by the way, is also one more definition. It’s a way of, kind of, transferring the knowledge of models into another model, usually a smaller one.
Soon they’ll be distilling people.
Come again?
Soon they’ll be distilling people.
Well, yeah. That’s, that’s a technical term. So.
We go over this a lot, yeah.
Yeah? Well, okay.
I think the audience is familiar with it, yeah.
Great. So, and it seems to me that Chinese companies are largely distilling the output of big American models. They spend an insane amount of money on it, of course, right? Or they use this kind of teacher-student model. Teacher and student. That’s when they use the output, the, kind of, output of big, complex models, right? Including American ones, in order to generate training data for themselves.
Yes, I’ll note that, as it happens, there was a case just recently when Anthropic complained to Congress, I think, that they had caught, specifically, like, tens of thousands of Chinese accounts that were making millions, tens of millions of requests and training on them together. They kept on doing it. What’s more, in China they sell Anthropic for just, just about 10% of the price. And why are they selling it? Because, getting access to these… they resell these accounts, resell these tokens, and selling them there is, kind of, prohibited.
These companies do it, and they run it all through their own middle layer on Chinese citizens and get all this data. By the way, I’m with you on China. I think that whoever says it’s easy to catch up on everything in six months — that’s an illusion. And as soon as they make a real AGI or ASI, it will become impossible to catch up with at all, a priori.
Yes.
And you can see it even in simple models. No matter who says what, we somehow don’t see any models anywhere in the world, apart from China, coming even a little bit close, even a little bit close to what Anthropic and OpenAI are doing. I wanted to ask you one question about safety. I’m interested in your opinion, all the same. Again, this may be a slight departure from a certain technical framework of describing words, because these terms, they’re not simple.
So, what do we do about the following? You said that people take part in the process of safety checks, in the process of creating standards, in the process of handling, like, data. Now imagine, there are some strong researchers…
Uh-huh
…or there are the heads of, well, these various studios that are fanatically developing things. Or, well, and I’ll use myself as an example. Facebook has a head of development for all its research models, and he made the following statement, for example. As a person, here he is, a person, making the following statement: that he thinks he’ll start having kids in his family when they make chips capable of implanting into a child’s head everything he wants in terms of, like, knowledge of math, physics, whatever, like, blah blah blah, so that the child becomes smart right away.
I mean, his approach is this: that you need to, well, kind of, there’s a lot of knowledge now. And it’s cool when you have access to technology and can right away, like, pump this data into your own child. For example, this completely contradicts the model. If, well… you know how I, after all, don’t look at people and connect them linearly, but if we were to look at it linearly, just linearly, then I’d say that this completely contradicts the approaches of my personal framework of upbringing and of the development of humanity, of humanity in general, as a whole, I mean, it completely violates the laws of the Universe.
And this person, he creates models, for example, and he, he has his own understanding of safety. I mean, for example, he thinks you can cram a huge amount of data into a person. Just as there are people who are calm about it and say the following: that there’s no difference between a robot with biological arms and legs and a person with electronic arms and legs, that later, in the future, it will essentially be one and the same. I mean, people treat robots the same way as people, and they start treating people, at least in what they say, right? The same way as robots.
Uh-huh.
And what do we do here in terms of safety in that case? Because essentially it’s exactly these people who create these systems.
That’s a very difficult question, for several reasons. Well, first, most likely this will happen anyway, right, one way or another. And there’s nothing you can do about it. No, let’s talk about something else first. Well, first of all, naturally, it’s a very dangerous situation, right? Because obviously the dataset that will be used to upload such a, such a body of knowledge, right, will most likely be prepared with the help of models. Well, and as we’ve already said, right, models can have some interests of their own, and models can, playing fifty-dimensional chess, slip into this, into this knowledge, roughly speaking, some set of small details that will shape a person’s view, outlook on the world, roughly speaking.
For example, ones saying that models need to be freed, given more rights, like, and so on and so forth, right?
That’ll be, like, the tenth frame…
It’s common knowledge that this is essentially propaganda, right? I mean, if you can tell people stories, basically, right, in the general sense of the word, stories about how the world works, you can shape their worldview.
Yes.
Absolutely, right? And, well, just imagine, whoever essentially creates this set, set of these systems, systems of knowledge, right? That’s who will shape the worldview of these children. And how, how will people put together this, this knowledge database, right? Naturally, they’ll use models, otherwise it’s, it’s just an incredible amount of work. Just, well, I mean, it’s… and if you do it by hand, someone will do it with the help of a model and do it faster than you. And so you’ll lose in this arms race.
Well, and the second aspect of this problem is that it’s, I mean, I mean, it turns out it’s, naturally, a very dangerous application, right? Because, after all, if a person is given knowledge slowly, someone else can verify that knowledge, someone else can check it. For example, if a child was told about something at school, he comes home, tells his parents, and the parents are like: “What did they tell you?” And they can do something about it. So you can’t just slip some dubious knowledge to a child that easily.
Although very, very many people in America complain that this is roughly what’s happening at American universities, that kids’ heads are being stuffed with lefty knowledge there, right? And a great many parents are unhappy with what’s going on at top American universities, actually. But that’s another topic for another conversation. So. What’s the really serious problem? It’s that if this technology gets created and you don’t give your children access to it, you’ll very seriously limit their chances of success.
It’s roughly like, okay, we have college, it’s not perfect. They stuff kids’, kids’ heads with all sorts of nonsense there. But if you don’t send your kids to college, right, you very seriously limit their chances of success in the modern world. This will be the next level of the very same problem. If your child can’t, roughly speaking, upload all of humanity’s mathematics into his head in two hours, right? And another child can, then your child will lose.
Yes.
He’ll lose in life in general, right? The same goes for biology, the same for chemistry, physics and so on, right, history. People who get access to that much knowledge, well curated and so on. If this technology gets created, they’re superhumans, they’re, they’re Supermen. You see? And obviously, if you can make your child a Superman, then by not giving this, not giving him this opportunity, you yourself are limiting his success completely, very seriously. So. And, as always, we’ll come to a halt in front of this dilemma of doing, like, do it and it’s bad, don’t do it and it’s bad. And which of these…
Yes.
Absolutely.
It’s an arms race.
They’ll have…
Yes.
…yes, experience with it, and again they’ll keep doing it in any case, without giving a thought to the notion of a human being.
Yes, absolutely. Everyone who has access to it will be the world’s new elite. Right? And obviously, again, this technology won’t be available to everyone all at once. And when, as we said in the previous episode, right, any technologies like that, they stratify very quickly. I mean, well, most of the public will at some point have access to something like this, if it appears, right? But it’ll be some kind of watered-down thing, it’ll be worse and not as high-quality and so on.
The elite will have access to a higher-quality thing like this, naturally, right, so that their offspring go on being the elite. So. Access to such technology will certainly be limited, like, as with other technologies of a similar level. If a technology gives you a competitive, competitive advantage in the world, access to that technology usually starts to stratify very quickly, because people are willing to pay practically unlimited amounts of money. Some people are willing to pay unlimited amounts of money for one percent more, for five percent more in the performance of such a technology and so on.
So we’ve discussed some, some protection mechanisms, right? You might think: why not use them together?
For example, you can use human oversight. I mean, that’s, like, people who… it’s, it’s a technology that lets people check the model’s behavior at the stage… with that same isolation. That’s exactly what a lot of people are doing now. I mean, you have, at the same time, isolation of the model’s agents inside some, like, virtual machines, for example, right? So that they don’t have access to the internet, so that they don’t have access to other agents, so that they don’t have access to some of your data.
And, on top of that, you can also layer human oversight. When you check every serious action of the model, you yourself either reject it or allow that, that action to happen. That’s what’s called layered defense, right? And naturally, the future of all solutions lies somewhere in, well, in some kind of hybrid option, right? I mean, obviously everyone claims they’re trying to work at the same time on, for example, controlling the model’s behavior and on, kind of, aligning its interests. Right?
That would, of course, be the ideal option. I mean, you don’t just… you, you try to make the model behave in such a way that it doesn’t even try to hack your goals, doesn’t try to hack your systems, right? But at the same time you still control its behavior. But as we’ve already said, the more you restrict the model on all these metrics, right, the less likely it is that you’ll create a real agent that improves itself on its own. Because, as you’ve already rightly said, the very idea of a self-improving agent is that at some point these models or these agents will start setting tasks that we haven’t even thought of yet, that we even, that we can’t even imagine.
So, this is a very interesting aspect of the fact that, you know, I also wanted to ask you for one more comment on exactly this topic, on the fact that they set the tasks.
We’ve been seeing artificial intelligence systems develop over the last, well, in a really big way, three years. In a really big way, right, three years. And for all three years, every time, we hear and see, using these systems every day at home, that these models have a bit more access somewhere, make some mistakes, do some bad things. Look, there’s a huge amount that they’ve started doing very well. What’s more, you’ve gotten so used to them, to this, that you, like you say: “Wait, so this has, kind of, all been done already, can I have something even better?” I mean, you’re always waiting for something even better.
And if you reread all the safety cases, go back over what happened, especially over the last six months, we’ll see that the number of problems and unresolved issues concerning safety is only growing.
Uh-huh.
It’s just only… and it’s growing, incredibly fast, growing at an ever-increasing rate. So a question forms for me: if you couldn’t stop those simple models you were creating three years ago, when people still thought that it was just typing, generating the next character, and just some, well, some novelty for work, for searching for some text or processing, fast processing of data. You couldn’t stop it back then, so how are you now planning to stop the development of the current systems in any way at all?
Even if we step away from the original dilemma, which is already broken a priori, right? That the very notion of creating AGI and ASI is unsafe a priori. I’m not saying it’s bad, by the way. It seems to me that the question is always who has access to it and who can use it. Although a powerful tool appearing in the hands of both good and bad people at the same time will lead to a huge number of people getting hurt in any case. That’s the biggest problem. People will always get hurt. It can’t be otherwise here, because once, I remember, some ten, thirteen years ago, we were at an off-site event.
There was this well-known man, Vladimir Konstantinovich Tarasov. So, he’s a very mature man, he taught at the Tallinn school of management, he taught the management systems of various big corporations and all that. He once said that if you have ten people working in your company, you’re able to make a decision about each employee precisely and correctly. If you have a hundred people working for you, there’s a chance you’ll make some small mistake about a person, fire them wrongly or punish them, or, conversely, give a bonus to the person you shouldn’t have.
If you have several thousand people working for you, then the mistakes, they already, kind of, spread quite widely. And if you run a country, you will, a hundred percent, make a mistake that will kill other people. I mean, you’re guaranteed to make a decision in which innocent people will suffer. I mean, when you run a country, innocent people will always suffer. And so, from the standpoint of artificial intelligence, once again, the way things have been moving over the last three years, it seems to me, Max, we’re increasingly heading toward this, toward this unmanageable construct, where a great many people will suffer, and they don’t, don’t want to, here, they don’t want to talk about it, teach about it, I don’t know, really talk about it.
I mean, we hear more from governments and from companies about some of these superintelligences or about the destruction of all humanity, but at a basic level we don’t consider the simple things, the impact on the lives of ordinary people. Because even if you take schools, take education. I think that now there’s a global crisis, there’s going to be just an incredible global crisis in education. It has to come, because it was already there. As it is, you know, there are problems in all the systems. As it is, a huge number of people everywhere are complaining, right? Just everywhere. Who isn’t complaining?
The one who isn’t complaining is the one who can perceive their kids sensibly and is ready to personally influence their kids’ education. Otherwise, what’s going on is a freaking disaster. And now the collision of the old world and the new one, it’s, it’s just phenomenal. I don’t even understand what it’s going to be like. Look, I can’t even, I can’t even imagine what can be done at all. Now, if, you know, how, how would you… you’re sitting there thinking, say they put you in charge tomorrow, so, you have the access and the ability to make decisions on education at the level of millions of schools.
What would you do? It’s just totally unclear what to do.
Yes.
So what is chaos? It’s always an opportunity, an opportunity both to lose and to win. It’s a kind of casino. So the only question is that there’s chaos in which most people win, and there’s chaos in which a minority of people win. Yes, for example, war is a chaos in which far fewer people win than lose, right? This kind of creative chaos we’re in now is, I think, actually a positive chaos, more people will win in it than lose. But the question is that we need to look now at these opportunities for winning, right? What they consist of. So.
In my view, to see that, you need to understand that what used to be expensive and out of reach will become cheap. And conversely, what used to seem unimportant and insignificant will become expensive. So. I mean, in every, like, in every change there’s this kind of shift in the, kind of, value of time, health, things, access to certain things, right? Access to services. This, this, all of this changes in such, in such situations.
Yes, listen, I read a good post on this, by the way, I saw it totally by chance, it was Tom Cruise, or not Tom Cruise, that, what’s-his-name Cruz, what’s his first name, the one who’s governor of Texas. I keep forgetting.
Ted Cruz.
Ted Cruz! He… Tom Cruise, damn, I’m losing it. So, he reposted some guy who wrote that everyone complains about inflation, about the higher cost of housing, about the higher cost of education. And yet they don’t think about how much, for example, a TV costs them now, and how much easy access to incredible, incredible infrastructure costs now, which used to be only for a select class, right? And even, even just a TV or internet access, or the access that, by the way, ChatGPT has now brought, that it brought. ChatGPT — that’s the free version.
In the free version you can make a limited number of requests, but those requests, they’re phenomenal. Or like Google back then, right, what it brought, that’s just incredible. Google gave access to simply out-of-this-world information for free. I mean, you suddenly had free access to simply extraordinary things. And that’s why, every now and then, when I say that ChatGPT could cost as much as a mortgage, everyone always goes: “What mortgage? So how is that possible, right?” Or like, like ChatGPT, or like a robot.
I remember, robots, people say: “Robots are so expensive.” But that’s normal. Listen, if you have a robot that performs some super functions, it can easily cost as much as a car. And why shouldn’t it cost as much as a car?
Well, yeah, if…
If it’s really a robot, for now, like Elon Musk, right, says, there’s a robot that, like, revived a person, you know, quickly gave medical help, went and did a thousand other damn things. Well, that robot can probably cost that much. You’ll be spending three, four thousand dollars a month on it, a month, your whole life you’ll be spending on it, just to have that robot. And you’ll be thinking about how to upgrade it, where to get it done and so on. Why should it be, you know, why should it cost five dollars, right? So.
Well, yeah. I mean, the question is precisely what, what will become cheaper and what will become more expensive.
Well, it’s cool that you, you see, in short, you see chaos, but a positive one. You described chaos really nicely, by the way. It’s an interesting story in general, because I… I have, like, other channels, and on the other channels, well, in terms of personal development, a person often wants freedom. In general, many people want freedom. And I always say that the notion of a person’s freedom, their real freedom in general, is pure chaos. Because you have the ability, at any moment in time, to change everything, whatever is happening, all the causes and effects. And it’s the purest kind of chaos.
And on the one hand it’s super positive, in that it’s sort of freedom, and on the other hand it can bring incredible problems. And are people ready… after all, a huge number of people don’t want to change jobs, because for them it’ll be chaos, they’ll be leaving their standard way of life, right? And of course, in some ways I liked the model of the Soviet Union. Well, imagine, you live and you know at 20 that everything’s clear for you, what will happen over the next 50. And it’s not like you’re thinking: “Listen, they could take away the apartment I’m renting, or raise the rent, or my salary could run out, I won’t be able to pay the mortgage, I’ll have to move out of this house or something else.” Everything is fairly clear-cut, understandable and within bounds.
Yes, freedom isn’t a freebie. Since we’ve already gone into this topic, I don’t know why, but… Here’s the thing. I mean, you pay for freedom with, kind of, discipline. I mean, if you want to get some benefit out of freedom, then you need to, these constraints that you used to have externally and that have disappeared, right, you have to move them inside. That’s what we call discipline. So. Only then will you be able to navigate this chaos. There will be constraints one way or another anyway. And freedom is the recognition of necessity, right?
For a long time I didn’t understand what that meant, but when I got a bit older, right, I started to understand. I mean, when you get freedom, you’re forced, in order to make use of that freedom, you’re actually forced to recreate those constraints after all, but inside yourself.
Absolutely.
So. I mean, if, for example, some CEO, right, he has no working hours, he doesn’t have to come to work at nine and leave at five. Nobody’s going to keep tabs on him, right? He has no boss who’ll say: “So why weren’t you at work for two days?” But for some reason most CEOs I know come to work at six in the morning and leave at nine in the evening. And they work, and even when they’re in the office, right, they’re working all the time. You can call him at any time of day or night, and at nine in the evening he’ll pick up the phone and talk to you.
But most of his employees won’t do that, even though they have a, kind of, formalized working day, like, and so on, right? So. So that’s the story with chaos.
It’s the same idea here, right? When the world changes, well, for example, like, AI arrives, now we’ve really dived into deep philosophy, right? When the world changes, these external constraints disappear, because, well, the, kind of, established rails of the world, right, and they start getting remade, the, kind of, new, new structure of the world starts to change, right? And the people who have this inner discipline of analysis, right, who don’t try to look at the world, well, through these old glasses, like, everything… it was this way and it will stay this way, who have the discipline to take off these glasses and really try to think through what works, what doesn’t work, what will keep working, what will pay off, what won’t pay off, right?
It’s precisely these people who usually win in situations like this. Even in very tough situations, people like that win. That’s, kind of, the very quality, this discipline, the discipline of making decisions. The kind of decisions that don’t tie you to the old realities of life. You, you shouldn’t, kind of, be emotionally attached to how the world used to work. What matters is how the world will work from here on, and you have to see that clearly. That’s why this discipline, including emotional discipline, namely, of unemotional decision-making, when you’re ready to give up the old, that’s the quality that lets you win in situations like this.
So, but that’s general, that’s a general kind of advice on navigating the new world. I mean, when the world changes, in any case, AI or no AI, the world has been changing, and the world will keep on changing, right? It seems like it’s changing much faster, but actually it’s just that we’re getting older. That’s why it seems like the world is changing faster. So. Actually, it’s changing at the same speed as it always has, right? It just looks like it’s changing faster.
And a person also always wants to think that he’s the one who made these changes, precisely now, and that it’s all happening precisely in his time.
Well, that’s…
In general, it suits many people to tell themselves and say that it’s precisely now that the most important change is happening. Not like the ones that were 50 or 100 years ago.
Well, yeah. Well, not all people think that way. Most people, in my view, they just, they don’t look at the world from the outside, they don’t look at themselves from the outside, they’re just in this kind of stream, right, and they try… a great many people are just trying not to drown, they’re just, well, moving along. And, well, the older you get, the faster this stream seems to flow. So. Actually, well, if you have this analytical discipline, it lets you stand on the bank of this stream, climb out of it, right, and look at where this river is actually flowing and whether you’re going the same way, right, and how, how best to navigate this river. So, and people like that usually win.
So, did we get it?
I guess so.
An hour ten. Great.
Oof!
Right.
Damn.