We have a very interesting topic today, because I see how the internet is bursting at the seams — if you take the secondary, third or fourth market, not just OpenAI, Anthropic or Google, but the huge number of other startups, companies or people who want to make money on this — bursting with headlines like the 15 best plugins for this system, the 20 best plugins for that system, and that system runs on some other system, and that system runs on the basis of, say, OpenAI or Anthropic. And you all know I have said this more than once: when studying artificial intelligence, I suggest everyone concentrate their attention as much as possible on serious companies and serious models.
If two and a half years ago you could still run tests in ChatGPT, in Claude, in DeepSeek, probably even in Gemini, today you need to focus yourself very clearly. If you can use one model, then use one — ChatGPT or Claude, whichever you like. If you develop software, you can also choose Claude or Codex. If you are broader but can use two models, you can pick some other models. Or use various middle layers after all, such as Perplexity or, say, Hermes and a great many others. I am in no way disputing that many of them have their own solutions, their own agent systems.
Or use, for example, various plugins inside these systems — there is a plugin for working with health, ChatGPT Health. Or, for example, two interesting updates have just come out — I think I will make an episode about this — about financial systems, separate financial systems in OpenAI and in Claude. These plugins they make, all sorts of them, a huge number — if you look at the plugin library in OpenAI or the plugin library in Claude — they really are very big, they help with files, with tables, with search, they let you study something purposefully: health or nutrition, fitness, finance, education.
What is a plugin, in short? It is when there is some predefined room, or some restrictions, or additional instructions, so that your system performs tasks in that subject area better. So if we take the example of the OpenAI Health plugin, it is essentially a plugin to which you can connect independent systems of your health, the ones it integrates with — in particular, you can connect Apple Health to it, and when answering your questions it will use its knowledge base concerning health, that is, the instructions it has about health.
And it already has an update — the feature existed, and it already has an update, a version update — and essentially it is a testing tool that answers this very question. And which question? What pays off better: using a plugin to solve a certain task — for example, calling a finance plugin in finance, or calling some code plugin when writing code, or a plugin for plotting a travel route or travel ideas — or not using a plugin. So this is the new feature; I will tell you separately about Claude and separately about OpenAI, very, very important structural information, because OpenAI released — I think, I will give you the exact date — OpenAI released it literally a few days ago.
Also, by the way, it was on about the same day, I see — they released a report called "Rethinking skills and instructions for GPT-6 Astra", and it is very interesting precisely for understanding plugins. And if you look at the whole market, what is everyone writing? There are 20 use cases you must do with GPT Astra, these things you must enable in GPT Astra, those things you must enable in GPT Astra. Again, people start pointing to some specific instructions. And I, as you know, am not a great fan of instructions, of all these restrictions that everyone wants to build around.
And in my concept, in my paradigm — we will soon have a very interesting episode about the main features of the current trend in artificial intelligence. I urge everyone to use the models as part of their daily routine, understanding that search in its usual form will not exist, most applications and software in their usual form will not exist, a huge number of the processes that exist now will change, a huge number of companies will change, the processes of those companies.
That is, the market is in for substantial changes in the coming years. What I am saying now is not bad. It is neither good nor bad. There is a certain given, an understanding, and arguing with it is pointless. Like arguing that people will keep using Google search the way they used to. It is already not used the way it used to be. There is already, at the very least, a very fast AI answer on top. We recently discussed this topic very seriously in the episode with Ilnar. So, if we talk about this claude plugin eval feature, what happens there?
What happens there is, essentially, repeated execution. That is, to choose a new plugin you pick the plugin, and for it — look, if you do not use Claude, that is no problem, you should understand this on the basis of how it works, including in OpenAI, or if you use Gemini, or if you use DeepSeek, which we have not used at all. This will help you understand what these additional plugins are in general. And even better, it will help you understand the middle layers and wrappers of different systems.
Once again, I categorically do not support various middle layers in general; I support them precisely in complex, very complex cases only. These middle layers that people make. A layer to create a mobile application easily, a layer to quickly make pages in a CMS, a layer for writing text, a layer for building proper storytelling, a layer for creating, writing, processing YouTube videos. Today, simply by using this software — somewhere, of course, something works automatically out of habit — very often, and this does not mean you have to give up all such things, but very often you miss completely new possibilities that have already appeared.
What Claude Code with Fable or GPT-6 Astra can do now — these are unreal things in terms of layout or website development, building the structure and strategy of a presentation. That is, these are new, entirely new paradigms of everything that is happening, of creating various analytical reports. These middle layers and plugins — these systems, by the way, will support them. That is, Anthropic or Claude Code will keep creating plugins for specialised systems — for, say, Power BI — they will keep doing it, because that is their corporate client or corporate customer.
So, what does this claude plugin eval do? It does repeated execution. What does that mean? It means that by default each task is run three times with the plugin and three times without it.
Very interesting, right? We will talk in a moment about repeated execution of various tasks in general. A clean environment — there is such a concept — means separate sessions without your memory, other plugins, or the usual user settings. Next, what happens? A check takes place: the various answers, the created files and the agent's actions in these, essentially, six sessions are evaluated. And a final score is assigned: the share of passed checks, weighted, averaged over the runs.
There are different commands inside — we will talk about them — there is a comparison: essentially you can look at it with the plugin, without it, or the difference that exists. And the runs and the model's evaluations consume your subscription limits. Accordingly, you have to understand that each of these runs may cost differently, and they consume your limits within your subscriptions. Subscriptions — or you pay via the API, the programming interface. So, there are six types of checks.
There is text match, there is tool call, there is checking the order of calls, presence of files and so on. And there is model grading and comparison with a reference. You can simply do this from the point of view of this technical method. I will tell you about the technical method now. Once again, I am not urging you all to run all these technical methods. This is interesting for people who are ready to dig deep. Those who have dug in — please write what you ran your tests on and what the difference was.
It is very important to learn and see this on the channel. And it is important for other people to see. And it is very interesting to me too. And do not forget to support our channel ToTheMoon. Your like, subscription, comment — a simple action. It very much improves our momentum, our development on the channel. But these check types are interesting too, because we need to understand what we are actually checking. Because it is not a simple story. How are we going to evaluate the output at all, especially if the question posed to the plugin is a fairly difficult one?
So you have prepared everything inside for the run, everything that happened. All these settings have been made for us. And essentially you get this result at the end. There is some estimated budget you may have. You ran it, and here is your result. How to read the result?
Suppose that on identical tasks it turned out, well, something like the following example. Again, I will give you some hypothetical figures, not real measurements. For example, you got 70% without the plugin and 90% with the plugin. That means there is a noticeable improvement on the chosen criteria. That is, 70 and 90. If your result came out 95% without the plugin and 95% with the plugin, then no quality improvement has been shown at all. And you need to compare, for example, how much time was spent on these tasks, what the cost was inside.
If, for example, the result is 95% without the plugin and 80% with it, you have to consider that the plugin may be getting in the way, or some test is set up, say, not quite right. And if you have a situation where it is 0% without the plugin and, say, 95% with it, you need to look into it properly. Look, when the differences are this strong, you need to work out whether the plugin gave a new capability or simply opened access, for example, to the necessary data. For example, you asked it to analyse what happened to you over a whole month with the number of steps you take each day, and in the plugin you have access to Apple Health, which collects the number of steps you take.
And without the plugin it did not know. Well, obviously the result will be 0% without the plugin and 95% with it. I think everyone here understands that. Because the fact that everything worked with the plugin does not yet mean it worked thanks to the plugin. That is exactly why this control group without it is needed. So, it is important to understand that a plugin and a restricting instruction are not the same thing — because we have all seen it on the internet, and at the dawn, remember, two or three years ago, a huge number of people said: take this instruction, take these prompts, it will help you enormously.
It is important to understand that a plugin and an instruction are not the same thing. Simply, for example, in Claude Code a plugin is an extension package. It can contain skills, it can contain additional agents, handlers for various events, it can contain connections to external systems — as in my example, a connection to Apple Health. A skill is, first of all, as we know, an instruction; sometimes it can come with some examples, it can come with files, even ready-made programs.
It is not a specially fine-tuned model. So I would separate the benefit here into different situations. And now, once again — I told you we would move on to OpenAI. And here is a fundamental thing — well, probably two fundamental things I want to convey to you about GPT-6 Astra and about the whole planet of plugins and instructions. So, what does an extension add? It adds access and various actions — for example, retrieving documents from a corporate system, changing a record, launching a specialised tool that exists.
And clearly there is a concrete benefit here, because the model gets a capability it otherwise had no way of having. And now, if we look at these additional plugins and even, in places, systems — there are separate huge settings in education or in security, in cybersecurity at OpenAI. These settings and plugins can, of course, connect — financial systems, say, connect directly to institutional data, for example to daily analytical reports, summaries, to various paid systems and so on.
Of course, you can supposedly connect your own system to that data separately yourself, and then to settings too. We will get to that, I will definitely talk about it. But importantly, a plugin can have this access and this action on systems. Next, what do these — well, what does this extension add? It adds special knowledge and ready-made solutions. That is, for example, the peculiarities of an internal system, a proven way of processing, say, some non-standard file — for example, some plugin for working with specialised audio formats will open these additional files for you, or a plugin that works for some super-cool archiving.
And the benefit here is that you do not have to search for or invent solutions again, or describe how it should work. That is, the system is basically very finely tuned to this, understands what it is and works specifically in that place. And the extension adds you a third thing: it adds rules and an order of work. That is, some report template may be contained there, a sequence of specific actions or checks the system will have to perform — for example, a mandatory approval it must obtain from you, or within itself in terms of its agent system.
That is, the result essentially matches your requirements. Here is something very important — here is the construction that occurs to me: often these additional instructions can end up worsening the result.
That is, possibly, if you do not give instructions, the result will be better. And you can see in Codex, in GPT-6 Astra and in Fable a large number of tasks without a description. I remember when Fable appeared, this was very clearly visible. You started writing fewer prompts, and this era of no prompts began. And look, this will only continue. It will not get worse. And thinking that someone can somehow configure a system better, uniquely, somewhere, and hand you some trick — that is a certain abstraction.
That is, your task is to learn to understand how these systems work in general, and how to apply that within your daily day. But an important story is that restrictions are not always a drawback. Here it splits. I am not a great fan of plugins. I would say I generally try categorically not to use them. I see problems in them. But still there is another side, and I want to devote attention to it. Many of you will see usefulness in this. Many of you have already suggested skills to me — skills, a heap of everything.
Clearly there are many. Once I opened Codex, and it tells me: you have something like 100, 40, well, some number of skills. I think, oh, excellent — so I am not sure that I created these skills. Right now it says that I have used 1,319 skills. Well, okay. That is, and it explored some 23 more skills. So the system itself creates something, itself makes these skills, itself describes them within my projects. That is, it does this for itself. But it was not me who said: please create skills, describe them.
This happened several times. And I want to say that I then also have problems with them. And I have not fully worked out for myself whether these AI skills need to be created at all. We will talk about that now. Or whether very clearly, rigidly described criteria are needed. So, once again, the restriction from this plugin is not always a drawback. And still, yes, we can understand that if, for example, some plugin has a restriction written into it not to send a letter without confirmation, then it will, for example, slow down your system, unlike the case where you say 'write a letter' and they take it and send it straight away.
But at least you will have very strong control, because such a rigid requirement exists. And once again, in many organisations, companies, processes, this is important to you.
So, now let us move on to OpenAI and to GPT-6 Astra to understand this case. That is, we have this question: are plugins needed at all, or not? Or are they not needed after all? And we should not use these middle layers, we should not do all this at all — no. There is no need, say, to go into Notion and continue there, using, for example, the artificial intelligence inside it to process a short table. There is no need to call Notion from inside ChatGPT for your — well, if it is some small project, a small table.
Or there is no need to go down even further, not into a Notion plugin but into some plugin for organising tasks. Yes, such plugins obviously exist. But rather to remove all this and say: I live my own life. Here is an example for you: a friend of mine just flew in from Italy — my friend and partner — and he photographs his food everywhere. And he says: well, I decided for myself that this is the simplest way. And I told him: this is the coolest way. He says: listen, there are different systems, plugins, some separate software solutions — these trained systems can help me a bit more in terms of nutrition, in terms of food.
And I told him: there is nothing better today, there is nothing better at all. The solution is simply to photograph your food constantly, in terms of what you eat, into an ordinary chat — into ChatGPT 5, or into Codex if you are connected to it; ChatGPT 5 is enough, of course. Because ChatGPT 5 will let you get far more of everything. It will let you turn not to one database of systems, not to two, not three, not four, but to a hundred, two hundred, to different countries, or different doctors, or whoever you want.
And it will have this information in... You can photograph however you like, you can weigh or not weigh, you can photograph the menu, you can dictate by voice — you can work with this information in completely different ways. Moreover, my partner has the Pro version, and not a single external system will ever in its life run an evaluation of your food through GPT-6 Astra Pro. Never — believe me, it will not do that. Because the requests will be too expensive to process — constant processing, for example, if you also keep talking, re-reading the entire context, and 'look at what I have been eating here for two years'.
And moreover, you can load doctors' notes and some test results into that chat as well, while other systems will simply be limited in this. Someone, again, may now argue that this is not so, that such systems will survive or will not survive. Well, this is merely my conclusion. So, OpenAI. This, by the way, is a good example of using and not using plugins. I do not recommend using plugins in a system like this. This is my strictly personal recommendation. I personally would not recommend it to myself, and I will not do it.
And I told my friend that in fact he chose the coolest path, not the limited one. So, on 11 September OpenAI published a piece called "Rethinking skills and instructions for GPT-6 Astra".
And they describe several problems. The first problem goes like this: long, overlapping skill descriptions make it harder to pick the right one. And overly broad activation conditions — these conditional instructions that come from outside — force instructions unrelated to the task to be loaded. And, essentially, detailed step-by-step scripts that helped previous models can over-constrain a stronger model. And Ilnar talked about this, about Astra — that Astra works differently.
That is, we must understand that we are in a new era and in a new age. And we need to start — all of you watching today's episode too — looking at this completely differently. Hear it once again: detailed step-by-step scripts that helped previous models can constrain a stronger model. Mandatory reading of large documentation before every small change consumes context and time. And by the way, I want to tell you what I noticed in Fable. Interesting, by the way, again, for those here who design a lot: I noticed in Fable that it has already become unprofitable for me to open a new chat every time when designing systems.
I have many systems going there. And it is more profitable for me to develop everything in one chat. And recently, by the way, in a very long context window in Fable, I asked about very long tasks — it has lasted, I think, a month and a half in one chat. How many times did I try to solve this task in various other chats? It was not getting solved. And by the way, I told you about it, remember, a few episodes ago — that I have a task that does not get solved. GPT-6 Astra again did not solve it.
There — ta-ta-ta-ta-ta. I finally solved it. Yesterday evening I solved it. That is it, the task is solved. So, this task was solved in that old context window. I think it is more than a month, a month and a half, two. I asked Fable there a few days ago, just after GPT-6 Astra had not managed it and I switched over. Maybe I should start a new context window? To which Fable tells me: no, no need. I am compressing everything here, everything is clear to me. Earlier even Fable itself used to say: indeed, we are oversaturated now, restart.
Well, Opus 4.8 definitely used to say that. But Fable too — I remember it saying so at some point. Now it continues working inside there, and it truly resolved the task for me. It carried it right through to the end; I liked that very much. And I even see that in Codex I have a large number of chats hanging there — even in them, because they are cloud-based, GPT-6 Astra Extra High or Ultra is not available now, but 5.6 Sol Extra High is available. Probably because they run for me from a different Codex, from a different computer — specifically the part of the tasks I use constantly, I use them specifically, because opening any new task causes some kind of, well, some kind of paradox in the system every time, instead of you simply developing inside the system.
I like that very much. That is, I very much want us to arrive, in systems development, at the point where, for example, you develop some system or manage a company, and you can essentially manage that company through one window. Clearly this one window probably should not, at some point, be presented as a single chat. That is, it should be some kind of — some specialised interface will come along; we will see what it looks like. Some interface of that kind, where things can pop up, open, with some details somewhere.
All these systems are trying to find it — as, in particular, in code, where a list of tasks to be delivered gets created, as Codex does it in terms of steps. It writes how many steps it has, what plans it has set, what goals it has set. Remember, this appeared in 5.6 Sol. And it is a very interesting thing. This topic — that mandatory reading of documentation before every small change consumes context and time — is of course incredibly fundamental, important, necessary for you to realise.
They propose an approach: a short description of purpose, loading details only when necessary, and regular review of old requirements. That is, it is also important to consider that, by the way, one instruction may be used by different models. That is, useful for one is not necessarily useful for another. And in this, by the way, I see a very big problem of the collision between Codex and Claude, when Fable does something and seems to say 'great', and then GPT-6 Astra comes and reports that something is off.
And they have these differences, because there is also an instruction there, it did not analyse everything, it analysed some part, and you sit and think: so how do they build it together. And it is not a simple story. And it is important to keep in mind this story that different models understand an instruction differently, different AI systems understand an instruction differently — the same way different people relate differently to what would seem to be the same things. And by the way, what I am telling you now are official engineering observations from OpenAI — important — not some study that gave, I do not know, some universal percentage of degradation.
There was another story at OpenAI: at the beginning of the year they released a report called "Testing Agent Skills Systematically with Evals".
And it proposed evaluating not the persuasiveness of the answer but four sides of the work: whether the result was achieved, whether the necessary actions were performed, whether the formatting requirements of the task were met, and how many resources were spent inside. And for the first set, 10–20 target requests were enough there, including those on which the skill should not activate. And there was this story where Codex was run while saving a structured log of actions, then programmatic checks analysed the file and the commands, and a separate model evaluation checked the more subjective requirements.
So, it also proposed tracking token consumption, and essentially this is a practical method for building checks, not a promise that, for example, the skill was useful. That is, what I am telling now — why I remembered this case — this story is not... there was a case, by the way, still with GPT 5.4, an experiment with excess context, where either all the descriptions were loaded at once or the needed ones were found on request, inside, for example, when you tell it some story about your health and give it this big context, or I, for example, say — I constantly load, I do not know, an entire ToTheMoon episode, imagine all the transcripts, or I load all the books at once, or it goes in bit by bit.
And there the system measured this too. But that is all the old system; I am just saying the task is not new. And there was an independent study by SkillBench — they did it in the summer on 87 tasks and 18 agent-model configurations with different prepared skills.
And these prepared skills raised the success rate from 35% to 50% — roughly 15% there. That is, they found that specialised skills really did help in many cases — substantially helped, by the way, improve everything else. And especially certain sets helped, for example, with efficiency. And I want to say that... And there was, by the way, a study of Agents.md files — these are, roughly, agent files for creating some systems — and the conclusion was the following result: files and instructions for working with repositories did not improve the success rate overall, but increased computational costs overall by more than 20%.
This, by the way — pay attention — is a wow topic, because all the systems were saying you have to train these Agents.md files on how to work, load them fully, everything. And it turned out — as you can understand it — that merely adding project descriptions and requirements did not guarantee improvement. Although, once again, the authors acknowledged the usefulness of such files for non-standard development rules. These are just some details for you.
Here the story is that Anthropic itself provides information about when a skill becomes unnecessary. The first type of this is skills that compensate for shortcomings in the model's capabilities. For example, some special technique for creating documents — as the model improves, they essentially become unnecessary. And we all knew this: earlier, for example, plugins were released that polished text better, because the model could not do it, for example. And now the model can do it — why use these third-party systems.
And this is about what I repeat everywhere, I talk about it a lot. Models will in any case develop incredibly strongly. Even if some feature of theirs does not work today, at some point it will start working. I remember a time when it was impossible to upload an archive — it was not analysed — or some large documents were not analysed. Now all this is becoming, essentially, a very relative problem. And the second type that Anthropic identifies is skills that capture your preferences and workflows.
That is, the model may be perfectly able to perform each step, but not know what order, format, criteria your particular team needs. And accordingly, such skills may remain useful longer. And you have to understand here that if you have some fundamental internal things of your own, and you launched a plugin that works from outside, you do not fully understand how it works, it has third-party instructions — then it is more likely to bring you harm than benefit. That is how the ChatGPT Health integration harmed me.
A very strange construction. Moreover, when I then started asking health questions in other chats, it constantly tried to load this plugin. Until I switched it off. By the way, a new ChatGPT Health update has just come out, and this setting — load or not load the plugin — gets set somewhere by default, and it turns out that I essentially cannot ask a single health question in ChatGPT without it using ChatGPT Health, or I have to constantly tell it not to use it. That is, all these connected plugins in general can only create problems.
For example, I remember I had a situation with a connected Photoshop plugin: I ask it to do something, and the system keeps trying Photoshop — it says 'opening Photoshop for the changes'. I say: do it yourself without that. And here a story arises that on the one hand plugins can help, on the one hand restrict, on the other hand restrict.
And here is the point: so that you truly understand how models work, so that you truly work out what this is, so that you truly live through new cases, so that you start working with artificial intelligence not as with ready-made software, which we have all been using anyway for the last years, decades, decades. For this you need to work directly with artificial intelligence all the time, without plugins; you need to work with the latest frontier models without plugins; you need to work with these systems inside, use ChatGPT in your everyday work, in your professional activity, in ordinary life as it is, try to do something in Codex or try to do something in Claude Code, not limit yourself only to professional activity, not limit yourself only to personal activity — only this will give you the most effective path, at least the one I see for myself, will let you truly, seriously understand artificial intelligence, understand how it works, and apply what you need within your every day, without any contentious, dodgy discussions nobody needs — and at the same time, if you want to test some additional plugins, just test them with an understanding of what they are, not with some limited trust and a lack of understanding of how these systems work, because for the most part you can accomplish these same tasks exactly the same yourselves, unless there is some
unique, extra-special exclusivity — for example, some plugin has opened access for you to a database you cannot connect to otherwise. What do you think about this topic — be sure to write in the comments, and see you in our next episodes.