Can Plugins and Skills Make AI Worse? New Rules for Working With Strong Models

A solo ToTheMoon episode with Alexander Volchek about plugins, skills and instructions — and whether the new strong models need them at all. The internet is bursting with lists of "the best plugins" for systems that themselves run on top of OpenAI or Anthropic; Alexander repeats his advice to concentrate on serious models and explains what a plugin is: a predefined "room" of restrictions and instructions so that a model works better in its area — like the health plugin at OpenAI, to which Apple Health connects.

The first occasion is the claude plugin eval command Anthropic has just added to Claude Code. It runs every task three times with the plugin and three times without it in a clean environment, evaluates answers, files and the agent's actions with six types of checks — text match, tool call, order of calls, presence of files, model grading, comparison with a reference — and returns the share of passed checks. The result has to be read through the control group: 70% without the plugin and 90% with it is an improvement; 95% and 95% is none, and what you compare is time and cost; 0% versus 95% most often means the plugin simply opened access to data. A plugin and a restricting instruction are not the same thing: a plugin is an extension package with skills, agents, event handlers and connections, a skill is an instruction, not a fine-tuned model. An extension adds three things: access and actions, special knowledge and ready-made solutions, rules and an order of work. But extra instructions often worsen the result: in Codex, GPT-6 Astra and Fable tasks without a description are solved no worse, and with Fable the "era of no prompts" began. Restrictions are still not always a drawback — a rule "do not send a letter without confirmation" slows the system down but gives control.

The second occasion is OpenAI's 11 September piece "Rethinking skills and instructions for GPT-6 Astra": long overlapping skill descriptions make choice harder, overly broad activation conditions load irrelevant instructions, and step-by-step scripts that helped earlier models constrain a stronger one; mandatory reading of large documentation before every change consumes context and time. OpenAI proposes a short description of purpose, loading details only when necessary and regular review of old requirements, and reminds that different models understand one instruction differently — Alexander sees this as the collision of Codex and Claude in his own projects and shares an observation from Fable, where he solved an old task in a context window more than a month old and the model refused to restart. Then come OpenAI's report "Testing Agent Skills Systematically with Evals" with four sides of evaluating a skill, the SkillBench study on 87 tasks with success rising from 35% to 50%, the Agents.md study with costs rising by more than 20% and no rise in success, and Anthropic's classification: skills compensating for a model's shortcomings die off, skills capturing a team's processes live longer. The host's own experience — the ChatGPT Health integration that barged into every health chat, and Photoshop, which the system tried to open instead of doing the job itself — illustrates the conclusion: work with frontier models directly, in ChatGPT, Codex and Claude Code, and test plugins with an understanding of what they are.

The value of the episode is that the argument "are plugins needed" is moved from a matter of taste to something measurable: Anthropic now has a tool for comparing with and without a plugin, OpenAI has engineering observations that old skills constrain the new model, and independent tests have numbers pointing both ways. The host does not deny the benefit of extensions where they give access to data or lock in a team's processes, but insists that understanding models comes only from working with them directly. The practical conclusion: before connecting someone else's add-on, check that it gives a genuinely new capability rather than extra context.