Do not trust rankings, reviews and “experts” — test AI yourself on your own tasks
31 episodes in 12 months · 42 total · first in July 2024
Every company has its own benchmarks, and every week someone “wins”. Bloggers and “experts” often do not really use AI themselves and do not know your processes. So I do not judge a system until I have bought it myself and run it on my own tasks.
Judge a new model not by a single question but over weeks, on real tasks. And treat all the loud claims and “innovations” with caution until you have checked them yourself.
Standing rule. That is what I do myself: I buy every new system and run it on my own tasks (No. 078, October 2025).
Where to start
Before you trust a review or a ranking, run the model for a week on your own tasks.
Do not buy an AI solution based on a demo and other people’s rankings: test it on your own processes and data.
When I said it
Quotes
“How do they know? Because they don't know your processes. The thing is, you have to know your processes, your system, what you're going to do, what kind of person you are, how you're going to maintain it.”
“And how can you even compare systems with each other in terms of tokens, when what you actually need to compare is the end result? It’s obvious. A person asked themselves a question: has the result been achieved or not.”
“The seventh rule: you shouldn't trust ratings, reviews and various experts. It doesn't matter what kind of expert it is. Whether it's Alexander Volchek, some expert, or it's Sam Altman, some expert, or it's Elon Musk, some expert, right?”
“Look, an important aspect when you work in such systems is the number of tests. You have to do hundreds of different tasks, even the strangest tasks, in order to figure out what works better or what works worse.”