Do not trust rankings, reviews and “experts” — test AI yourself on your own tasks
23 episodes in 12 months · 31 total · first in July 2024
Every company has its own benchmarks, and every week someone “wins”. Bloggers and “experts” often do not really use AI themselves and do not know your processes. So I do not judge a system until I have bought it myself and run it on my own tasks.
Judge a new model not by a single question but over weeks, on real tasks. And treat all the loud claims and “innovations” with caution until you have checked them yourself.
Standing rule. That is what I do myself: I buy every new system and run it on my own tasks (No. 078, October 2025).
Where to start
Before you trust a review or a ranking, run the model for a week on your own tasks.
Do not buy an AI solution based on a demo and other people’s rankings: test it on your own processes and data.
When I said it
Quotes
“Because you need a little, a little, well, like, some interval of time, for a few weeks or a month to pass in order to test different cases. Because those cases that I tested, it's hard for me to say yet that it's, yes, how much it's improved.”
“…I personally bought, probably, all the systems there were. In general, I think, all of them. I bought the absolute maximum Runway, all the systems.”
“And for all people, well, my personal recommendation, I strive for this myself, is to be very careful with any such claims, innovations, and to really test and check.”
“How do they know? Because they don't know your processes. The thing is, you have to know your processes, your system, what you're going to do, what kind of person you are, how you're going to maintain it.”