Inference

Inference on ToTheMoon is discussed almost entirely in terms of money and hardware rather than algorithms. The term surfaces when the hosts argue about where the real economics of AI sits: who pays for each answer, on which chips it is computed, and why speed of response is turning into a separately priced product.

As early as 2024, episode 007 brought inference up twice — as Groq's striking speed on a small model, and as a cost line into which companies pour investor money without positive unit economics. Discussing reasoning models in episode 024, the hosts split the cycle into pre-training, post-training and inference time and called the last the most important, because it teaches the model to think. In episode 040 and episode 043 inference became an argument about NVIDIA: since serving a model is simpler than training one, the company is more exposed to competition in that market.

From 2025 the conversation moved to speed and price. Episode 070 used Groq and Cerebras to make the point that companies often need not the largest model but cheap, very fast processing of millions of requests. Episode 098 explained the jump in model speed as architecture, inference, context handling and tool use improving together, and episode 105 predicted a surcharge for accelerated inference on dedicated hardware. Episode 106 offered the view that inference by itself is highly profitable and the companies would be in the black if the training race stopped. In episode 127 OpenAI's own chip was read as control over the cost of inference rather than prestige.