r/singularity
The next AI hardware race might be about inference
- upvotes
- 27
- comments
- 15
Post
Highlighted: the lines this signal was extracted from
The NVIDIA/Groq news got me looking at how fast different LLM providers actually serve tokens. Everyone obsesses over training bigger models, more GPUs. But running these things in production is a completely different problem. If AI agents actually end up in daily workflows, response speed and inference cost are going to matter way more than people think. I pulled some public numbers from different providers and Artificial Analysis. The gaps surprised me: * NVIDIA Groq 3 LPX: Gemma 4 31B - 3,400 tok/s (Artificial Analysis, 100K context) * Cerebras: GPT-OSS 120B - 3,000 tok/s official vs 1,715 tok/s in AA * Groq: Llama 3.1 8B - 1,800 tok/s; GPT-OSS 120B - 500 tok/s * SambaNova: Llama 3.3 70B - 476 tok/s I think AI infrastructure is shifting. Inference hardware is becoming as important as training hardware. Are specialized chips for serving models the next AI hardware race?