Google's New AI Just Broke My Brain

TL;DR

Google's TurboQuant compresses AI memory 30-40% and speeds up processing 40% by combining three old mathematical techniques in a novel way.

Key Points

• 1.TurboQuant cuts KV cache memory by 30-40% and speeds up prompt processing by ~40%. Media claimed 4-6x less memory and 8x faster attention, but independent benchmarks show more modest — still impressive — real-world gains.
• 2.The technique combines three decades-old ideas, not new inventions. Quantization (chopping number digits), random rotation to spread vector energy evenly, and the 40-year-old Johnson–Lindenstrauss Transform are merged to minimize information loss during compression.
• 3.The random rotation trick is the clever core insight. Before rounding off numbers, rotating vectors spreads their 'energy' across all directions so truncation loses a little from everywhere rather than everything from one place.
• 4.Independent researchers successfully reproduced and benchmarked the technique within a week of publication. Community members even coded it up independently, confirming real benefits especially for long-context tasks like analyzing large PDFs, movies, or codebases.
• 5.The paper is controversial — other researchers flagged overlap with prior techniques they felt wasn't fully acknowledged. The paper was ultimately accepted for publication, but not all concerns were considered resolved, highlighting ongoing credit disputes in AI research.