State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

TL;DR

China's open-weight models and US frontier labs are in a genuine race where compute budgets and organizational culture now matter more than ideas.

Key Points

• 1.DeepSeek R1 (January 2025) shocked the field with near state-of-the-art performance at allegedly far lower compute cost, sparking a wave of Chinese open-weight competitors including Kimi (Moonshot AI), MiniMax, Zhipu AI GLM, and Qwen.
• 2.Chinese companies release open-weight models strategically because US enterprises won't pay API subscriptions to Chinese firms for security reasons, so open weights are a trojan horse for global AI market influence.
• 3.Nathan and Sebastian predict more open-weight model builders in 2026 than 2025, with consolidation not expected until later; MiniMax and Moonshot AI have already filed IPO paperwork targeting Western mindshare.
• 4.Anthropic's Claude 3.5 Opus dominates the X/Twitter echo chamber for coding tasks, with Claude Code praised for its agentic, warm interface that handles full projects autonomously.
• 5.Google Gemini is predicted to keep closing the gap on ChatGPT in 2025-2026, backed by TPU infrastructure that avoids NVIDIA's massive margins and gives Google a full-stack cost advantage.
• 6.OpenAI's defining trait is landing paradigm-shifting research products — o1 thinking models, Deep Research, Sora — while also using o1 as a cost router that reduces GPU load from most users.
• 7.All three hosts use multiple models simultaneously: Claude Opus for code/philosophy, Gemini for long context and quick lookups, ChatGPT Pro for information queries, and Grok-3 Heavy for hardcore debugging.
• 8.Transformer architecture from GPT-2 remains essentially unchanged at its core; advances come from tweaks like Mixture of Experts (sparse activation of many expert feedforward networks), Multi-head Latent Attention, Grouped Query Attention, and Sliding Window Attention.
• 9.Mixture of Experts allows models to scale parameters without proportional compute increase — a router selects which expert networks process each token, enabling DeepSeek-V3's efficiency gains.
• 10.FP8 and FP4 training precision improvements (highlighted in NVIDIA announcements) increase tokens-per-second throughput (e.g., 10K to 13K tokens/sec), enabling faster experimentation without changing model architecture.
• 11.Three distinct scaling law axes now exist: pre-training (model size × data), reinforcement learning training (trial-and-error duration), and inference-time scaling (compute spent at generation), each showing log-linear performance gains.
• 12.Tool use (web search, Python interpreter calls) is identified as the key unlock for reducing hallucinations — models like Jamba pioneered this in open-weight form rather than memorizing answers.
• 13.Fully open models releasing training data and code include OLMo (Allen Institute for AI / AI2), LM360's K2, Apertus (Swiss), Hugging Face SmolLM, and Stanford's Meerkat; this list was nearly just AI2 in 2024.
• 14.US and European labs are moving toward larger MoE models: Mistral Large 3 released a giant MoE in December 2024, and NVIDIA Nemotron and RCAI have teased 400B+ parameter MoE models for Q1 2026.
• 15.Sebastian's books "Build a Large Language Model from Scratch" and "Build a Reasoning Model from Scratch" use GPT-2 as a base and add components incrementally to recreate LLaMA 3, OLMo, and other architectures, arguing code is the most precise way to understand these systems.