I'm scared about the future of security

TL;DR

AI models can now find real zero-day exploits trivially, ending the era where security relied on scarcity of elite hacker attention.

Key Points

• 1.AI is already finding critical vulnerabilities at scale. Claude Opus 4.6 generated 500 validated high-severity vulnerabilities and found 22 Firefox zero-days before release; all major React and Next.js exploits discovered this year were AI-assisted.
• 2.GPT-5 impressed elite hackers at Defcon with obscure Windows knowledge. A security expert who believed only ~5 people understood a specific Windows bug watched GPT-5 theorize its location and mechanics, producing a genuine 'what the f***' reaction.
• 3.GPT-4.5 Pro solved an unsolvable Defcon cryptography puzzle in 16 minutes. The 'C Shanty' puzzle from Gold Bug had fewer than 10 human solvers ever; the model solved it autonomously using Python execution and reasoning traces without internet search.
• 4.OpenAI is routing security-related requests away from o3/o4 models. Because o3 and o4's security capabilities are so dangerous, OpenAI silently downgrades suspected security/cyber-misuse prompts to o2 — the same tactic previously used only for mental health crises.
• 5.A trivial bash script loop can generate hundreds of exploitable CVEs. Anthropic red-teamer Nicholas Carlini spammed Claude Code with 'find me an exploit, start with this file' across every source file in a repo, then verified findings in a second pass — success rate was nearly 100%, including a SQL injection in Ghost CMS.
• 6.Security has always relied on attacker scarcity, not true hardness. Most software is safe not because it's unbreakable but because elite hackers are few and busy; AI ends this 'post-attention scarcity' era, making every router, printer, and regional bank database a viable target.
• 7.The leaked Anthropic 'Mythos' model post signals extreme internal concern. A leaked unfinished blog post about an unreleased model called Claude Mythos explicitly flagged cybersecurity risk assessment as a prerequisite for release, indicating Anthropic fears its own upcoming model's offensive capabilities.
• 8.Regulatory backlash could make things worse by pushing research underground. Politicians will likely craft incoherent security regulations in response to ransomware headlines, imposing asymmetric costs on defenders while open-weight Chinese models gain the same capabilities within 9 months regardless.