This week on Authorized Intelligence, Cal Al-Dhubaib speaks with Peter Gostev, AI Capability Lead at Arena, about how frontier AI models are evaluated, where current benchmarks fall short, and what their growing capabilities mean for enterprise security. They discuss measuring real-world performance, the cost of completing tasks, and why organizations should be using AI more actively to strengthen their defenses.
What You’ll Learn
- Why model evaluations need to keep evolving: As the ways people use AI become more complex, benchmarks must move beyond static tests and reflect real-world tasks.
- What makes a model useful: Performance is only part of the equation. Cost, efficiency, speed, and user satisfaction all factor into evaluating a model.
- Why benchmark scores need context: Strong results in controlled environments don’t necessarily translate to flawless performance in the real world.
- How organizations should approach AI security: Rather than focusing only on restricting internal AI tools, security teams should use AI to proactively test their systems and prepare for increasingly autonomous attacks.
Key Moments
-
00:00 What does BullshitEval reveal about frontier models?
-
04:00 How can real-world usage improve model evaluations?
-
07:30 Why should organizations measure the cost of completing a task?
-
10:45 What do cybersecurity benchmarks tell us about AI capabilities?
-
13:00 How should enterprises rethink AI and security?
-
18:00 What AI developments could reshape entire industries?
About Peter Gostev
Peter Gostev is the AI Capability Lead at Arena, where he works on evaluating the performance and capabilities of frontier AI models. He has explored new approaches to model evaluation, including BullshitEval, a benchmark designed to test whether models recognize flawed or nonsensical premises rather than confidently accepting them.
Listen & Subscribe
If you enjoyed this episode of Authorized Intelligence, subscribe wherever you listen to podcasts and leave us a rating or review. Have a question or a topic you’d like Cal to explore? Get in touch with us through authorizedintelligence.fm.