
As AI takes on more complex work, people need ways to understand how well it performs. Does a model follow instructions? Does it complete the task? How does one model compare to others?
In this episode, Matt Perault speaks with Arena cofounder and CEO Anastasios Angelopoulos about evaluating AI through real-world use. What began as a research project at UC Berkeley comparing chatbot responses has grown into a leading platform for measuring how models perform as people use them to get work done.
They discuss Arena’s Alignment Index, a measure of how often models misbehave, for example by acting without permission or reporting work as done when it isn’t. They explore what those measurements can—and cannot—tell us about AI’s capabilities and risks, and how competition to improve an Index score can encourage model developers to improve.
The conversation also examines why evaluation methods need to keep evolving, and why public policy should leave room for new evaluators to compete rather than locking in a handful of approved methods or incumbents.
For policymakers, better evaluations can inform judgments about capabilities and risk. For businesses, they can help guide model selection. And for builders, competition creates room to develop new ways of measuring AI as the technology advances.
Enjoy.
Topics covered:
02:06: Arena’s origins from chatbot comparisons to real-world work
06:06: Why independent evaluation matters
07:06: What makes an evaluator neutral?
10:06: Ecological validity and how a testing environment shapes measurement
14:06: Evaluating longer conversations and complex tasks
15:06: The Arena Alignment Index
18:06: How misalignment appears in real use
20:06: Separating measurement from policy debates
22:06: What evaluations can tell us about acceptable risk
26:06: Breaking down broad risks into measurable behaviors
28:06: Policy choices for the evaluation ecosystem
29:06: Why evaluation frameworks need room to evolve
32:06: Customer demand and competition as incentives
36:06: Building trust when rankings are contested
39:06: What Arena plans to measure next
40:06: What Arena’s leaderboards show about model competition today
43:06: Evaluating open and proprietary models
Resources:
Follow Anastasios on X: https://x.com/ml_angelopoulos
Follow Arena on X: https://x.com/arena
Learn more about the Arena Alignment Index: https://arena.ai/blog/ai-alignment-index
Follow Matt Perault on X: https://x.com/MattPerault
Subscribe to the a16z AI Policy Brief: https://a16zpolicy.substack.com/
The content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. Please note that a16z and its affiliates may maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.
This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit a16zpolicy.substack.com
Flere episoder fra "a16z AI Policy Brief"



Gå ikke glip af nogen episoder af “a16z AI Policy Brief” - abonnér på podcasten med gratisapp GetPodcast.








