Old School / New Tech podcast

Talk: Your agent isn't stupid. It's blind.

0:00
26:18
15 Sekunden vorwärts
15 Sekunden vorwärts

Own the knowledge, rent the thinking: building organizational memory for AI agents

You can give an AI agent access to your company's wiki, Slack, CRM and project tracker, and it still won't know who owns a decision, which information is current, or what it's allowed to share. That isn't a model problem. It's a memory problem.

In this episode, recorded live at AI DevCon in Brooklyn, Ran Aroussi shares what VarOps learned from a year of building organizational memory for AI agents. The findings were tested against three public benchmarks, and the failures and negative results are published too.


Three architectural decisions that changed everything

  1. Store claims, not chunks. Distill information into structured knowledge, with evidence, timestamps and provenance attached.
  2. Make time and absence explicit. Track what changed, keep the history, and teach agents when to say "I don't know."
  3. Enforce permissions before inference. Models should never receive information that the person asking isn't authorized to see.

The results

  • 99.8% evidence retrieval on LongMemEval
  • 35 million tokens ingested for $8.24
  • Fully local models reaching 84–96% of all-frontier performance
  • Local memory paired with rented frontier reasoning reaching 96–99%
  • 45 of 45 adversarial isolation probes refused or contained

The bigger lesson

AI agents don't need bigger context windows or endless conversation histories. They need a governed understanding of the organization they work for. The model is a seat we rent. The memory is the asset we own.


Links


00:00 Why Agents Get It Wrong
01:03 The WhatsApp Contract Trap
03:33 Three Trust Questions
04:35 Stop Dumping Data
05:53 Living Org Model
08:27 Why RAG Breaks
09:51 Claims With Provenance
10:47 Distillation Works
13:06 Benchmarks And Costs
15:15 Permissions And History
16:50 Saying I Don't Know
19:51 Security Before Inference
20:41 Role Based Answers Demo
22:44 Attack Testing Results
23:28 Local And Hybrid Models
25:15 Key Takeaways And Wrap

Weitere Episoden von „Old School / New Tech“