AI Engineering: Measure the PRs not the Code

AI agents can now be weaponized to run elite-level cyberattacks at scale — and your perimeter won't stop them. Box CTO Ben Kus lays out exactly how enterprises defend against agentic threats without locking down the tools their teams actually need. Ben Kus spent years at IBM's BigFix patching vulnerabilities at enterprise scale. Now at Box — where 95% of the codebase is permissions, security, and compliance — he's dealing with a new threat model: cheap, scalable AI agents that give any attacker access to elite hacking capabilities previously limited to nation-state actors. Only 14% of companies in VentureBeat's September survey said they could shut down a misbehaving AI agent within minutes. Ben explains why that gap is dangerous and what the architecture of a real defense looks like. The conversation gets technical fast. Ben walks through the "lethal trifecta" (untrusted input + sensitive data access + outside communication = critical risk), how Box uses action guardrails to sandbox agents even if they're compromised, why 44% of enterprise agents running on shared human credentials is a ticking clock, and how newer Anthropic models have dropped prompt injection success rates from 60% to roughly 2%. The second half shifts to retrieval. Ben breaks down why standard RAG fails at enterprise scale — permissions gaps, vector space degradation, poor lexical matching — and what Box's context retrieval system (hybrid embeddings + reciprocal rank fusion + agentic search) actually delivers: 90% token cost reduction and 4–5x speed improvement over naive model-provider retrieval. He also addresses the harness question directly: 59% of companies that swapped models in production saw performance drop, and the reason is usually the harness, not the model. 🎙️ GUEST: Ben Kus | CTO, Box 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat --- **CHAPTERS** 00:00 Intro — AI agents as elite-scale attack tools 02:00 Episode overview: Box CTO Ben Kus on agentic security and retrieval 03:00 What's actually changed in enterprise security: speed, severity, AI-assisted vulnerability discovery 04:20 How AI finds multi-domain vulnerabilities humans miss (PHP, font exploits) 05:00 When agents become attackers: democratizing elite hacking 06:20 Why perimeter defense is dead and what layered security actually looks like 08:00 VentureBeat survey: only 14% of companies can kill a misbehaving agent in minutes 09:20 Agent identity and credential sharing: when it's acceptable and when it's not 10:40 Misaligned agents: the agent-with-your-identity threat model 12:20 The lethal trifecta: untrusted input + sensitive data + outside communication 13:00 How Anthropic models dropped prompt injection rates from 60% to 2% 13:20 Box's approach: action guardrails, sandboxing, MCP server restrictions 14:00 Prompt injection via documents: how Box cleanses untrusted content 15:20 Action Guardrails feature: containing damage when agents are compromised 16:20 The SaaSpocalypse question: will Box become just a connector? 17:00 Why systems of record survive the agentic shift (headless platforms, MCP) 18:40 Agent-to-agent delegation: when Claude routes work to the Box agent 21:20 Why enterprise search has always been broken — and what's changing 22:40 Vector embeddings, semantic search, and the limits of pure RAG 24:00 Reciprocal rank fusion: combining lexical and vector search at scale 25:00 Box's "librarian" model: returning pages, not books 26:20 Context retrieval in Box Works: 90% token reduction, 4–5x speed improvement 29:40 Why throwing away shipped work every 3–6 months is now a core skill 31:00 59% of companies saw performance drop when swapping models — why the harness matters more than the model 33:00 Custom vs. generic harnesses: how the answer has changed three times in one year 33:40 Closing thoughts --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #CyberSecurity #RAG #AIInfrastructure Learn more about your ad choices. Visit megaphone.fm/adchoices

Home Depot cut delivery failure rates from 5.3% to 2.8% in Chicago using a purpose-built agentic system — and it's protecting 1.5 million at-risk deliveries a year across weather, geospatial, and customer preference signals. Home Depot CIO Angie Brown breaks down exactly how the sixth-largest U.S. e-commerce operation ($25B+ in online sales, 2,300+ stores) built its production AI stack from the ground up. The architecture is a layered agent hierarchy: a thin client planner routes to domain agents (selling, post-purchase, delivery), which hand off to fine-grained sub-agents by category — paint, power tools, configurables. Underneath sits a knowledge foundation built on Google BigQuery, hybrid search, unstructured vector databases, and a data mesh, all wrapped with agent identity management, security guardrails, and safety evals. On the model side, Home Depot runs Google ADK as its primary orchestration layer with Azure OpenAI held as an active counterweight to evaluate output quality — and Claude for code generation with OpenAI models used for code review. The Magic Forge internal platform handles their most compute-intensive workloads. Multi-model tension is a deliberate architectural choice, not an accident. Two production results stand out beyond delivery: a conversational IVR (Google NGA) now live in ~1,000 stores is handling 3x more customer questions without human transfer, and an AI-powered blueprint takeoff workflow for contractors turned quote turnaround from one-plus weeks down to 24–48 hours. Angie also covers where the architecture still has rough edges — deterministic escalation rules being layered back onto the conversational IVR — and what's next: per-SKU assortment optimization and agentic planogram generation across 30–40K SKUs per store. 🎙️ GUEST: Angie Brown | CIO, The Home Depot 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat --- 00:00 Intro & delivery failure rate results 00:57 Home Depot's technology footprint: 2,300 stores + e-commerce scale 03:20 Magic Apron walkthrough: AI-guided product discovery online 06:00 Fulfillment AI: how estimated delivery dates are computed in real time 08:20 Order Intelligence: weather, geospatial, and customer preference signals 10:40 Agent architecture deep dive: client planner, domain agents, sub-agents 13:20 Meta-agent question: how context inference replaces a top-level orchestrator 15:00 Full AI stack: Google ADK, MCP, LLM eval, prompt management, vector DBs 16:20 Multi-model strategy: Google vs. Azure OpenAI vs. Claude — how they decide 19:00 Magic Apron consumer rollout: what worked and what didn't 19:20 B2B pivot: List Builder AI and blueprint takeoff for contractors (24–48hr quotes) 22:20 How the agent framework has evolved: less summarization, more model intelligence 24:20 IVR in 1,000 stores: conversational AI replacing tree-based phone menus 28:00 Associate augmentation: Magic Apron in-store mode and plumbing aisle story 30:20 Customer adoption of AI IVR: what the call recordings actually reveal 32:20 Deterministic escalation controls layered onto conversational AI 32:40 LLM-referred traffic and the future of agentic commerce 37:00 What's next: agentic assortment optimization and planogram generation at scale 39:00 Outro --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #RetailAI #LLMInfrastructure #AgenticAI Learn more about your ad choices. Visit megaphone.fm/adchoices

A $600 day of AI coding spend sounds alarming — until you divide it by the PRs shipped. At ~$30/PR, it reframes the entire ROI conversation: the spend isn't the problem, spend with no return is. This episode comes from a live panel Sam moderated at VentureBeat Transform, featuring Emilie Schario (Kilo Code, acquired by Anaconda the morning of the panel), Amol Jain (Replit), and Jared Go (Symbiotic). The conversation cuts straight to what agentic coding actually looks like in production — not in demos. On architecture: Kilo routes across 500+ models and processes 10 trillion tokens/month, using expensive frontier models for planning and cheaper open-weight models for execution. Replit built an internal fleet of agents running in isolated cloud VMs — connected to GCP, Datadog, and Slack — that autonomously handled a gnarly multi-day bug and produced a merged PR with zero human intervention. Symbiotic runs tiered monthly token budgets ($150 / $450 / exception) managed directly by engineering managers, not finance. On org design: code review — not code generation — is now the bottleneck. Replit risk-scores every PR; low-risk ones auto-merge. The panel is aligned that "what does code review mean in 2025" is the defining question for the next 6–12 months. On the 6-month outlook: open-weight models take majority share of coding tasks, frontier models reserved for architecture decisions; 24–48 hour unattended agent work loops become standard; and model sovereignty (geographic data routing, on-prem isolated environments) becomes a hard enterprise requirement. --- 🎙️ GUESTS: Emilie Schario | Head of Engineering, Kilo Code (acquired by Anaconda) | Amol Jain | VP Engineering, Replit | Jared Go | Co-founder, Symbiotic 🎙️ HOST: Sam Witteveen | VentureBeat --- 00:00 Intro & Cost Per PR framing 00:40 Panel intro: Kilo Code, Symbiotic, Replit 02:20 Do engineers still read code? (Instacart CTO's 97% stat) 03:00 How Replit uses risk-scored PRs to auto-merge 04:20 Physical AI and safety gates for code review (Symbiotic) 05:00 Kilo Code: 500+ models, 10T tokens/month, Anaconda acquisition 07:20 Multi-model routing: frontier for planning, open-weight for execution 09:40 Symbiotic's stack: Cursor + homegrown SymCloud agentic harness 11:20 Replit's internal self-driving software engineering tool 12:20 MCP servers, skills repositories, and the 6-hour autonomous bug fix 13:20 Build vs. buy: where the real cost is maintenance, not construction 15:40 Replit replaces a seven-figure SaaS tool built entirely in-house 17:40 Token maxing: how teams burned their annual AI budget by July 18:20 Cost per PR as a ROI metric for AI coding spend 21:00 Symbiotic's tiered monthly token budget system for 900 engineers 23:20 Replit's 3x engineering productivity — and the GPT-5.5 Pro Max bill that forced a reckoning 24:20 Audience Q&A: model routing as a product layer for ISVs 26:20 Audience Q&A: PR review bottleneck, code deletion, and what review looks like next 29:40 6-month predictions: open-weight dominance, 24–48hr agent work loops, model sovereignty --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AgenticCoding #AIAgents #LLMInfrastructure #AIDeployment --- Learn more about your ad choices. Visit megaphone.fm/adchoices

66% of enterprise AI teams are actively engineering toward zero human oversight — but only 4.5% fully trust their automated evals. Half have already shipped an agent that passed evals and still failed a real customer. VentureBeat's latest research exposes a dangerous gap between deployment ambition and evaluation maturity. The most-used eval platform is OpenAI's native tooling. The second most common answer: no dedicated eval platform at all. Meanwhile, 51% of enterprises are relying on a hyperscaler's built-in guardrails as their primary AI security layer — not a deliberate security architecture. Fable's 19-day market absence changed the conversation. Enterprises that had built production workflows on frontier models got a wake-up call about single-vendor dependency. The result: GLM 5.2 is gaining serious developer traction as an open alternative competitive with Opus 4.8, and Tencent's new Hai 3 — less than half the size — is drawing attention for agentic workloads that don't require cutting-edge coding performance. Sam and Matt break down why companies like Shopify, LinkedIn, and Pinterest were already on open models, and why others are now catching up fast. On infrastructure: neo-clouds are currently used by just 2% of respondents, but 45% name them as their top planned investment area over the next 12 months. The market hasn't calcified. Matt and Sam preview what they expect to hear at VentureBeat Transform from engineering leaders at Intuit, Brex, Instacart, and Mastercard — including how Intuit bootstrapped and pivoted its agent architecture mid-build, and why agent conversations are generating higher-quality behavioral data than traditional user research ever did. 🎙️ HOSTS: Matt Marshall · Sam Witteveen | VentureBeat __ If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z — Chapters: 00:00 Research data drop: eval gaps, zero-HITL targets, and hyperscaler security dependency 01:00 Chinese models breaking into enterprise — developer fad or structural shift? 02:00 GLM 5.2 and Tencent Hai 3: what they do well and where they fall short 03:20 How Shopify, LinkedIn, and Pinterest are already running open models in production 05:40 Fable's 19-day outage: why 19 days changed enterprise AI vendor strategy 07:00 Intuit's agent build: bootstrapping, mid-build pivots, and real-time behavioral data 08:40 Neo-cloud infrastructure: 2% current usage, 45% planned investment next 12 months 09:20 Is the enterprise AI market still wide open? Anthropic vs. OpenAI for enterprise lead 11:20 The routing question: when do you send workloads to smarter vs. cheaper models? 12:20 Transform preview: what to expect from Amazon, Databricks, Brex, Instacart, Mastercard --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #LLM #MLOps #AgenticAI — Learn more about your ad choices. Visit megaphone.fm/adchoices

Shopify's distillation pipeline cuts production AI costs by up to 30x — and in some cases, the smaller model outperforms the frontier model on the narrow task. That's not a trade-off. That's a win on accuracy, latency, and cost simultaneously. Farhan Thawar, VP & Head of Engineering at Shopify, runs AI across one of the largest commerce platforms on earth. In this episode, he breaks down the exact infrastructure decisions Shopify made to avoid being locked into any single model provider — and why 29% of enterprise AI projects die from token costs, not model failure. Shopify built an internal LLM proxy that routes tokens across every major provider, enabling automatic failover when any one goes down. On top of that, their Universal Distillation Platform (UDP) lets any R&D team distill a frontier model (Opus 4, GPT-5+) down to a fine-tuned open source model (Qwen and others) for a specific subtask — in roughly a day, with evals baked in. Results range from 2x to 30x cheaper, faster, and more accurate than calling the frontier API for everything. Shopify currently runs roughly half a dozen of these distilled models in production, with more being added. Farhan also details River, Shopify's internal agentic substrate — a public-only Slack agent that queries their data warehouse, reads their PM system, and improves its own answers when engineers jump in to correct it. Plus: how Shopify governs AI-generated code at scale, why they killed their token leaderboard, how UCP is positioning Shopify's catalog for agentic commerce, and what a two-to-three year horizon looks like when agents start holding spending budgets and buying autonomously. 🎙️ GUEST: Farhan Thawar | VP & Head of Engineering, Shopify 🎙️ HOST: Sam Witteveen | VentureBeat __ If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z — **CHAPTERS** 00:00 Intro — Infrastructure-first philosophy at Shopify 02:00 Episode overview: token economy, distillation, and the cost crisis killing AI projects 03:00 Toby's "AI reflexivity" mandate and what it actually means for engineers 05:20 Ecosystem strategy: when to build vs. leave room for third-party developers 06:20 Developer tooling stack: GitHub Copilot (2021), Claude Code, Cursor, Codex, and Shopify's own River 08:00 LLM proxy architecture: bulk token purchasing, multi-provider failover, and usage reporting 09:00 Model agnosticism: why Shopify lets engineers choose their own harness 10:00 AI adoption beyond R&D — finance, HR, sales, and the Qwik internal deployment platform 11:20 Code governance: who owns AI-generated code going to production 13:00 Token maxing, leaderboards, and the shift from AI reflexivity to AI leverage 14:20 Circuit breakers: how Shopify catches runaway token spend without hard limits 15:20 LLM proxy deep dive: uptime, insights, and cross-team learning from usage data 16:40 River: Shopify's agentic substrate, public-channel-only design, and emergent HITL behavior 18:40 Model distillation explained: teacher/student models, narrow tasks, and the trade-offs 21:00 Universal Distillation Platform (UDP): how any team submits a distillation job in ~one day 22:20 Tangle: open source pipeline visualization for distillation workflows 22:40 Who uses UDP today, and Farhan's vision for auto-selecting the distillation target model 24:00 Evals in practice: golden datasets, Toloka for data generation, threshold-based deployment 26:20 Sim Gym: simulating A/B tests for small merchants without enough traffic 27:40 Pulse: async AI insights on store performance and conversion 28:20 GPU infrastructure trade-offs: when running your own inference makes sense at scale 29:20 Frontier vs. distilled model split in production — and why dev tokens stay frontier 30:40 Should Shopify train its own coding model? Why Farhan says not yet 31:40 UCP protocol, agentic commerce, and how Shopify's catalog surfaces in every LLM 35:00 Early signals: agentic commerce growth rate and the shift away from SEO 36:00 What developers and entrepreneurs should build for now given multi-channel uncertainty 37:20 River as a hive-mind agent — and what "truly agentic" actually means 38:40 Two-to-three year forecast: agents with spending budgets, autonomous purchasing, proactive outreach 40:00 Project Glasswing, model access loss (Fable), and the case for multi-provider architecture 42:00 Why every company should have a backup plan — and how Shopify built theirs years ago 43:20 Wrap-up --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #LLM #MLOps #AgenticAI — Learn more about your ad choices. Visit megaphone.fm/adchoices

If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z _ He negotiated seat-based (unlimited) licenses before token costs exploded — and projects only a 20–30% spend increase when that changes. MassMutual's CIO rebuilt a COBOL mainframe app into a working web prototype in 7 days — work that used to take a 15-person SI team 90 days. That's not a pilot. That's a new build-vs-buy equation. Sears Merritt, CIO at MassMutual, runs AI inside one of America's most regulated legacy environments. In this conversation, he breaks down the architecture decisions, cost structures, and security posture behind real, production deployments — not roadmaps. On infrastructure: MassMutual routes all agentic tool calls through centralized API gateways with identity and access controls, using Amazon Bedrock as a proxy layer. That multi-harness design preserves model optionality while enforcing FinOps discipline. On model selection: a trust score rubric drives every model decision, balancing cost against user experience. In their IT contact center, that rubric led them to choose the more expensive model after users said the quality gap was worth two extra seconds of inference time. Productivity results are concrete: 30% boost in developer output across the SDLC, call resolution time dropping from 10 minutes to under 1 minute for specific call types, cost per interaction from dollars to cents. On the security side, MassMutual is embedding AI into its SDLC for vulnerability scanning and compressing cyber response cycles from days to hours — building agentic tier-one and tier-two capabilities to match the accelerated threat landscape that frontier models like Mythos have exposed. 🎙️ GUEST: Sears Merritt | Head of Enterprise Technology & Experience, MassMutual 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat — If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z — 00:00 Intro & COBOL Modernization Preview 00:01:15 Guest Introduction: Sears Merritt, MassMutual CIO 00:02:00 Multi-Vendor Strategy & Avoiding Lock-In 00:02:30 How MassMutual Evaluates AI Tools (Cost vs. Experience Rubric) 00:03:15 12-Month Contracts and Switching Optionality 00:03:30 AI Standards Cycle: MCP, A2A, and the Early Internet Analogy 00:05:15 Measuring Developer Productivity: 30% SDLC Boost 00:06:15 Contact Center Results: 10 Minutes to 1 Minute, Dollars to Cents 00:07:00 Managing Token Cost Explosion 00:07:30 Seat-Based vs. Consumption Licensing Decision 00:08:45 Token Maxing While the All-You-Can-Eat Window Is Open 00:09:45 Building FinOps Infrastructure for Model Routing and Optimization 00:11:15 Outcome-First Model Selection: When to Pay for Opus vs. a Cheaper LLM 00:13:15 Trust Score Framework: How MassMutual Picks the Right Model 00:15:00 Sponsor: OutShift by Cisco 00:15:30 Claude, OpenAI Codex, and Multi-Harness Agentic Architecture 00:16:30 API Gateway Design: Identity, Access, and FinOps Controls 00:17:30 What the Usage Analytics Revealed (And What Merritt Was Afraid to Find) 00:18:15 Projected Token Cost Increase: 20–30% Off Unlimited Plan 00:19:45 Power Law Usage: Top 10% Consuming 80% of Tokens 00:20:45 COBOL Mainframe Modernization: The 7-Day Prototype Workflow 00:22:30 The Full AI-Assisted COBOL Migration Playbook 00:24:00 Implications for IBM and Mainframe-as-a-Service Providers 00:25:15 Open Source Models, DeepSeek, and the Cost Efficiency Question 00:27:45 Chinese Models in a Regulated Environment: Evaluation Criteria 00:29:30 Agentic Security: Identity Management and the Evolving Threat Landscape 00:30:00 How Frontier Models Changed the Threat Velocity (Not the Threat Types) 00:31:15 Fighting AI With AI: Agentic Tier-1 and Tier-2 Cyber Capabilities 00:31:30 Project Glasswing and the CISO Community Response 00:32:30 Embedding AI Into the SDLC for Security Scanning 00:34:00 Closing: When Will Agentic Standards Consolidate? Advice for Builders --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #LLMInfrastructure #AIDeployment #MLOps Learn more about your ad choices. Visit megaphone.fm/adchoices

Pinterest's open-source AI stack costs 90% less than frontier models — and their custom-trained recommender outperforms off-the-shelf alternatives by 30% in accuracy. Pinterest CTO Matt Madrigal breaks down exactly how they did it, and what enterprise AI teams can actually replicate. Madrigal walks through the full architecture behind Navigator 1, Pinterest's conversational shopping assistant built on Qwen 3 VL — and the specific decision to rip out its native vision encoder and replace it with PinCLIP, Pinterest's proprietary multimodal embedding layer. That swap alone closes a 20x inference latency gap and makes the economics work at 620 million monthly active users. This is the clearest public explanation yet of how a scaled platform operationalizes the "core vs. context" principle for model selection: open-source and custom-built where it touches the user, frontier models where speed-to-prototype matters more than cost. The conversation also covers the Taste Graph — Pinterest's knowledge graph across hundreds of billions of pins and 15 billion boards — and how post-training on that proprietary data lets a smaller, fit-for-purpose model beat a larger frontier model on production metrics. Madrigal details their eval framework: gold set benchmarks, product-level evals tied to engagement and merchant click outcomes, and a structured A/B test pipeline that runs from engineer PRs through to live user signal. On the organizational side: how Pinterest manages a "default yes" multi-IDE policy (Cursor, Windsurf, Claude Code, Codex) without collapsing security posture, how they segment sandbox environments between ML engineers with Taste Graph access and general application developers, and why Madrigal measures AI coding ROI in token usage and experimentation velocity — not lines of code. 🎙️ GUEST: Matt Madrigal | CTO, Pinterest 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat 00:00 Show Intro and Guest 01:17 Open Source Cost Breakdown 02:20 Pinterest Multimodal Roots 02:37 PinClip and Embeddings 05:46 Core vs Context Models 07:43 Navigator 1 Assistant Stack 11:52 Benchmarking and Evals 13:29 Accuracy from Proprietary Data 17:16 Taste Graph Explained 18:29 Taste Graph in Training 22:22 Fighting AI Slop 25:16 Developer Tools and Velocity 27:57 Tool Choice and Governance 28:56 Security Sandboxes and CICD 30:57 Wrap Up If you enjoy these conversations, you need to be in Menlo Park this July. VB Transform 2026 is VentureBeat's flagship enterprise AI event, built entirely around one question: How do you orchestrate AI autonomy at scale? July 14–15, Hotel Nia. Real projects, proprietary research, no fluff. 50% off for listeners with code BEYONDTHEPILOT: https://bit.ly/4fK4F6z Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #OpenSourceAI #AIInfrastructure #LLM #MachineLearning Learn more about your ad choices. Visit megaphone.fm/adchoices

Enterprise GPU hoarding is over. LinkedIn CTO Erran Berger and VentureBeat analyst Rob Strechay break down what comes next — and the infrastructure math most enterprises are only now being forced to confront. VentureBeat's Q1 research shows GPU availability anxiety dropped from 20.8% to 15.4% among enterprise teams, while cost-per-inference and TCO concerns jumped from 34% to 41% — a number that's still climbing. The hoarding phase is giving way to an audit phase, and the companies that didn't build the instrumentation to understand their workloads are now paying for it. Erran Berger explains how LinkedIn runs one of the few remaining at-scale applied ML shops outside the hyperscalers — owning the full stack from bare metal GPU clusters to member-facing products. That means LinkedIn engineers can optimize custom CUDA kernels, compress embeddings, prune models for throughput, and adapt networking and storage per workload — trade-offs that are simply unavailable on public cloud instance menus. The result: a rigorous ROI framework that evaluates not just current traffic costs, but the traffic shape agents will drive in 2–3 years. On the market side, 72% of enterprises admit they lack sufficient control over their AI infrastructure. Open-source inference tools like vLLM and LLMD are seeing rapid adoption, while 17% of organizations have moved to full-stack ownership. Hyperscalers report 60–80% of workloads have already shifted from training to inference — and most enterprise teams are still figuring out how to staff and instrument for that reality. 🎙️ GUEST: Erran Berger | CTO, LinkedIn 🎙️ ANALYST: Rob Strechay | VentureBeat 🎙️ HOST: Matt Marshall | CEO, VentureBeat --- 00:00 Intro: The GPU Hoarding Hangover 00:10 Guest Introductions 02:00 VentureBeat Q1 Data: GPU Panic Fades, TCO Concerns Rise 03:00 LinkedIn's Early Shift to Inference ROI Discipline 04:00 Budget Moving Into Inference Optimization and Control 07:00 LinkedIn's Full-Stack Advantage: Kernels, Pruning, Embedding Compression 08:00 Private AI and Sovereign Stacks: What the Q1 Data Shows 09:00 Open Source Inference Tooling: vLLM, LLMD, RDMA 10:00 Data Sovereignty at LinkedIn Scale: Member Data and Board-Level ROI Framing 12:00 Why Instrumentation Beats GPU Hoarding 13:00 Planning for Ambient Agent Traffic — Not Just Today's Workloads 14:00 Closing Advice for the Enterprise CTO Staring at 5% GPU Utilization --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIInfrastructure #MLOps #InferenceOptimization #GenerativeAI --- Learn more about your ad choices. Visit megaphone.fm/adchoices

The CEO who built one of the most-starred RAG frameworks on GitHub (47,000 stars) just publicly declared that frameworks like his are becoming obsolete — and then pivoted his entire company around that conclusion. Jerry Liu, CEO and co-founder of LlamaIndex, joins Matt Marshall and Sam Witteveen to explain exactly what broke in the AI stack, why 95% of his team's code is now AI-generated, and where the real defensibility in enterprise AI infrastructure actually lives in 2026. The conversation covers the specific architectural shift that made RAG orchestration frameworks less central: agent reasoning has improved to the point where dumb tools plus smart agents outperform sophisticated retrieval pipelines, coding agents have collapsed the cost of custom integrations, and model providers like Anthropic are consolidating the harness layer around MCP, sandboxes, and session state. Jerry walks through Anthropic's managed agent diagram as a real architectural reference point and explains why engineering leaders should prioritize modular interfaces over implementation investment — because parts of your current stack will need to be thrown away in months, not years. On SaaS survival, Jerry argues the companies that retain value are those becoming systems of record — and that the real opportunity is building AI agents that automate labor on top of their platforms, not defending UI/UX that agents are now bypassing. On LlamaIndex's own bet: document understanding — parsing PDFs, tables, charts, and forms at higher accuracy and lower cost than frontier models — is the context layer every agent stack needs regardless of which model wins the next benchmark cycle. LlamaParse and the newly released open-source ParseBench (April 13) are the commercial expression of that thesis. If you're evaluating your AI stack architecture, deciding how much to build vs. buy, or trying to understand where horizontal tooling still has a moat, this episode is the conversation. 🎙️ GUEST: Jerry Liu | CEO & Co-Founder, LlamaIndex 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat --- **CHAPTERS** 00:00 Intro — LlamaIndex's origin and RAG framework origins 02:00 How LlamaIndex started: GPT-3, 4K context windows, and GPT Index 04:00 Why AI frameworks are becoming less useful in the agentic era 07:00 What changed in the stack: agent reasoning, coding agents, and RAG's evolution 09:00 How Anthropic's managed agent diagram reframes enterprise architecture 13:00 The lock-in question: managed agents, session state, and stack modularity 16:00 Should you build horizontal tooling? Why Jerry says probably not 18:00 Open vs. closed: the Apple/Android analogy applied to frontier labs 21:00 The abstraction level is rising — English is the new programming language 24:00 SaaS market cap destruction: who survives agents eating software 28:00 The "full stack builder" emergence and the future of SaaS seats 31:00 Buy vs. build for agents: the AI recruiter thought experiment 33:00 LlamaIndex's pivot: document understanding as defensible infrastructure 36:00 Why frontier models won't commoditize specialized document parsing 38:00 LlamaParse deep dive: zero-shot accuracy, tables, charts, handwriting 41:00 LightParse, ParseBench, and designing for agent consumers 44:00 Wrap-up and where to follow LlamaIndex --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #LLMInfrastructure #RAG #AIArchitecture --- Learn more about your ad choices. Visit megaphone.fm/adchoices

Cisco's OutShift deployed a multi-agent network configuration system that raised error detection from 10–15% to 100% and cut full change validation from 2–3 weeks to 6–7 minutes. The reason it worked — and why most enterprise multi-agent deployments still fail — comes down to a single gap nobody is talking about: agents can connect, but they cannot think together. Vijoy Pandey, SVP and General Manager of OutShift by Cisco, joins Matt and Sam to explain why A2A, MCP, and existing agent protocols solve connectivity but leave out an entire layer: shared cognition. OutShift's research identifies this as a missing "Layer 9" — a semantic and cognitive communication stack above today's syntactic protocols — and they're already building it. The conversation covers the four pillars of enterprise-grade multi-agent infrastructure (discovery, identity/access, communication, observability), why standard IAM models break when agents enter the picture, and how OutShift extended OpenTelemetry with Microsoft to cover multi-agent evaluation. Vijoy introduces three new cognition-state protocols — SSTP (Semantic State Transfer), LSTP (Latent Space Transfer), and CSTP (Compressed State Transfer) — and explains the staged rollout path for each, including a published MIT collaboration called the Ripple Effect Protocol. The healthcare scheduling case study is particularly instructive: three independent third-party agents — insurance, diagnostics, scheduling — each with competing optimization functions and siloed context, and zero shared intent. That's the real multi-vendor, multi-org enterprise problem. Vijoy explains what an orchestrator can't fix, and what a cognitive fabric layer would. 🎙️ GUEST: Vijoy Pandey | SVP & General Manager, OutShift by Cisco 🎙️ HOSTS: Matt Marshall | VentureBeat, Sam Witteveen | VentureBeat --- **CHAPTERS** 00:00 Intro & Cold Open: Agents Connect But Can't Think Together 00:03 Welcome & Guest Introduction: Vijoy Pandey, OutShift by Cisco 00:04 Do Agents Work Outside Coding & Customer Support? Challenging Amjad Masad's Diagnosis 00:05 What's Wrong With A2A and MCP? The Four Pillars of AGNTCY 00:08 Identity & Access Management for Agents: Why IAM Breaks and What TBAC Fixes 00:12 The Network Digital Twin: How OutShift Achieved 100% Error Detection in Production 00:13 From 2–3 Weeks to 6–7 Minutes: Real Results From Deployed Multi-Agent Networking 00:15 Agents Can Connect But Can't Think Together: The Core Thesis 00:20 The Cognitive Revolution Analogy: Shared Intent, Shared Context, Collective Innovation 00:25 The Healthcare Scheduling Case Study: Three Competing Agents, Zero Shared Intent 00:31 Why Orchestrators Fail in Multi-Vendor, Multi-Org Environments 00:36 Introducing Layer 9: SSTP, LSTP, and CSTP — The Cognition-State Protocol Stack 00:41 What OutShift Is Building Now: Protocols, Fabric, and Cognition Engines 00:44 MIT Collaboration: The Ripple Effect Protocol and Phase One Rollout 00:46 Cisco's 40-Year Networking Playbook Applied to the Internet of Cognition 00:49 Closing: Where to Find the Research, AGNTCY, and OpenClaw Integration --- Subscribe to VentureBeat: https://www.youtube.com/@VentureBeat Apple Podcasts: https://podcasts.apple.com/us/podcast/venturebeat/id1839285239 Spotify: https://open.spotify.com/show/4Zti73yb4hmiTNa7pEYls4 Website: https://venturebeat.com LinkedIn: https://www.linkedin.com/company/venturebeat Newsletter: https://venturebeat.com/newsletters #EnterpriseAI #AIAgents #MultiAgentSystems #AIInfrastructure #LLM — “Scaling Out Superintelligence”  Vijoy Pandey, January 2026. The technical whitepaper detailing the Internet of Cognition architecture, three-layer stack, and cognition state protocols.  Internet of Cognition Interactive Demo Clickable walkthrough showing per-agent activity, intent, context, and collective reasoning across a multi-agent SRE system.  “A Layered Protocol Architecture for the Internet of Agents”  Fleming, Muscariello, Pandey, Kompella. The OSI Layer 8/9 extension.  AGNTCY Open source multi-agent infrastructure under Linux Foundation governance. Covers discovery, identity, communication, observability.  Formative members: Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat.  Learn more about your ad choices. Visit megaphone.fm/adchoices