(Re)Engineering the Inference Stack

In this episode, Nicole is joined by Stephen Bates, AMD Fellow in the AI Group, and Alon Horev, Co-founder and CTO of VAST Data, to explore one of the biggest architectural shifts happening in AI today: the move from training to inference. Using AMD and VAST's expanded partnership as a starting point, the conversation examines why KV cache has become a critical part of AI infrastructure, how long-context and agentic AI are changing the way systems are designed, and why storage, networking, and open software are becoming just as important as the GPUs themselves in delivering fast, scalable inference.

AI infrastructure is evolving faster than ever, reshaping how organizations approach performance, efficiency, and long-term architectural decisions. In this episode of Shared Everything, Nicole talks with Anat Heilper, Director of AI Architecture at VAST Data, about what optimization really means in 2026. Drawing on more than 14 years at Intel developing AI accelerators and large-scale training and inference systems, Anat explores the shifting balance between compute, memory, and data movement, the importance of KV cache reuse, and why inference optimization now extends far beyond the GPU. The conversation also looks ahead to the demands of agentic AI, multiple models, and emerging physical AI applications. Anat explains why scale, flexibility, and shared infrastructure are increasingly critical as enterprises move AI into production. With models and frameworks changing rapidly but infrastructure investments expected to last for years, the central challenge is clear: how to optimize for today's workloads while building systems ready for whatever comes next.

At CoreWeave, building AI infrastructure at massive scale means rethinking the datacenter as a system. Jacob Yundt, Senior Director of Compute Architecture at CoreWeave, joins us to explore how GPU infrastructure is evolving as racks become tightly integrated computers and power, cooling, networking, storage, firmware and software orchestration become inseparable parts of the architecture. We dig into the demands of inference, heterogeneous GPU fleets, workload placement, observability and resiliency, and why the next generation of AI infrastructure may require much more than simply scaling up the designs we use today.

AI development is undergoing a fundamental shift as coding agents, advanced models, and preconfigured infrastructure dramatically compress the time required to build sophisticated applications. In this episode, VAST Data's Brian Verkley and Ram Bansal explore the emerging AI builder stack and how bringing together Cursor, NVIDIA models, CoreWeave GPUs, and the VAST AI OS enables developers to create functional video AI agents in just hours. Drawing on real-world examples spanning robotics, urban infrastructure, manufacturing, and video analytics, they examine how removing the complexity of data ingestion, processing, and infrastructure management opens up new possibilities for developers. The conversation also explores what these rapid development cycles reveal about the future of software engineering, the growing importance of agentic tools, and the challenges of moving AI applications from prototype to production.

Akamai has spent nearly three decades distributing infrastructure closer to where users and data live. Now AI inference is giving that architecture a new job. Akamai’s John Alexander and Ari Weil join to discuss the shift from centralized AI factories to distributed inference, the emergence of the AI grid, and the infrastructure required to make real-time AI work at scale. They dig into orchestration across CPUs and GPUs, latency and data locality, networking and power constraints, and why data engineering becomes increasingly difficult as inference spreads across locations. The conversation also explores how Akamai, VAST Data and NVIDIA are bringing distributed compute, high-speed data pipelines and a global namespace together to create a more unified architecture for real-time and agentic AI.

AI agents are moving from tools that assist people to systems that can search data, make decisions and take action on their own. That shift creates a new set of infrastructure problems for enterprises. In this episode, VAST Data co-founder and CTO Alon Horev explains why agents need secure sandboxes, detailed observability and tightly controlled access to enterprise data and tools. He also looks at what happens as companies deploy thousands of agents at once, putting new pressure on search, structured data systems and real-time data access, and why these challenges become even harder as agentic systems extend from the datacenter to the edge and physical AI.

VAST Data CEO and co-founder Renen Hallak joins us to explore what happens as AI moves from a handful of models to a world of millions of agents continuously using, fine-tuning and creating models of their own. We dig into model management, confidential computing, AI ownership and the growing importance of model weights as enterprise IP, along with the infrastructure required to route workloads across models, data and GPUs securely and efficiently. Renen also looks ahead to an agent economy operating at a scale far beyond systems designed for humans, and why the software infrastructure layer will be critical to making that future manageable, secure and economically viable.

AI is changing the I/O profile of supercomputing, and CINECA is seeing that shift directly on Leonardo, where AI now accounts for more than half of the workload. Daniele Cesarini, Head of AI/HPC Architecture at CINECA, explains why AI training and inference introduce data-access patterns that differ sharply from traditional simulation, putting new pressure on parallel file systems, metadata performance, small-file access and GPU data pipelines. The conversation gets into CINECA’s move toward a more flexible data architecture capable of serving both HPC and AI, the role of object and multiple access protocols alongside conventional HPC storage, and how lessons from Leonardo are informing the architecture of CINECA’s next AI infrastructure, Italia, where keeping more than 8,000 GPUs efficiently fed becomes a data problem as much as a compute problem.

AI factories are changing what research computing looks like as HPC and AI converge around increasingly powerful, data-hungry infrastructure. HPC industry veteran and current head of the charge for research and education in the Americas at VAST, Chris Ginder, joins us to discuss how research institutions are approaching shared AI infrastructure, why GPUs alone don’t define an AI factory, and the storage, networking and data architecture required to keep those systems productive. The conversation explores what this shift means for traditional HPC environments and, ultimately, how researchers can turn massive compute capability into useful scientific work.

Some of the most valuable lessons in computing never make it into the success stories. In this episode, HPC and AI veteran Glenn Lockwood draws on firsthand experience across scientific computing, national labs and hyperscale AI (NERSC, SDSC and Microsoft on its Fairwater supercomputer) to revisit ambitious systems that broke new ground, while revealing the technologies, assumptions and architectural bets within them that didn’t work quite as expected. From attempts to make distributed computing complexity disappear, to a high-performance storage technology users largely chose not to use, to enormous AI infrastructure bets overtaken by changes in how frontier models are built, these are remarkably candid stories from inside consequential computing projects. The failures expose lessons about complexity, adoption, general-purpose versus specialized technology, and what happens when workloads evolve faster than infrastructure can.