Module 7: The LLM Application Loop

In this episode, we bring everything together by following one LLM application from the user's first request to the final response. We see retrieval, context, tools, model calls, state, memory and orchestration working as one system, revealing the bigger lesson of the module: building with LLMs is not just about what the model can do, but deciding what the model should do and what the rest of the application should handle.

In this episode, we map the modern LLM application stack and explore where frameworks, SDKs, runtimes and protocols actually fit. We look at how tools like LangChain, LangGraph, LlamaIndex, provider SDKs, durable workflows and MCP solve different problems, and why understanding the architecture matters more than memorizing product names.

This episode separates two concepts that are often confused: state, which keeps track of what is happening now, and memory, which allows information from the past to become useful later. We explore how applications create continuity around a model, and why good memory is not about remembering everything, but remembering the right things at the right time.

This episode breaks down what orchestration actually means, from sequencing and routing to retries, parallel execution and human approvals, and explores the different ways those flows can be managed. Most importantly, we separate orchestration from the model itself and show why it is really about controlling how work moves through an application.

Who actually decides what happens next inside an LLM application? This episode explores the difference between decisions made by code and decisions made by the model, and why real applications often use both. We follow the loop that emerges when a model can request information or actions, receive the results and decide what to do next, revealing what developers mean when they talk about “owning the loop” and setting the stage for orchestration.

If LLM applications can be built with regular code and APIs, why do frameworks exist at all? This episode explores what happens as a simple application grows and starts needing retrieval, memory, routing, multiple models, retries and tracing. We look at what frameworks actually take off the developer’s plate, when those abstractions become useful, and why sometimes plain code is still the better choice.    

What actually sits behind an LLM application? This episode takes one simple request and follows it beneath the surface, revealing how the model, application code, APIs, external data and context work together to produce something genuinely useful. As the request gets more complex, we begin to see why concepts like memory, tools and orchestration enter the picture. It is a practical look at what we are really building when we say we are building with LLMs.

This episode closes out Module 6 by tackling the question that has been getting louder since large context windows arrived. If a model can hold hundreds of thousands or even millions of tokens at once, do we still need all the architecture we just spent this module building? We explore why RAG was never just about fitting text into a small prompt, what retrieval is actually doing that a large context window cannot, and how the shift from compression to curation changes what good RAG looks like today. We cover when long context is genuinely the better tool, when retrieval still matters deeply, and why in most real enterprise systems the best answer is both working together. The episode closes with the argument that RAG is not disappearing. It is maturing. And everything we built in this module is part of that stronger foundation. By the end you will have a clear and honest picture of where these two approaches fit, and why understanding both puts you well ahead of most people working in this space.

This episode addresses the category of questions that vector search fundamentally cannot answer, questions about relationships between things. We explore what a knowledge graph is and why traversing connections between entities requires a completely different data structure than semantic similarity search. We break down Microsoft's GraphRAG approach, how it extracts entities and relationships from documents during indexing, uses community detection to identify clusters of related knowledge, and generates summaries that enable global queries across an entire corpus rather than just local document retrieval. We cover the cost improvements brought by LazyGraphRAG, the hybrid vector-plus-graph pattern most production teams are moving toward, Neo4j as the go-to graph database, and a lighter-weight entity extraction approach for teams not ready for a full knowledge graph. By the end you will understand when relationships matter more than text and how to build systems that can answer both kinds of questions.

This episode addresses a retrieval failure that has nothing to do with your index and everything to do with the query itself. We explore the vocabulary gap between how people ask questions and how documents are written, and why even strong embedding models cannot always bridge it. We break down three techniques that fix the query before the search runs: query rewriting to reformulate casual language into formal search terms, HyDE which generates a hypothetical answer and uses that as the search query instead of the question, and multi-query expansion which generates multiple phrasings to cast a wider retrieval net. We also cover step-back prompting for queries that need broader conceptual grounding before searching. By the end you will understand why the question itself is often the highest-leverage thing to improve in a retrieval pipeline.