2-7-2. Threat Modeling for LLM Systems: A Step-by-Step Guide

This chapter examines prompt injection and jailbreak attacks, which exploit a language model's inherent inability to distinguish between authoritative developer instructions and untrusted user data. It covers the mechanics of direct and indirect injection, categorises common jailbreaking techniques, discusses the limitations of defensive prompt engineering, and outlines a layered mitigation strategy to better defend production systems.Amazon.com: LLM Primer VII AI Security: Defending LLM Systems Against Prompt Injection, Jailbreaks, and Adversarial Threats: 9798185644065: SHIMODA, SHO: Books

This chapter examines data security and privacy throughout the LLM lifecycle. It explores the inherent risks of training data, such as copyright issues, personal information (PII) contamination, and data poisoning. Additionally, it details how models can leak sensitive information through memorization and extraction attacks, and outlines operational defenses for securing systems, including input redaction pipelines, encryption, tenant isolation, and data retention policies.Amazon.com: LLM Primer VII AI Security: Defending LLM Systems Against Prompt Injection, Jailbreaks, and Adversarial Threats: 9798185644065: SHIMODA, SHO: Books

This chapter adapts traditional threat modeling frameworks (such as STRIDE, PASTA, and attack trees) specifically for the unique vulnerabilities of LLM systems. It guides defenders through identifying AI-specific assets and adversaries, and provides a step-by-step procedure for building a living threat model that can be maintained alongside the system's codeAmazon.com: LLM Primer VII AI Security: Defending LLM Systems Against Prompt Injection, Jailbreaks, and Adversarial Threats: 9798185644065: SHIMODA, SHO: Books

This chapter explains that AI security fundamentally and structurally differs from traditional software security. Instead of finding and patching clear bugs in readable source code, defenders must secure probabilistic models whose behaviors are driven by billions of uninterpretable weights and training data. Consequently, the security focus shifts from ensuring code correctness to managing and restricting an unbounded range of unpredictable inputs and outputs. Amazon.com: LLM Primer VII AI Security: Defending LLM Systems Against Prompt Injection, Jailbreaks, and Adversarial Threats: 9798185644065: SHIMODA, SHO: Books

This episode covers Chapter 7, examining why Large Language Models confidently generate false information. We discuss the probabilistic nature of "hallucinations," the dangerous gap between fluency and correctness, and practical strategies like calibration and hybrid verification to align model confidence with reality.Amazon.com: LLM Primer VII AI Security: Design Safe and Robust AI System eBook : SHIMODA, SHO: Kindle Store

This episode covers Chapter 6, focusing on the security implications of connecting models to external data (RAG). We discuss how this introduces new trust boundaries, the dangers of malicious document injection where attackers plant traps in your knowledge base, and the necessity of validating documents before they enter the model's context.Amazon.com: LLM Primer VII AI Security: Design Safe and Robust AI System eBook : SHIMODA, SHO: Kindle Store

This episode covers Chapter 5, detailing how to build disciplined pipelines around an AI model. We discuss strategies for sanitizing user inputs to catch attacks early, the importance of structured prompting to reduce ambiguity, and why output moderation is essential to catch policy violations that slip through earlier defenses.Amazon.com: LLM Primer VII AI Security: Design Safe and Robust AI System eBook : SHIMODA, SHO: Kindle Store

This episode explores Chapter 4, detailing how attackers manipulate model behavior through crafted inputs like instruction overrides. We discuss why prompt injection is an inherent property of instruction-following systems rather than a standard bug. The episode covers jailbreaking techniques like role-playing and obfuscation, and why defense requires architectural layers rather than just better prompts.Amazon.com: LLM Primer VII AI Security: Design Safe and Robust AI System eBook : SHIMODA, SHO: Kindle Store

This episode breaks down Chapter 3, tracking data risks from training to deployment. We discuss how models can memorize sensitive training data, the subtle dangers of leakage through generated outputs, and the critical importance of treating user prompts and logs as sensitive assets.Amazon.com: LLM Primer VII AI Security: Design Safe and Robust AI System eBook : SHIMODA, SHO: Kindle Store

This episode covers the systematic approach of Chapter 2, moving beyond vague security worries to concrete risk analysis. We discuss how to identify unique AI assets—like prompts, logs, and retrieval indexes—and map the expanded attack surface of API-based systems to build durable defenses.Amazon.com: LLM Primer VII AI Security: Design Safe and Robust AI System eBook : SHIMODA, SHO: Kindle Store