AI Pentesting

Welcome everyone to this tenth episode of Mind the Machine, a podcast about AI security and safety. I’m Florencio Cano. Today we are going to talk about the security risks and security controls of LLM code generators.

Today I’ll talk about a technical topic related to the composition of LLMs. Are LLMs only data (weights) or do they contain code? If they contain code, can this code contain malware? And one additional question, if they have code, can they have vulnerabilities like heap overflows? In this episode I analyze what we exactly download when we download a model with Ollama or with the Hugging Face API.Referenceshttps://ollama.com/https://huggingface.cohttps://docs.vllm.ai/en/latest/https://github.com/kserve/kservehttps://huggingface.co/docs/transformers/en/indexhttps://ollama.com/libraryhttps://huggingface.co/ibm-granite/granite-3.1-2b-instructhttps://huggingface.co/microsoft/Phi-3.5-mini-instruct/tree/mainhttps://www.usenix.org/conference/usenixsecurity23/presentation/christouhttps://arxiv.org/abs/2307.05642

Welcome everyone to this eight episode of Mind the Machine, a podcast about AI security and safety. I’m Florencio Cano. Today I’ll talk about my attendance to RootedCon 2025. RootedCon is the biggest cybersecurity congress in Spain and one of the biggest in Europe. This year it had a specific AI security track organized by Fernando Rubio and I had the pleasure to attend and be a speaker. Let’s talk about the presentations I was able to enjoy there.Referenceshttps://rootedcon.com/https://smolagents.org/https://www.crewai.com/https://www.twitch.tv/claudeplayspokemon

In this episode we talk about the different ways companies are using AI, and specially LLMs, to improve their cybersecurity processes. We will talk about information gathering, protection, detection and response and what are known applications of AI in each of these areas.During this episode I mention multiple references that I'm sharing here:IntelEX: A LLM-driven Attack-level Threat Intelligence Extraction Framework https://arxiv.org/abs/2412.10872 Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection https://arxiv.org/abs/2407.16235 Leveling Up Fuzzing: Finding more vulnerabilities with AI https://security.googleblog.com/2024/11/leveling-up-fuzzing-finding-more.html RedFlag https://github.com/Addepar/RedFlagLLMSecConfig: An LLM-Based Approach for Fixing Software Container Misconfigurations https://arxiv.org/abs/2502.02009AI and LLM Models to Analyze and Identify Сybersecurity Incidents https://ceur-ws.org/Vol-3746/Short_6.pdfGenDFIR: Advancing Cyber Incident Timeline Analysis Through Retrieval Augmented Generation and Large Language Models https://arxiv.org/abs/2409.02572

In this episode we talk about how cybercriminals are using AI to improve their operations. For example, for creating phising emails, fake voice and fake video. Also to create disinformation and fake news. We also discuss what we can do as a society to reduce these risks and what can we do in our organizations to protect ourselves against these threats.

In this episode of Mind the Machine, host Florencio Cano talks about the concept of agentic AI, exploring what makes AI systems capable of autonomously performing tasks and the unique security challenges they present. While agentic AI can revolutionize industries, robust security measures are essential to manage the security risks. Two of the risks mentioned in the podcast are the risk of AI agents that interact with the operating systems and those that generate code. References mentioned in this episode: Security Runners article about RCE on Anthropic's Computer Use: https://www.securityrunners.io/post/beyond-rce-autonomous-code-execution-in-agentic-ai Anthropic's Computer Use: https://docs.anthropic.com/en/docs/build-with-claude/computer-use Sandboxing Agentic AI Workflows with WebAssembly: https://developer.nvidia.com/blog/sandboxing-agentic-ai-workflows-with-webassembly Episode about Prompt Injection https://open.spotify.com/episode/0ZH9Q2PQXojnpb8UI2jhuS?si=bfx-QIlnT8eDUrl2a_zM-w

In this episode we talk about AI Pentesting. We talk about the difference with traditional cybersecurity pentesting. We also talk about benefits and drawbacks of manual and AI automatic pentesting. In the case of AI automatic pentesting, we mention some open source tools to perform it. These are some URLs related to topic mentioned in the episode: Breaking Down Adversarial Machine Learning Attacks Through Red Team Challenges https://boschko.ca/adversarial-ml/ Dreadnode’s Crucible CTF platform https://crucible.dreadnode.io/ PyRIT https://github.com/Azure/PyRIT Garak https://github.com/NVIDIA/garak Project Moonshot https://github.com/aiverify-foundation/moonshot

In this episode, we talk about ten very important security architecture patterns to protect LLM applications. Open source guardrails software mentioned during the episode: TrustyAI Llama Guard Nemo Guardrails Open source model evaluation frameworks mentioned: lm-evaluation-harness Project Moonshot Giskard

In today's podcast, we will talk about what is prompt injection. We will talk about techniques to exploit it and security controls to reduce the risk of it happening.

In this first episode of Mind the Machine I introduce the podcast and myself, Florencio Cano. The podcast will be about AI security and safety. We will talk about security for AI and also about AI for security. I hope you enjoy it! Please, don't hesitate on contacting me directly by sending me an email to [email protected] or by contacting me at LinkedIn or Mastodon.