How good is AI at Hacking? Benchmarking Models for Vulnerability Detection with Philippe Dourassov

Are frontier AI models really the best at hacking and finding security vulnerabilities? In this episode of Bad Dependencies, host Mackenzie Jackson speaks with Philippe Dourassov, head of technical AI pentesting at Aikido, to break down real-world AI benchmarking results.Philippe explains how they evaluate models by testing whether agents can rediscover recently disclosed CVEs in open-source software without prior training data or cheating. You will learn why an agent harness is critical for exploring codebases and orchestrating runs, and why massive models like Mythos or GPT-6 Astra, despite high scores, might not be the most practical choice.Discover how running cheaper open-weight models like DeepSeek V4 Pro across multiple non-deterministic attempts beats expensive frontier models at a fraction of the cost, completely changing the economics of offensive security.

In this episode of Bad Dependencies, host Mackenzie Jackson sits down with security researcher Charlie Eriksen to dissect a massive software supply chain attack hitting the Mastra AI ecosystem. Breaking down how an attacker compromised a maintainer's account to inject a malicious transitive dependency (easy-day-js) across over 140 packages, they explore the sophisticated social engineering tactics behind the breach. The duo also discusses critical defensive strategies—from package manager cooldown periods to upcoming NPM security changes—and warns developers about why build pipelines have become the latest critical attack surface. Chapters00:00 – Introduction00:28 – The Mastra AI Ecosystem Attack Explained02:18 – The Payload: Remote Access Trojans (RATs) & Crypto Stealers03:26 – Phishing the Maintainer: The "Microphone Trick" & North Korea05:45 – Reach of the Attack & Incident Response Playbook08:47 – Preventative Measures: Cooldown Windows & Closing the OIDC Door13:08 – NPM Version 12 and the End of Post-Install Scripts16:05 – The Next Attack Surface: GitHub Actions & Governance20:06 – Outro (And One Last Bad Vibe)

In this episode of Bad Dependencies, host Mackenzie Jackson sits down with security researcher Joe Leon to dissect a major shift in Google API key sensitivity. For years treated as benign public identifiers, these same keys became high-risk vectors following the integration of Google Gemini, allowing threat actors to rack up enormous cloud bills and access cached files. Joe reveals his startling discovery that deleting a compromised GCP API key didn't instantly revoke it, allowing it to authenticate requests for up to 23 minutes, a flaw Google initially dismissed as "expected behavior" before later prioritizing it as a critical bug.Report "Google API keys keep working after you delete them" https://www.aikido.dev/blog/google-api-keys-deletionReport: Google API Keys Weren't Secrets. But then Gemini Changed the Rules https://trufflesecurity.com/blog/google-api-keys-werent-secrets-but-then-gemini-changed-the-rules

In this episode of Bad Dependencies, we analyze the reported leak of GitHub's source code and the sale of thousands of its repositories. We map out the chain of events leading up to the incident, including recent compromises of a Visual Studio Code extension and a PyPI package. The discussion covers the tactics of the threat actor group Team PCP, the practical limitations of rapid credential rotation at scale, and why implementing a cooldown period for dependency updates can help safeguard your development pipeline. -- This episode is sponsored by Aikido Security: Protect your developer environments from supply chain attacks with Aikido Device Security. Learn more at aikido.dev/protect/device-protection.

In this episode of Bad Dependencies, we dive into the "wormy" chaos of the latest supply chain attack hitting the JavaScript ecosystem. Join researcher Charlie Eriksen as he breaks down how the threat actor group TeamPCP compromised the widely-used TanStack ecosystem and successfully pivoted into Mistral AI. We explore the technical "perfection" of this attack: a lethal combination of pull_request_target misconfigurations, GitHub Actions cache poisoning, and OIDC signature abuse. Charlie also sheds light on a terrifying new trend, the attackers have open-sourced their worm, complete with a "dead man's switch" designed to wipe infected machines if credentials are revoked.

In this episode of Bad Dependencies, Mackenzie and security researcher Charlie Erickson break down a fast-moving software supply chain attack led by Team PCP.Starting with the compromise of Trivy, the attackers leveraged stolen credentials to spread into ecosystems like NPM and LiteLLM, impacting widely used developer tools and AI infrastructure. The conversation explores how the attack evolved, including worm-like behavior, credential harvesting, and ransomware tactics.Charlie shares real-time insights into the attackers’ methods, motivations, and the ongoing nature of the incident, along with practical advice on mitigation such as credential rotation, dependency pinning, and securing CI/CD pipelines.

In this episode, I sit down with Charlie Eriksen, the researcher who uncovered the Shai Hulud 2.0 campaign, for a deep dive into one of the wildest supply-chain attacks we’ve seen. What began as a strange detection quickly unraveled into a worm that spread across npm, GitHub, and even a compromised Open VSX extension.“Patient Zero” was AsyncAPI, where the attackers exploited a subtle GitHub Actions flaw that let them run malicious code inside the org’s own CI pipelines without their pull request ever being merged. Unmerged PR → full RCE → stolen org-level credentials.From there, the worm propagated through packages, harvested secrets with TruffleHog, dumped them into tens of thousands of GitHub repos, and, most shockingly, contained a wiper mode that deleted a victim’s entire home directory if it couldn’t create new repos.It’s a fascinating and slightly terrifying look at how modern supply-chain attacks actually work under the hood. Give it a listen.

In this episode of Bad Dependencies, Mackenzie Jackson and Charlie Eriksen dive into one of the most sophisticated malware incidents to target developers — the OpenVSX compromise. They unpack how attackers hid malicious code using Unicode obfuscation, discuss the shift from npm to VS Code extension attacks, and explore how the open-source ecosystem is responding. The episode also covers npm’s new token policies, trusted publishing, and what these changes mean for the future of supply chain security.Chapters:00:00 – Introduction & Discovery02:00 – What is OpenVSX and How It Works03:40 – Anatomy of the Malware Attack05:00 – Unicode Obfuscation and Detection08:20 – Attackers Move from npm to VS Code11:00 – npm’s Security Policy Overhaul17:40 – Trusted Publishing and the Future of Supply Chain Security

In this episode, host Mackenzie Jackson is joined by Charlie Erikson and Daniel Pereira to uncover the story of Shai-Hulud — a self-propagating worm that shook the NPM ecosystem. Like the great sandworm of Arrakis, it surfaced suddenly, exfiltrating secrets and spreading through unsuspecting packages.Daniel recounts his discovery and the frustrating desert-like silence from major platforms as he tried to raise the alarm. Charlie dives into the worm’s anatomy, from environment variable theft to GitHub action exploits, showing how attackers evolved their tactics from the earlier NX breach.Together, they reflect on what it takes to fight worms in the shifting sands of open source, and why the community needs faster ways to respond before the next Shai-Hulud emerges.

Charlie Eriksen and I sat down for a candid chat with Josh Junon, the maintainer of chalk and debug, who found himself at the center of one of the largest npm supply-chain attacks.Josh talks openly about: ✅ How the phishing attack actually worked ✅ What it felt like to have his packages hijacked ✅ The lessons for every open source maintainer and company that relies on npmIt’s a rare, first-hand account of what it’s like to be the person behind the breach, raw, honest, and essential listening for anyone in tech.We are releasing it live at 16:00 CEST , 07:00 Pacific Time