254 Attack Paths in 17 Hours: Thomas Allen on Autonomous Pen Testing, AI Agents, and Whether SOC 2 Still Means Anything
Thomas Allen is the Chief Information Security Officer at Foresite Cybersecurity, where he has spent about 15 years. He came up as a sysadmin and systems architect, then built himself into a pen tester, an incident responder, and a GRC lead before taking the CISO seat. He describes his career as getting in a boat on a river and lifting the paddles out of the water.
Joe and Thomas get into what AI agents should and should not be allowed to do, why a year of clean penetration tests missed what an autonomous test found in 17 hours, whether anyone reads the SOC 2 reports they collect, and what the CMMC pause actually changed.
Thomas draws a hard line on AI: it should never act on behalf of a human without a human seeing it. He will let an agent read email. He will not let one answer it. His test for AI security is simpler than most frameworks make it sound, which is that AI security is data security, and if the model cannot reach sensitive data it cannot leak it.
On the pen testing story, the detail that makes it work is that the autonomous tests had been running monthly for a year or two alongside the annual manual test. On one particular run, a scheduled task happened to fire during the window and exposed credentials. That task was running under a domain admin account. The odds of a human tester doing recon at that exact moment were close to zero.
Thomas is also direct about where compliance artifacts have lost their meaning. He used to joke about dropping a 1939 Mickey Mouse into the middle of his company's SOC 2 report just to see if anyone would notice. He talks through automated GRC services that promise reports, CPA firms that rubber stamp them, and the company whose exposed Dropbox revealed six or seven clients holding identical reports.
And he and Joe disagree productively on cyber insurance panels, forensics firms, and whose interests those teams are serving when the deductible gets waived.
In this episode:
00:01 What people are using AI agents for that they shouldn't
04:50 The AI that got out of the lab, and the free red team argument
08:10 Prompt engineering, and why you have to tell it everything
10:40 Keeping agentic AI in bounds: AI security is data security
13:30 Why a single control never saves you
15:00 The phishing test he guarantees he wins every time
16:25 Trust boundaries, and what happens if the AI outgrows them
19:10 Go slow to go fast
19:45 From paramedic to sysadmin to CISO
24:00 254 attack paths in 17 hours, and what really happened
26:30 Is autonomous hacking already happening
28:00 Cyber insurance, panel forensics teams, and conflicts of interest
31:45 Can you still trust a SOC 2 report
38:00 The CMMC pause, 320 control objectives, and the CUI problem
49:30 How a CISO asks the board for money
52:30 "Are we safe?" and how he answers it
54:45 Star Trek or Skynet
57:50 Who Foresite works with, and where to find Thomas
Foresite Cybersecurity was named the 2026 Google Cloud Security Partner of the Year for North America. Find them at foresite.com, or connect with Thomas on LinkedIn.
For the CMMC thread, Thomas recommends following Jacob Horn on LinkedIn.
Ransomware Rewind is hosted by Joe Erle, Cyber Group Practice Leader at C3 Risk and Insurance Services. If you got something out of this one, subscribe, share it with someone who needs it, and leave a comment. We answer all of them.
More episodes from "Ransomware Rewind"
Don't miss an episode of “Ransomware Rewind” and subscribe to it in the GetPodcast app.
