From Tools to Teammates: (Dis)Trust in AI for Cybersecurity with Neele Roch

When we collaborate with people, we build trust over time. In many ways, this relationship building is similar to how we work with tools that leverage AI. As usable security and privacy researcher Neele Roch found, “on the one hand, when you ask the [security] experts directly, they are very rational and they explain that AI is a tool. AI is based on algorithms and it's mathematical. And while that is true, when you ask them about how they're building trust or how they're granting autonomy and how that changes over time, they have this really strong anthropomorphization of AI. They describe the trust building relationship as if it were, for example, a new employee.” Neele is a doctoral student at the Professorship for Security, Privacy and Society at ETH Zurich. Neele (and co-authors Hannah Sievers, Lorin Schöni, and Verena Zimmermann) recently published a paper, “Navigating Autonomy: Unveiling Security Experts’ Perspective on Augmented Intelligence and Cybersecurity,” presented at the 2024 Symposium on Usable Privacy and Security. In this episode, we talk to Neele about:How security experts’ risk–benefit assessments drive the level of AI autonomy they’re comfortable with.How experts initially view AI: the tension between AI-as-tool vs. AI-as-“teammate.”The importance of recalibrating trust after AI errors—and how good system design can help users recover from errors without losing their trust in it.Ensuring AI-driven cybersecurity tools provide just the right amount of transparency and control.Why enabling security practitioners to identify, correct, and learn from AI errors is critical for sustained engagement.Roch, Neele, Hannah Sievers, Lorin Schöni, and Verena Zimmermann. "Navigating Autonomy: Unveiling Security Experts' Perspectives on Augmented Intelligence in Cybersecurity." In Twentieth Symposium on Usable Privacy and Security (SOUPS 2024), pp. 41-60. 2024.

In this episode, Heidi gets a taste of her own medicine and is interviewed by co-host John Robertson about her newly-released book Human-Centered Security: How to Design Systems That Are Both Safe and Usable. We talk about:Why Heidi’s experience as a UX researcher prompted her to write Human-Centered Security.Places in the user journey where security impacts users the most.Why cross-disciplinary collaboration is important—find your security UX allies (people in security, legal, privacy, engineering, product managers, to name a few).Practical security UX tips like secure by default, guiding the user along the safe path, and being really careful about the words you use.Technical users—IT admins, engineers, security analysts—are users, too and why it’s so important to thoughtfully design the security user experience for them. (Spoiler: they help keep the rest of us safe!)

The cybersecurity industry often fixates on “behavior change,” expecting users to take on unrealistic tasks instead of designing safer, smarter systems. Matt Wallaert (founder of BeSci.io and author of Start at the End: How to Build Products that Create Change) explains behavioral science isn't about forcing behavior change. Instead, it's about understanding people so a thoughtfully-designed system can influence more secure outcomes.Whether you’re a UX designer, a security engineer, or a CISO, you influence security behaviors. Here’s how you can move towards more secure outcomes:Stay Ahead of Threat Actors: Cybercriminals use behavioral science to their advantage. People designing the security user experience must not only catch up but outpace them.Define Clear Outcomes: Don’t just say “we want users to be secure.” Know exactly what behaviors you want and why. Vague goals lead to vague results.(as Matt explains, saying things like “I want people to be more secure” isn’t helpful. In fact, many people don’t know what “more secure” means in the context of their product or organization).Ask Better Questions: Use tools like the “sufficiency test.” For example, sure, it might be nice if users created complex passwords—but users don’t necessarily have to be the ones doing it. Why can’t the system create a complex password for them (as password managers do)?Understand promoting and inhibiting pressures. These concepts will help you design systems that are more resilient because they are built with people in mind. There are reasons people do and do not do things—when you understand why, you can develop systems that will be more effective in encouraging the behaviors you want. Security practitioners: tired of being perceived as the “department of no”? Matt explains how behavioral science can help you better collaborate with cross-disciplinary teams.Bonus: UX designers, after this episode you may never create another persona.

“Technical people need to better understand the laws and regulations and lawyers need to better understand the technology and processes in place. When that happens, when those worlds come together, that’s where you can meaningfully make things happen.” -Justine Phillips, Partner at Baker McKenzieIn this episode, we talk about:Essential questions product teams should ask legal experts when integrating AI into new products and features.In particular, why it’s important for designers and engineers to question the source of the data they are using for AI-powered products and features.The need to anticipate international security and privacy regulations, which are constantly changing, including emerging regulations that could impact companies developing IoT devices.Justine Phillips is a Partner at Baker McKenzie, where she is co-chair of data+cyber for the Americas. She is the author of Data Privacy Program Guide: How to Build a Privacy Program That Inspires Trust.

What do CISOs have to say about the security tools their teams use?:“When we introduce a level of complexity in the system, it undermines security. Every moment wasted trying to use a tool effectively benefits the adversary.” - Matt StamperIn this episode, we talk to cybsecurity leaders Bill Bonney, Gary Hayslip, and Matt Stamper about:The ever-evolving role of the CISO and what CISOs care about most.What product teams designing security software need to understand:Security tools need to operate across varied ecosystems (which means your product team needs to understand those ecosystems).Complexity is the enemy of security. Yes, UX matters.Context-switching means security teams waste time. Instead, security tools need to present the right information at the right time.Why CISOs are excited to leverage AI in security tools—and what concerns them the most.Bill Bonney, Gary Hayslip, and Matt Stamper are seasoned CISOs and cybersecurity leaders. They are co-founders of the CISO Desk Reference Guide—a series of books including topics such as security policy, third-party risk, privacy, and incident response—which provide actionable insights for security leaders.

In this episode, we talk about: Security tools don’t get a free pass when it comes to involving end users as part of the design process. People studying and building ML-based security tools make a lot of assumptions. Instead of wasting time on assumptions, why not learn from security practitioners directly?Businesses (and academia) are investing a great deal in building ML-based security tools. But are those tools actually useful? Are they introducing problems you didn’t anticipate? And even if they are useful, how do you know security practitioners will adopt them?Why are adversarial machine learning defenses outlined in academic research not being put into practice? Jaron outlines three places where there are significant roadblocks: First, there are barriers to developers being aware of these defenses in the first place. Second, developers need to understand how the threats impact their systems. And third, they need to know how to effectively implement the defenses (and, importantly, be incentivized to do so).Jaron Mink is an Assistant Professor in the School of Computing and Augmented Intelligence at Arizona State University focused on the intersection of usable security, machine learning, and system security. In this episode, we highlight two of Jaron’s papers:“Everybody’s Got ML, Tell Me What Else Do You Have”: Practitioners’ Perception of ML-Based Security Tools and Explanations.”“Security is not my field, I’m a stats guy”: A Qualitative Root Cause Analysis of Barriers to Adversarial Machine Learning Defenses in Industry

In this episode, we talk about:The role misaligned incentives play in security behaviors.How Serge and his team approach security-focused UX research. Looking upstream at the security decisions made by software engineers and, in turn, the situations they are often placed in due to resource constraints and competing priorities at their organizations.Learning from other industries with highly-skilled professionals (shout-out to the humble check list!)Regulations and policy changes will likely place greater liability on the organizations shipping software.Serge Egelman is the Founder and Chief Scientist at AppCensus and Research Director at International Computer Science Institute (ICSI). He’s written countless research papers on usable security and privacy. Most recently, his research centers around improving the user experience for users who are responsible for safeguarding their customer’s data (such as software engineers).

Shante Perrin, a cybersecurity leader, and her team use cybersecurity software to not only to detect and respond to cybersecurity threats but also, as Shante describes, to help paint a picture for their customers:“We like to build a timeline of events to build that picture, create that story so we can deliver it to the customer and explain why we felt it is suspicious. In other words, why are we bothering you about this?”In this episode, we talk about:Building stories from data: analysts must translate technical information into clear, understandable narratives for customers.If people designing cybersecurity software can design better, more effective experiences for analysts, analysts can do a better job of communicating these narratives to their customers.How security analysts at different levels perceive and handle threats differently—and how that changes what they need or expect from cybersecurity software.How thinking like an attacker can help security analysts—but only if the tools they use provide them with the right information at the right time. Shante Perrin is a cybersecurity leader and is currently the director of a managed services team. She led a cybersecurity team for a Fortune 100 company as an MSSP and has been a security analyst and security operations center (SOC) lead.

In this episode, we talk about: The need for human-centered security—in order for security measures to be effective, they must center around people, making usability as crucial as technology. We explore the gap between research and practice, highlighting the need to bring cybersecurity research into real-world application. Human-centered security research can’t possible be effective if no one knows about it or finds it challenging to implement.The importance of collaboration, advocating for more shared spaces where researchers and practitioners can come together to address pressing cybersecurity challenges.Julie Haney is a Computer Scientist and Human-Centered Security Researcher and program lead at NIST (National Institute of Standards and Technology). She was formerly a Computer Scientist at the United States Department of Defense. In the episode we refer to two of Julie’s publications: “From Ivory Tower to Real World: Building Bridges Between Research and Practice in Human-Centered Cybersecurity” and “Towards Bridging the Research-Practice Gap: Understanding Researcher-Practitioner Interactions and Challenges in Human-Centered Cybersecurity.”

Security analysts respond to security detections and alerts. As part of this, they have to sift through a mountain of data and they have to do it fast. Not in hours, not in days. In minutes.Tom Harrison, security operations manager at Secureworks, explains it perfectly, “We have a time crunch and it’s exacerbated by the other big issue security analysts have: we have an absolute ton of data that we have to sift through.”In this episode:Tom explains that security analysts are forced to go back to a pile of data with each subsequent question in their workflow. That’s a huge waste of time. And a terrible user experience. Tom says, “It would lead to better accuracy, faster triage, and a better user experience if you can just take me directly to the answer or at the very least a subsection that has the answer I’m looking for.”What does this mean for you as a UX designer designing security products? You need a deep understanding of security analyst workflows to help them identify and respond to attacks as quickly as possible.That way, you can design security products that support users who are under intense pressure to do things quickly. Tom describes how the UX can “guide or complement the workflow.”Tom talks about what gets him excited about integrating AI into security analyst workflows—and what has him worried, as well.Tom Harrison is a Security Operations Manager at Secureworks. We dubbed Tom an “ideas machine” and a fierce advocate for the security analyst user experience. In fact, Tom is conducting UX research in the field better than most UX researchers. He’s a passionate teacher and shares his knowledge and resources in a free security reference guide.