Dark Side of Testing: The Failures We Cause Ourselves - Bart Knaack

Contract-driven testing lets a tester work against a realistic mock instead of waiting for 50 integrated systems to behave, and that is what Phil Royston is bringing into a large bank. We talk about why mocks built on a developer's imagination produce unreliable tests, how agreed contract tests turn into mocks for API consumer and provider, and why the pitch to senior management fits on one slide while the middle managers hold the veto. Phil also explains why approved business requirements were not testable ones, what he means by parasitic end-to-end testing, and why his advice to any test manager is to wait for the crisis.

Building trust in teams starts with far smaller things than a team-building event, and that is what I talk through with Michael Kutz and Linda Van De Vooren. Michael brings Stephen Covey's waves of trust, from self-trust outward, and his rule that a kept promise about one issue in the code base is one of the Lego stones trust is built from. Linda talks about the inner critic she has practiced with for over ten years, about limitless honesty with a new client, and about why rest and fun come first. We also get into a 50,000 credit limit that became 500,000 through one extra zero, who owns the fix after that, and what makes an app trustworthy when it tells you what went wrong.

Mental skills keep testers thinking when AI offers to do it for them, and Ard Kramer puts it bluntly: you can't outsource your brain. With him I talk about why an idle brain loses its connections, how his cue HAH (human, AI, human) keeps the thinking on our side, and which four skills get a tester into the flow: goal setting, visualization, self-talk and stress control. Ard turns the test goal around, because one passing test confirms nothing, so his hypothesis aims at the moment the payment interface fails. He also explains the Zeigarnik effect, why an open task keeps your subconscious busy overnight, and why AI works best as a critical partner that asks you the questions.

Testing AI models means testing a magic box: nobody has the source code, and the behavior is neither controlled nor understood. Michael Bolton calls that strongly magical, and with him I talk about what follows from it. His principles are simple: if something matters very much, don't depend on magic, and if you use it anyway, responsibility demands that you test it and maximize testability. Simple output checks don't stand a chance against a language model, so testing becomes experiencing, exploring and experimenting, like peeling an onion. And Penn and Teller's clear plastic cups show why we fall for these models in the first place: the insight we see in their text comes from us.

Testing is supposed to prevent failures in production, yet sometimes the test itself is what breaks things. Bart Knaack calls this the dark side of testing, and we talk about test data leaking out of poorly protected environments, test systems that give attackers a look at your code, and why testing in production needs a risk assessment of its own. Bart would rather ask questions than hand out checklists, because a ticked box stops people thinking. He also tells the story of a team with 10,000 end-to-end test cases that ended up needing only 50.

Quality hunting grew out of Rik Marselis's frustration with bug hunting, and I wanted to know how it works when the goal is information about quality. Rik walks me through a session: teams of about four people, three hours of exploratory testing and a goal agreed up front by asking stakeholders what would help them sleep well at night. He explains why the report structure comes first, why the prize should stay small and why the point just before several sprints go to production suits a hunt well.

What does it take to turn a risky one-time event into a conference that testers travel across borders for? With Okan Çakmak I talk about how the Istanbul Software Testing Conference grew out of something missing: after the pandemic, the big Turkish testing events stayed online, and people lost the chance to meet, network and hug each other in person. He tells me how the team wondered whether 50 or 100 people would come, then welcomed 200, and how the "light in their eyes" of those first participants convinced them to keep going.

What happens when AI ships poor quality faster, and nobody notices because the pipeline stays green? With Michaël Pilaeten I talk about why test automation without a clear strategy is a problem that AI only accelerates, not solves. We get into the question of what separates a tester from a test automation engineer, why those two roles need each other but are not interchangeable, and how chasing 100% coverage on millions of lines of legacy code can be a waste everyone agrees to quietly.

What separates a handful of agents from a real agentic system? With Nishan Portoyan I talk about exactly that line, and why most teams are still on the wrong side of it. We walk through a full testing pipeline where eight to twelve agents handle everything from requirement review to performance test conversion, and I keep coming back to the question of where the human has to step in and why the AI simply cannot replace that judgment. Nishan is direct about what these systems cannot do: they do not think, they gather information and hand it back, and anyone expecting otherwise will run into trouble fast.

What happens to quality when anyone can generate code with a prompt but nobody checks whether it actually works? With Olivier Denoo I talk about why that question puts expert testers in a stronger position than many managers currently assume. We get into the real risk of AI that always sounds right, the edge cases a lost suitcase in Brussels reveals, and why the hardest unsolved problem in testing is still the same one Olivier ran into three decades ago: describing precisely what you want. The skills I keep coming back to in this conversation are business understanding, communication, and the ability to spot what the AI confidently got wrong.