AI Testing Costs, How to Prevent Runaway Token Bills with Arthur Hicken

What happens when AI agents can not only write mobile app code, but also validate their own work automatically? In this episode, I sit down with Maestro Co-founder and CEO Leland Takamine to explore one of the biggest shifts happening in software testing right now: agentic mobile testing. Leland shares how his team went from solving mobile performance testing challenges to building one of the fastest-growing mobile automation frameworks used by companies like Microsoft, Meta, Amazon, and DoorDash. We dive deep into: How AI coding agents are changing mobile testing workflows What "closing the agentic feedback loop" actually means Why deterministic testing still matters in the age of AI How Maestro MCP lets AI agents validate mobile app changes automatically Why mobile test maintenance may finally become manageable The future role of testers as AI-generated code explodes Leland also gives a live demo showing an AI agent building, validating, debugging, and generating a reusable mobile test completely autonomously. If you care about AI testing, mobile automation, MCP servers, or the future of QA engineering, this episode will likely change how you think about testing workflows over the next few years. Try it out now for yourself:  Maestro Studio: https://testgld.link/mstudio Maestro MCP docs: https://testgld.link/maestromcp

AI-powered testing tools are exploding across software engineering teams… but so are the hidden costs. In this episode, Joe sits down with Arthur Hicken to unpack the growing problem of runaway AI token usage, unexpected LLM billing, and the operational risks of deploying AI agents into testing and DevOps pipelines. Inspired by Arthur's article on the emerging "Token Tax," this conversation explores why many teams are underestimating the true cost of AI automation. You'll learn: Why AI-generated testing can create unexpected scaling costs How runaway AI agents and infinite loops happen Real-world examples of massive AI billing surprises Why deterministic problems shouldn't always use LLMs The hidden risks of "vibe testing" and autonomous AI remediation How QA teams can monitor, test, and control token usage Why performance testing and service virtualization matter more than ever in AI systems Practical strategies to avoid expensive AI deployment mistakes Whether you're a software tester, automation engineer, QA leader, or DevOps practitioner, this episode will help you think more strategically about AI testing before costs spiral out of control.

AI-powered testing tools promise faster automation and less maintenance, but most require teams to abandon their existing frameworks. In this episode, we explore Alumnium, an opensource AI-native end-to-end testing solution created by Alex Rodionov, an engineer at Airbnb and a tech lead on the Selenium project. Instead of replacing tools like Playwright or Selenium, Alumnium adds an AI layer on top, helping teams: Reduce test maintenance by removing brittle locators Build more resilient, self-healing tests Write less code while increasing coverage Run tests across web and mobile using intent-based steps We also go beyond the hype and break down what actually matters for real teams: Why AI-driven tests can still become flaky The performance and cost tradeoffs of LLM-based execution What "context rot" is—and how it impacts long test runs How to safely introduce AI into existing test suites without rewriting everything Check it out now: https://testguild.me/alumAI

AI is changing how we build and test software, but most teams are still struggling to turn AI-generated tests into real production value. Use code TESTGUILD3 try for yourself free now for 3 months: https://links.testguild.com/Kobiton In this episode, we break down what actually works when it comes to AI-powered mobile test automation, especially when running tests on real devices not simulators from Claude. You'll learn: How teams are generating and running Appium tests using natural language in minutes Why AI-generated tests often fail—and how to avoid costly false positives The real impact of AI on test automation roles and responsibilities How real device testing exposes issues AI alone can't catch Practical ways to reduce test maintenance while increasing coverage We also explore a major shift happening right now: AI is making it easier to create tests—but dramatically increasing the volume of code and risk that needs to be validated. That means one thing: Testing isn't going away—it's becoming more critical than ever. If you're a QA engineer, automation engineer, or DevOps leader trying to keep up with AI-driven development, this episode will give you a clear, practical perspective on what to focus on next.

AI coding tools are helping teams move faster than ever, but there's a hidden cost. In this episode, we break down new insights from a DevOps industry report revealing a growing "velocity paradox": teams are shipping more code, but experiencing more failures, rollbacks, and burnout. You'll discover why AI adoption is heavily skewed toward coding, but not testing, pipelines, or observability, and how that imbalance is creating fragile systems that break under pressure. More importantly, you'll learn what high-performing teams are doing differently to maintain quality while scaling speed. What You'll Discover: ✔️ Why AI is increasing deployment failures (and how to stop it) ✔️ The "velocity vs quality" trap hurting modern DevOps teams ✔️ How to reduce flaky tests and pipeline instability ✔️ Why observability and feature flags are now critical, not optional ✔️ Practical ways to improve your CI/CD pipeline for AI-driven development ✔️ The role of QA engineers in the age of AI (and why it's growing, not shrinking) If you're a tester, automation engineer, or DevOps leader trying to keep up  

AI is changing how we build and test software, but most teams are struggling to turn that promise into real results. In this episode, we break down what it actually takes to scale quality engineering across global teams without creating bottlenecks, burnout, or broken processes. You'll learn: why most test automation and transformation initiatives fail how to separate AI hype from reality what high-performing teams are doing differently to ship faster with confidence Today's expert, Sunita McCoy, a Global Engineering Leader and Transformation Specialist, shares practical insights from leading large-scale engineering transformations, including: how to build a culture that supports AI adoption why "quality as a phase" is dead how to shift toward treating quality as a product If you're a QA leader, automation engineer, or DevOps professional trying to improve reliability, reduce risk, and future-proof your skills in the age of AI, this episode gives you a clear path forward.

Mobile test automation is still one of the biggest bottlenecks in modern software delivery. In this interview, QApilot's Co-founder Aditya Challa explains why most AI testing approaches fail and how to fix them. Learn more about QApilot: https://links.testguild.com/flutterqa If your mobile tests are flaky, slow, or hard to trust, you're not alone. Most teams are trying to apply LLM-based AI to problems that actually require deterministic reliability—and that's where things break down. In this video, you'll learn: Why mobile test automation breaks at scale The real issue with "99% accurate" AI in testing LLMs vs deterministic AI (and why it matters for mobile apps) How flaky tests destroy confidence in your pipeline How QApilot approaches mobile testing differently What reliable, scalable mobile automation should look like What this means for you: Fewer false positives, faster releases, and mobile tests you can actually trust. 00:00 Why Mobile Test Automation Is Still Broken 01:10 QApilot Overview 01:51 Why Mobile Testing Tools Fail 03:13 Why Appium Isn't Enough 05:09 QApilot's Approach to Mobile Testing 07:10 Scaling Mobile Testing Across Devices 08:02 Autonomous Testing + Human in the Loop 10:55 How QApilot Works (Architecture + Agents) 13:45 Real Example: Mobile App Crawling in Action 16:31 Finding Bugs Automatically (Performance + Accessibility) 18:52 Device Farms & Real Device Testing 21:50 Future of Mobile Testing (SRE + AI + Quality Layer) 27:06 Real Customer Results & Case Study 31:02 Why QApilot Focuses Only on Mobile 34:04 Where QApilot Fits in CI/CD 36:00 How to Try QApilot + Final Advice

Are you the only tester on your team—and expected to ensure quality across everything? In this episode, we break down the growing challenge of solo QA testing in the age of AI-driven development—where code is generated faster than ever, but confidence hasn't caught up. Christine Pinto shares real-world insights from her experience as a solo tester and now as a founder building tools designed to help testers reduce risk, collaborate better, and make smarter release decisions. You'll learn: Why "all tests passing" doesn't mean your product is safe The hidden risks of AI-generated code and test automation How to shift from test coverage to risk-based testing Practical ways solo testers can avoid burnout and isolation How to bring collaboration back into QA—even if you're the only tester Why better requirements still matter more than better AI

What if your production logs could automatically generate new test cases? In this episode, Joe Colantonio sits down with Tanvi Mittal to break down how AI-powered log mining is changing the way teams approach software testing, quality engineering, and DevOps. Most teams ignore production logs or use them only for debugging. But those logs contain real user behavior, real failures, and real edge cases—the exact scenarios your test suite is probably missing. 👉 Learn how to: Convert production logs into automated regression tests Use AI to detect real-world failure patterns Apply shift-right testing to catch bugs earlier (and smarter) Handle the challenge of testing non-deterministic AI systems Reduce flaky tests and automation debt with real data If you're working with Playwright, Selenium, Cypress, or AI-driven testing tools, this episode will give you a completely new way to think about test coverage.

How do you ensure software quality when the system you're testing doesn't give the same output twice? Go to https://links.testguild.com/inflectra and start your free 30-day trial, no credit card, no contract required. That's the core challenge facing every QA team building or testing AI-powered applications today and it's breaking all the rules we've relied on for decades. In this episode of the TestGuild Automation Podcast, I sit down with Adam Sandman, co-founder of Inflectra, to get into what non-deterministic AI testing actually means in practice, why traditional pass/fail testing no longer cuts it, and what quality professionals need to do differently right now. We cover: Why AI-generated code is raising the stakes for QA teams while budgets stay flat The fundamental difference between deterministic and non-deterministic systems — and why it changes everything about how you test How to set acceptable risk thresholds for AI systems (hint: it depends on whether you're building an e-commerce chatbot or an air traffic control system) Why testers who embrace AI as a tool — not a threat — will be the ones leading their organizations forward How a live demo failure at a conference inspired Inflectra's new non-deterministic testing tool, SureWire If you're a tester, QA manager, or automation engineer trying to figure out how to keep up with AI-driven development without losing your mind — or your job — this one's for you.