SE Radio 633: Itamar Friedman on Automated Testing with Generative AI

Catherine Nelson, author of the new O’Reilly book, Software Engineering for Data Scientists, discusses the collaboration between data scientists and software engineers -- an increasingly common pairing on machine learning and AI projects. Host Philip Winston speaks with Nelson about the role of a data scientist, the difference between running experiments in notebooks and building an automated pipeline for production, machine learning vs. AI, the typical pipeline steps for machine learning, and the role of software engineering in data science. Brought to you by IEEE Computer Society and IEEE Software magazine.

Jonathan Horvath of Z-bit discusses physical access control systems (PACS) with host Jeremy Jung. They start with an overview of PACS components and discuss the proprietary nature of the industry, the slow pace of migration to open standards, and why Windows is commonly used. Jonathan describes the security implications of moving from isolated networks to the cloud, as well as credential vulnerabilities, encryption using symmetric keys versus asymmetric keys, and the risks related to cloning credentials. They also consider several standards, including moving from Wiegand to the Open Supervised Device Protocol (OSDP), as well as the Public Key Open Credential (PKOC) standard, and the open source OSDP implementation that Jonathan authored. Brought to you by IEEE Computer Society and IEEE Software magazine.

Cody Ebberson, CTO of Medplum, joins host Sam Taggart to discuss the constraints that working in regulated industries add to the software development process. They explore some general aspects of developing for regulated industries, such as healthcare and finance, as well as a range of specific considerations that can add complexity and effort. Cody describes how translating regulatory requirements into test specifications and automating those tests can help streamline software development in these regulated environments.  Brought to you by IEEE Computer Society and IEEE Software magazine.

Nick Tune and Jean-Georges Perrin join host Giovanni Asproni to talk about their proposed approach to modernizing legacy systems. The episode starts with some high-level perspective to set context for the approach described in their book, Architecture Modernization (Manning, 2024). From there, the discussion turns to important details, including criteria for deciding which aspects to revisit; some of the activities, processes, and tools; and the importance of data engineering in modernization efforts. Nick and Jean-Georges describe how to successfully implement an architecture-modernization effort, and how to fit that work with the teams' other priorities. The episode finishes with some warnings about the typical risks associated with modernizing a legacy system, and suggestions on how to mitigate them. This episode is sponsored by QA Wolf.

Steve Smith, founder and principal architect at Nimble Pros, joins host Jeff Doolittle for a conversation about software quality. The episode begins with a discussion of why software quality matters for businesses, customers, and developers. Steve explains some patterns and practices that help teams design for quality. They discuss in detail the practices of testing and quality assurance, and the conversation wraps up with suggestions for fostering a culture of quality in teams and organizations. Brought to you by IEEE Computer Society and IEEE Software magazine.

Sriram Panyam, CTO at DagKnows, discusses SaaS Control Planes with SE Radio host Brijesh Ammanath. The discussion starts off with the basics, examining what control planes are and why they're important. Sriram then discusses reasons for building a control plane and the challenges in designing one. They explore design and architectural considerations when building a SaaS control plane, as well as the key differences between a control plane and a data plane. This episode is sponsored by QA Wolf.

Stevie Caldwell, Senior Engineering Technical Lead at Fairwinds, joins host Priyanka Raghavan to discuss zero-trust network reference architecture. The episode begins with high-level definitions of zero-trust architecture, zero-trust reference architecture, and the pillars of Zero Trust. Stevie describes four open-source implementations of the Zero Trust Reference Architecture: Emissary Ingress, Cert Manager, LinkerD, and the Policy Engine Polaris. Each component is explored to help clarify their roles in the Zero Trust journey. The episode concludes with a look at the future direction of Zero Trust Network Architecture. This episode is sponsored by QA Wolf.

Jim Bugwadia, CEO of Nirmata and a committer to the Kyverno projects, joins host Robert Blumen for a discussion of policy-as-code and the open source Kyverno project. The discussion covers the nature of policies; policies and security; policies and compliance to standards; security scans that generate reports compared to tools that allow or deny operations at run time; Kyverno as a kubernetes service; the Kyverno helm charts; the components of Kyverno; bootstrapping a kubernetes cluster with Kyverno; installing policies; implementing policies; customizing policies; packaging and installing policies; kubernetes dynamic admission controllers; the Kyverno admission controller; securing Kyverno itself; observability of Kyverno; types of reports and messages available to cluster users. This episode is sponsored by QA Wolf.

Itamar Friedman, the CEO and co-founder of CodiumAI, speaks with host Gregory M. Kapfhammer about how to use generative AI techniques to support automated software testing. Their discussion centers around the design and use of Cover-Agent, an open-source implementation of the automated test augmentation tool described in the Foundations of Software Engineering (FSE) paper entitled “Automated Unit Test Improvement using Large Language Models at Meta“ by Alshahwan et al. The episode explores how large-language models (LLMs) can aid testers by automatically generating test cases that increase the code coverage of an existing testing suite. They also investigate other automated testing topics, including how Cover-Agent compares to different LLM-based tools and the strengths and weaknesses of using LLM-based approaches in software testing.

Goran Petrovic, a Staff Software Engineer at Google, speaks with host Gregory M. Kapfhammer about how to perform mutation testing on large software systems. They explore the design and implementation of the mutation testing infrastructure at Google, discussing the strategies for ensuring that it enhances both developer productivity and software quality. They also investigate the findings from experiments that quantify how mutation testing enables software engineers at Google to write better tests that can detect defects and increase confidence in software correctness. Brought to you by IEEE Computer Society and IEEE Software magazine.