Decentralizing Data: Navigating the Shift to Data Mesh

SummaryIn this podcast episode, Vaughn and Mark Planagumà discuss various aspects of data strategies and the implementation of Data Mesh. Mark shares his background in data engineering and his experience in building data platforms for different companies. They explore the use of Domain-Driven Design in data strategies and the role of contracts in data architecture. Mark explains the concept of Data Mesh and how it shifts the focus from centralized data warehouses to domain-driven, decentralized data products. They also discuss the implementation of data governance and automation, the influence of operational software architectures on data strategies, and the design of a semantic layer in a Data Mesh. The conversation explores the maturity in operational and analytical architecture, the influence of Domain-Driven Design on Data Mesh, the tooling required for Data Mesh, the future of analytics, challenges with AI and metadata, and where to learn more about Data Mesh.TakeawaysDomain-Driven Design can be applied to data strategies to organize data by domains and enable domain owners to take responsibility for their data.Data Mesh is a paradigm shift that emphasizes decentralized, domain-driven data products instead of centralized data warehouses.Contracts play a crucial role in data architecture by defining the metadata and governance rules for data products.Implementing data governance and automation can help ensure the discoverability, accessibility, and reusability of data in a data mesh.Organizational structure needs to align with the principles of Data Mesh, with domain-driven teams owning and managing their data.A semantic layer in a Data Mesh helps organize and aggregate data products by domains, making it easier to discover and consume data.Operational software architectures can influence data strategies by providing the infrastructure and tooling for data products in a Data Mesh. Data is typically behind operational and application maturity, but Data Mesh is emerging to bridge the gap.Domain-Driven Design plays a significant role in shaping Data Mesh and enabling interoperability between operational and analytical systems.Existing tools like lake houses and data warehousing can be leveraged to support Data Mesh, focusing on creating interoperable data products.The future of analytics lies in improving data quality, metadata management, and leveraging AI to interact with data in a more natural and business-focused way.ChaptersPlease note these are approximate locations! We are trying new tools and hope you find this helpful.00:00 Introduction and Background05:32 Using Domain-Driven Design with Data Strategies09:20 Understanding Data Mesh11:18 The Role of Contracts in Data Architecture28:21 Influencing Organizational Structure for Data Mesh34:00 Semantic Layer Design in Data Mesh37:36 Impact of Operational Software Architectures on Data Strategies37:52 Maturity in Operational and Analytical Architecture42:30 Domain-Driven Design and Data Mesh47:08 Tooling for Data Mesh53:29 The Future of Analytics01:01:08 Challenges with AI and Metadata01:09:36 Learning More about Data MeshMarc Planagumà, is a native of Olot (Catalonia) with degrees in Telecommunications from UPC. He is a prominent figure in data engineering and governance. He serves as the Data Platform & Governance Director at Adevinta Spain, where he has spearheaded the development and implementation of Lakehouse architecture and Data Mesh paradigm, focusing on scalability, autonomy, and effective governance by design. Hosted on Acast. See acast.com/privacy for more information.

In this episode of the Add Dot podcast, Vaughn Vernon and Paul Rayner discuss the evolution of the Domain-Driven Design (DDD) community in North America. The conversation highlights the importance of fostering connections and providing valuable learning experiences.Throughout the conversation, Vaughn and Paul share insights into the complexities of modernization efforts, particularly in large organizations with legacy systems. They stress the importance of strategic thinking, focusing on core domains, and avoiding the "boil the ocean" approach. The episode concludes with a teaser for the upcoming Explore DDD conference in Denver, Colorado, scheduled for March 12-15, 2024, featuring keynotes by Eric Evans and Vaughn Vernon.Paul Rayner is a developer, instructor, coach, consultant, and popular conference speaker with over thirty years of software development experience. Paul provides DDD and EventStorming training and coaching through Virtual Genius.Paul is the founder and chair of the Explore DDD conference, the premier Domain-Driven Design conference in North America, and co-founder of DDD Denver. He is also the author of The EventStorming Handbook, and a co-author of Behavior-Driven Development with Cucumber. He lives in Denver, Colorado. Hosted on Acast. See acast.com/privacy for more information.

In this podcast episode, Vaughn interviews Jacqui Read, a .NET developer turned software architect and author of the book "Communication Patterns: A Guide for Developers and Architects." Jacqui discusses the inspiration behind her book, emphasizing the importance of soft skills in conjunction with technical expertise. She highlights her experience in various domains and how she integrated diverse ideas into her work, leading to the identification of communication patterns and anti-patterns. The conversation delves into the reputation of programmers as poor communicators and the potential for improvement through Jacqui's insights.Jacqui's book covers a broad spectrum of communication, including verbal, written, non-verbal, and visual communication. Jacqui emphasizes the significance of visual communication, which constitutes a substantial portion of the book. She addresses the inclusion of illustrations, particularly discussing considerations for grayscale printing and offering links to color versions on the accompanying website. The podcast touches on sections of the book dedicated to the communication of knowledge, documentation, and the challenges of remote communication in today's distributed teams and companies.Jacqui Read is an internationally recognised solution and enterprise architect, and author of "Communication Patterns: A Guide for Developers and Architects", with hands-on experience and expertise architecting and coding software systems. She specialises in assisting businesses to create and enhance architecture practices, construct evolutionary architectures, and untangle and extract value from data and knowledge.Alongside consulting, Jacqui teaches public and private workshops and speaks at international conferences on topics such as architecture practices, technical communication, and architecture decisions. Her professional interests include collaborative modelling, knowledge management, Domain Driven Design, sociotechnical architecture, and modernising enterprise architecture practices. Outside of work she enjoys gardening and attempting to strum her ukulele and sing at the same time. Her website is https://jacquiread.com. Hosted on Acast. See acast.com/privacy for more information.

Vaughn and Karen discuss the possible benefits for and threats to humanity in the use of AI. They dip into Karen's overlapping work in blockchain and AI and her unique accomplishments and awards in her blockchain and AI work. These topics take both Vaughn and Karen way back to the days of OS/2 and its superiority to Windows, which they both experienced.Karen describes her current work and what's coming next, with key benefits to humanity.They wrap up their conversation sharing information on how other people, including listeners and others, can learn about practical and beneficial work with blockchain and AI.Karen Kilroy is a lifelong technologist with heart. A full-stack software engineer living in Northwest Arkansas, she has written three publications for O'Reilly: Blockchain Tethered AI (2023), AI and the Law (2021), and Blockchain as a Service (2019).Karen is CEO of Kilroy Blockchain, which was the Winner of IBM Watson Build Challenge in 2017, North America, for their AI app RILEY. Karen is also co-founder of Friends of Justin, a non-profit 501(c)(3) AI research lab focused on the interaction between humans and AI. Hosted on Acast. See acast.com/privacy for more information.

Vaughn and Asanka—WSO2's CTO— discuss a relatively radical and fresh approach to cloud applications and services development. The tools include the domain-driven Cell-based Architecture, the Ballerina programming language, and the Choreo cloud platform. This purpose-built trio is composed as one powerful offering to give software engineers the ability to focus their efforts on delivering cloud-native applications and services. Of course, engineers are not required to use the Ballerina programming language. They may use Java and any other choice of language. Yet, those looking for a practical approach to functional programming that offers asynchronicity and handles service integrations well, Ballerina is a language worth trying. Together, this architecture-language-platform trio deserves consideration for use in your future enterprise.Asanka Abeysinghe, WSO2's CTO, is a technology visionary with over 20 years of experience designing and implementing scalable distributed systems, microservices, and business integration solutions. He advances WSO2's corporate reference architecture, collaborates with customers and industry analysts, and drives the company's technology mission. Asanka is also a contributor to the Apache Software Foundation and a sought-after speaker at global events. Hosted on Acast. See acast.com/privacy for more information.

Vaughn Vernon has a conversation with Alfredo Muñoz regarding the transformation and modernization within banks, why it's so urgently needed, and how efforts can be successful. Their discussion encompasses BIAN, Domain-Driven Design, Data Mesh, and why focusing on business capabilities as business, service, events, and data domains is critical. We address why banks are so resistant to change but why they must learn a new mindset in order to both survive and flourish in the age of cloud and digital banks. For example, a bank transaction that has 1,000 COBOL calls will never work in the cloud, nor will a transaction that makes 100 API calls. How can new technologies and approaches to software development help banks, insurance, airlines, telecom, and other industries that heavily rely on mainframe power successfully switch out of the old status quo?Alfredo is an Enterprise Architect, and Banking Transformation Expert with more than 25 years of experience in large scale banking transformations in Europe, Asia and Latam, always involving first tier banksHe currently works in the Global Center of Excellence for Banking and Financial Market at IBMDuring the past years he has developed the methodology for the modernization of corebanking systems, including the adoption cloud infrastructure in coexistence with legacy systems, the BIAN Industry Standards, the industrialization and standardization of the design and development processes and the use of low code development frameworks with the primary objective to deliver transformation at scale, as required for large Financial Institutions.He has authored leading points of view, methods and assets on core systems transformation to Cloud. https://www.fintechna.com/authors/alfredo-muñoz-ríos/https://www.linkedin.com/in/alfredomunoz/ Hosted on Acast. See acast.com/privacy for more information.

Vaughn and Carola discuss human cognition and how our minds deal with complexity through recognizing shapes, organizing into hierarchies, and categorizing by types. This introduces what sustainable architecture is and why we naturally benefit from this brain-friendly way of designing software. Domain-Driven Design is a thread woven throughout their conversation, which ties together Carola’s first book on sustainable architecture and her new book. It's title is "Domain-Driven Transformation," which she has coauthored with colleague Henning Schwentner. It's been newly accepted by Pearson Addison-Wesley to be published as part of Vaughn's signature series.Dr. Carola Lilienthal is Managing Director of WPS - Workplace Solutions GmbH and has been developing software architectures according to the principles of Domain Driven Design with her teams for more than 15 years. Carola has been regularly analyzing the future viability of software architectures on behalf of her customers since 2003. She is author of the book “Sustainable Software Architecture”, “Domain-Driven Transformation” and translated Vaughn Vernon’s book “Domain-Driven Design Distilled” into German. Hosted on Acast. See acast.com/privacy for more information.

Vaughn and Fran meander through the topics AsyncAPI and Event-Driven Architecture using Fran's "thinking out loud" style. They examine the motivation for AsyncAPI, which rescued a startup product team from prolific boilerplate infrastructure code. Every microservice they created suffered from enormous amounts of repeated code, which only got worse with frequent exponential growth of microservices. They discuss other areas such as inter-service shared AsyncAPI documents, versioned schema definitions, topic naming with blended message (e.g. command and event) names, topic versioning pros and cons, YAML vs KDL, add-on tools and templates, and how all these influence productivity.Fran is the founder of the AsyncAPI Initiative and also works as Director of Engineering at Postman. In his spare time, he enjoys practicing different sports and disciplines, especially those related to circus. Hosted on Acast. See acast.com/privacy for more information.

Vaughn and John discuss API portfolios and the increasing number of APIs being made available for public consumption. From John's deep knowledge and background in APIs, the timeline of early to contemporary technologies is considered for perspective. From there, they discuss monetization of strategic APIs using his innovative HyperCurrent product. In their dialogue, API First, REST, GraphQL, gRPC, asynchronous message-based and event-driven, the future of APIs and API products, as well as using APIs while applying DDD are covered. It's an opportunity to tap the knowledge of a long-time API design and tools expert.John D'Emic is the CTO and Co-Founder of HyperCurrent, the leading Enterprise API Monetization Platform. He is an architect, full-stack developer, and author with over 20 years of experience. Besides co-authoring both editions of Mule in Action, John has published numerous blogs, articles, and white papers. Hosted on Acast. See acast.com/privacy for more information.

Vaughn discusses generative AI with Cassidy Williams (@Cassidoo) regarding the SaaS AI product that she and her team have developed. They cover the challenges of generative AI that supports accuracy in content repurposing, such as: customers don't want content generated as a talk transcript with all images from the speaker's slide deck. Customers want the talk repurposed to contain the most salient points and related images that are suitable for, e.g., a blog post. What qualifies software as an AI, and what tools can help software engineers make the leap to AI application development? The two also discuss the good, the bad, and the ugly of ChatGPT, and how ethical companies can develop AI products with trustworthy ethics.Read our Contenda-generated blog post "Reactive DDD Modeling Uncertainty" and try Contenda for yourself at: https://contenda.co Hosted on Acast. See acast.com/privacy for more information.