Data Lakehouses & Apache Iceberg

Rosalind Whitley (Dir. Product @Kentikinc) and Ethan Smith (Staff SRE @ An Enterprise Security Company) talk about network observability, the evolution of SRE for networking, flow analysis, and the complexity and tradeoffs or network design and operations at scale.SHOW: 872 SHOW TRANSCRIPT: The Cloudcast #872 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT OUR OTHER PODCAST: "CLOUDCAST BASICS" SHOW SPONSOR:While data may be shaping our world, Data Citizens Dialogues is shaping the conversationFollow Data Citizens Dialogues on Apple, Spotify, YouTube, or wherever you get your podcastsSHOW NOTES:Kentik - The Network Observability Platform (homepage)Topic 1 - Welcome to the show. Tell us about your background and the areas you focus on today.Topic 2 - I want to begin with the complexity of networking, and not just in the sense of routing, etc. Can you walk us through some of the things you have to consider, and potentially tradeoff, as you’re thinking about network design and operations (e.g. performance, security, costs, logging, troubleshooting, etc.)Topic 3 - Usually there are many teams involved in making a large organization work. How do you collaborate across teams, and what types of technical hurdles get in the way of delivering the service? (maybe discuss logging challenges)Topic 3a - How has the role of SRE and Network Engineer evolved as we’re now dealing with this new complexity? You have all the normal performance, high availability, observability stuff, but also things like egress costs, etc. Topic 4 - We talked earlier about the challenge of cost management when running a large SaaS service. What are some of the ways that you explore or model costing behaviors and look to improve them?Topic 5 - As you’re working in a single cloud (e.g. AWS), there’s often a belief that working only in that ecosystem is the best approach, as everything is integrated or billing is simpler. How did you go about exploring alternatives that worked in your cloud, but gave you more flexibility for your services?Topic 6 - How do you tend to think about lines of demarcation between your team, the SaaS tooling provider and the cloud provider? FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

While there will be plenty of evaluation of how the 2024 US elections played out, we take a look at 13 tips that we can learn from the campaigns that can be used for tech communications.SHOW: 871SHOW TRANSCRIPT: The Cloudcast #871 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK: http://bit.ly/cloudcast-cnotwCHECK OUT OUR NEW PODCAST: "CLOUDCAST BASICS"SHOW NOTES:Dithering - Trump Tech PolicyStratechery - Big Tech, Little Tech, Chips and HardwarePivot - How Trump will Impact Tech, EconomyCube Podcast - https://www.thecube.net/cube-podcastSHOW SPONSOR:Upgrade to the next generation of the cloud.  Oracle Cloud InfrastructureCAN THE TECH INDUSTRY LEARN ANYTHING FROM POLITICAL CAMPAIGNS?It’s very difficult to disrupt the incumbent or most well-known nameIt’s more difficult to prove “what if” (future) vs. something that already happenedFigure out messaging velocity and variability (change the story vs. stick to the story)Plans tend to defeat a lack of plans (or vagueness)Fear is as powerful a motivator as hope/joy/possibilitiesComplex topics (eg. democracy, DevOps) are difficult for people to understand, if they aren’t well defined.People tend to make decision with their wallet, based on their current situationAre you speaking to a niche audience, or a large audience? If your competition defines you, you need to reframe it quicklyIf your competition finds a liability, you need to address it quicklySometimes small things can be the center of the discussionKnow where your audience is getting their information from (traditional vs. modern channels)Are you trying to get people to align on an issue, or a set of virtues (align to a group)?FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

Brian Gracely (@bgracely) and Brandon Whichard (@bwhichard) discuss the top stories in Cloud and AI from October 2024.SHOW: 870SHOW TRANSCRIPT: The Cloudcast #870 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNETCLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT OUR OTHER PODCAST - "CLOUDCAST BASICS"SHOW SPONSOR:While data may be shaping our world, Data Citizens Dialogues is shaping the conversationFollow Data Citizens Dialogues on Apple, Spotify, YouTube, or wherever you get your podcastsSHOW NOTES:Link to October 2024 news and articlesSEGMENTS COVERED IN THE SHOW:Good Old Fashioned Cloud NewsThe AI Innovation Continues FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

Keri Olson (@ksolson20, VP AI for Code at @IBM) talks about coding assistants across the software development lifecycle, the future of agents, and domain-specific assistants.SHOW: 869SHOW TRANSCRIPT: The Cloudcast #869 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK: http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT OUR OTHER PODCAST:  "CLOUDCAST BASICS" SHOW SPONSOR:While data may be shaping our world, Data Citizens Dialogues is shaping the conversationFollow Data Citizens Dialogues on Apple, Spotify, YouTube, or wherever you get your podcastsSHOW NOTES:IBM Watsonx Code Assistant (homepage)IBM Watsonx Code Assistant for Ansible Lightspeed (homepage)IBM Watsonx Code Assistant for Z (homepage)Topic 1 - Welcome to the show. Tell us about your background, and then give us a little bit of background on where you focus your time at IBM these days?Topic 2 - Developer code assistants have become one of the most popular areas of GenAI usage. At a high level, how mature are the technologies that augment developers today? Topic 3 - Software development has an entire lifecycle (Generate, Complete, Explain, Test, Transform, Document). It’s easy for developers to just plug in a service, but is that often the most effective way to start using GenAI in the software development lifecycle? Topic 4 - Software developers are notoriously picky about what tools they use and how they use them. GenAI doesn’t “guarantee” outputs. Are there concerns that if different developers or groups use different coding assistants, that it could create more challenges than it helps? Topic 5 - What is a holistic way to think about code assistants? How much should be actively engaged with developers, how much should be behind the scenes, how much will be automated or agentic in the future? Topic 6 - In the past, we essentially had “real developers” (people who wrote code) and things like Low-Code for “citizen developers” on process tasks. Do you expect to see code assistants bringing more powerful skills to people that previously hadn’t identified as a real developer? (e.g. the great idea on a napkin that turns into a mobile app)FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

When you’re asking VCs for $6B+, you probably need to promise more than normal returns. So let’s just change all of humankind over the next 5-10 years. Seems simple enough. SHOW: 868SHOW TRANSCRIPT: The Cloudcast #868 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK: http://bit.ly/cloudcast-cnotwCHECK OUT OUR NEW PODCAST: "CLOUDCAST BASICS"SHOW NOTES:“Machines of Loving Grace” (Dario Amodei - Anthropic)Why AI companies are dropping the doomerism (Decoder from The Verge)“The Intelligence Age” (Sam Altman - OpenAI)THINGS THAT AI COULD IMPROVE (according to Dario Amodei)Biology and HealthNeuroscience and MindEconomic Development and PovertyPeace and GovernanceWork and meaningFEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

Edward Hieatt (@edwardhieatt, Chief Customer Officer @mech_orchard) talks about the challenges of app modernization, leveraging AI, and the process to succeed from legacy to modern. SHOW: 867Want to go to All Things Open in Raleigh for FREE? (Oct 27th-29th)We are offering 5 Free passes, first come, first serve for the Cloudcast CommunityRegistration Link - www.eventbrite.com/e/916649672847/?discount=Cloudcastfree  Instructions:Click reg linkClick “Get Tickets”Choose ticket optionProceed with registration (discount will automatically be applied, cost will be $0)SHOW TRANSCRIPT: The Cloudcast #867 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT OUR OTHER PODCAST - "CLOUDCAST BASICS" SHOW NOTES:Mechanical Orchard (homepage)Startup led by ex-Pivotal CEO lands $50M to modernize apps (TechCrunch)Topic 1 - Welcome to the show. Tell us about your background, and then give us a little bit of background on Mechanical Orchard.Topic 2 - Many of the MO team come from Pivotal (especially Pivotal Labs), as well as being involved with the Agile Manifesto, Extreme Programming, etc. How did the mission of the company get focus on modernizing existing applications?Topic 3 - Modernization projects have traditionally been really costly, with a low success rate. Why is now the right time to focus on this area?Topic 4 - I’m really interested in how MO technology works. It seems like a variation of a digital twin, mixed with some AI capabilities. Give us the big picture of how this is a different approach to modernization.Topic 5 - How much culture/process change is needed with MO to be successful? Topic 6 - What do the stages of success look like with your approach to application modernization?FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

After several months of travel and giving keynote at multiple events, Brian gives some new tips for building keynotes, capturing audience attention, managing regional differences and avoiding mistakes.SHOW: 866SHOW TRANSCRIPT: The Cloudcast #866 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK: http://bit.ly/cloudcast-cnotwCHECK OUT OUR NEW PODCAST: "CLOUDCAST BASICS"SHOW NOTES:Building a Keynote Presentation (Cloudcast Eps. 798)THE CHALLENGES OF KEYNOTES ACROSS DIFFERENT LOCATIONSDifferent types of keynotes - Entertainment (celebrity), Vision (CEO), Informative (multiple)Different types of keynote audiences - Technical, Business, MixedDifferent types of keynote context - New, Updates, Different styles of keynote - Storytelling, Demonstrations, Multiple Speakers/PanelsOther considerations - Audience language, Slides vs. Words, Size of the room, Length of presentation slot, Before/After sessions. FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

Alex Merced (@AMdatalakehouse, Senior Tech Evangelist, @dremio) talks about everything data and we dig deep into Apache Iceberg and DataLakehouses.SHOW: 865Want to go to All Things Open in Raleigh for FREE? (Oct 27th-29th)We are offering 5 Free passes, first come, first serve for the Cloudcast Community -> Registration Link Instructions:Click reg linkClick “Get Tickets”Choose ticket optionProceed with registration (discount will automatically be applied, cost will be $0)SHOW TRANSCRIPT: The Cloudcast #865 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK: - http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT OUR OTHER PODCAST: - "CLOUDCAST BASICS" SHOW NOTES:Dremio (homepage)Hands-on with Apache Iceberg TutorialApache Iceberg Crash CourseData Lakehouses and Apache Hudi (Cloudcast Eps. 694)Apache Iceberg, the Definitive Guide (eBook)Apache Iceberg (homepage)Iceberg + Nessie Catalog (homepage)Iceberg + Polaris Catalog (homepage)AlexMerced.comDataLakehouseHub.comTopic 1 - Welcome to the show. Tell us a little bit about your background. Topic 2 - It’s been a little while since we talked about Data Lakehouses, can you give us a little bit of background on this space, and what the most recent dynamics are around these technologies.Topic 3 - What are the typical integrations with a Data Lakehouse? How are users/developers typically interacting with Data Lakehouse technologies? [The marketplace for Iceberg catalogs like Nessie and Polaris]Topic 4 - How does an open data format like Apache Iceberg fit into the bigger picture of data lakehouses, or large scale stores of data? Topic 5 - How does Dremio enable Iceberg? How does Dremio sit in the intersection of Data Lakehouse, Data Mesh and Data Virtualization trends all of which come from the same fundamental problem, the growing scale of data use cases.Topic 6 -  We’ve seen companies start to rethink their data in the cloud strategies. Are you seeing on-premises making a comeback for large data applicationsFEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

Are we sure that the next big thing in AI is going to come from OpenAI, or even one of the big cloud providers with $Billions to spend? History tells us that it might come from someone smaller. SHOW: 864SHOW TRANSCRIPT: The Cloudcast #864 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK: http://bit.ly/cloudcast-cnotwCHECK OUT OUR NEW PODCAST: "CLOUDCAST BASICS"WHERE DID THE CRITICAL “NEXT” INNOVATIONS COME FROM?Internet infrastructure (router) came from CiscoWeb Browser / Web Server came from NetscapeMainstream Linux came from Red HatSocial Media came from Twitter, FacebookCloud came from AmazonSmart Phone came from AppleAI came from ?? (OpenAI, NVIDIA) What’s Similar? What’s Different?Funding, Founders, Market CompetitorsFEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod

Emily Freeman (@editingemily) and Forrest Brazeal (@forrestbrazeal), co-founders of Freeman and Forrest talk about the intersection of technical products and marketing and the evolution of marketing to reach an increasingly technical audience including the rise of influence marketing.  SHOW: 863Want to go to All Things Open in Raleigh for FREE? (Oct 27th-29th)We are offering 5 Free passes, first come, first serve for the Cloudcast CommunityRegistration Link - www.eventbrite.com/e/916649672847/?discount=Cloudcastfree Instructions:Click reg linkClick “Get Tickets”Choose ticket optionProceed with registration (discount will automatically be applied, cost will be $0)SHOW TRANSCRIPT: The Cloudcast #863 TranscriptCLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT OUR OTHER PODCAST - "CLOUDCAST BASICS"SHOW NOTES:Freeman and Forrest (homepage)DevOps for Dummies (book)97 Things every Cloud Engineer should know (book)Forrest Brazeal (homepage)Developing Multi-Cloud Skills (Eps.602 Cloudcast)Topic 1 - Welcome to the show. Topic 2 - Where does the line between technical and marketing stop and startTopic 3 - Is there a new blueprint being created for how to have conversations about new technologies, or helping people find useful information? [new media | influencers]Topic 4 - Give us a sense of how crowded the markets are today, and why it’s so important to be able to communicate about the value/impact of the technologies.Topic 5 - Let’s talk about how projects get paid for - priorities, awareness, etc.Topic 6 -  What does good and bad look like these days?FEEDBACK?Email: show at the cloudcast dot netTwitter: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpod