AI Testing Highlights // Special MLOps Podcast Episode

//Abstract This panel speaks about the diverse landscape of AI agents, focusing on how they integrate voice interfaces, GUIs, and small language models to enhance user experiences. They'll also examine the roles of these agents in various industries, highlighting their impact on productivity, creativity, and user experience and how these empower developers to build better solutions while addressing challenges like ensuring consistent performance and reliability across different modalities when deploying AI agents in production. //Bio Host: Diego Oppenheimer Co-founder @ Guardrails AI Jazmia Henry Founder and CEO @ Iso AI Rogerio Bonatti Researcher @ Microsoft Julia Kroll Applied Engineer @ Deepgram Joshua Alphonse Director of Developer Relations @ PremAI A Prosus | MLOps Community Production

Lauren Kaplan is a sociologist and writer. She earned her PhD in Sociology at Goethe University Frankfurt and worked as a researcher at the University of Oxford and UC Berkeley. The Impact of UX Research in the AI Space // MLOps Podcast #272 with Lauren Kaplan, Sr UX Researcher. // Abstract In this MLOps Community podcast episode, Demetrios and UX researcher Lauren Kaplan explore how UX research can transform AI and ML projects by aligning insights with business goals and enhancing user and developer experiences. Kaplan emphasizes the importance of stakeholder alignment, proactive communication, and interdisciplinary collaboration, especially in adapting company culture post-pandemic. They discuss UX’s growing relevance in AI, challenges like bias, and the use of AI in research, underscoring the strategic value of UX in driving innovation and user satisfaction in tech. // Bio Lauren is a sociologist and writer. She earned her PhD in Sociology at Goethe University Frankfurt and worked as a researcher at the University of Oxford and UC Berkeley. Passionate about homelessness and Al, Lauren joined UCSF and later Meta. Lauren recently led UX research at a global Al chip startup and is currently seeking new opportunities to further her work in UX research and AI. At Meta, Lauren led UX research for 1) Privacy-Preserving ML and 2) PyTorch. Lauren has worked on NLP projects such as Word2Vec analysis of historical HIV/AIDS documents presented at TextXD, UC Berkeley 2019. Lauren is passionate about understanding technology and advocating for the people who create and consume Al. Lauren has published over 30 peer-reviewed research articles in domains including psychology, medicine, sociology, and more.” // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Podcast on AI UX https://open.substack.com/pub/aistudios/p/how-to-do-user-research-for-ai-products?r=7hrv8&utm_medium=ios 2024 State of AI Infra at Scale Research Report https://ai-infrastructure.org/wp-content/uploads/2024/03/The-State-of-AI-Infrastructure-at-Scale-2024.pdf Privacy-Preserving ML UX Public Article https://www.ttclabs.net/research/how-to-help-people-understand-privacy-enhancing-technologies Homelessness research and more: https://scholar.google.com/citations?user=24zqlwkAAAAJ&hl=en Agents in Production: https://home.mlops.community/public/events/aiagentsinprod Mk.gee Si (Bonus Track): https://open.spotify.com/track/1rukW2Wxnb3GGlY0uDWIWB?si=4d5b0987ad55444a --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Lauren on LinkedIn: https://www.linkedin.com/in/laurenmichellekaplan?utm_source=share&utm_campaign=share_via&utm_content=profile&utm_medium=ios_app

Dr. Petar Tsankov is a researcher and entrepreneur in the field of Computer Science and Artificial Intelligence (AI). EU AI Act - Navigating New Legislation // MLOps Podcast #271 with Petar Tsankov, Co-Founder and CEO of LatticeFlow AI. Big thanks to LatticeFlow for sponsoring this episode! // Abstract Dive into AI risk and compliance. Petar Tsankov, a leader in AI safety, talks about turning complex regulations into clear technical requirements and the importance of benchmarks in AI compliance, especially with the EU AI Act. We explore his work with big AI players and the EU on safer, compliant models, covering topics from multimodal AI to managing AI risks. He also shares insights on "Comply," an open-source tool for checking AI models against EU standards, making compliance simpler for AI developers. A must-listen for those tackling AI regulation and safety. // Bio Co-founder & CEO at LatticeFlow AI, building the world's first product enabling organizations to build performant, safe, and trustworthy AI systems. Before starting LatticeFlow AI, Petar was a senior researcher at ETH Zurich working on the security and reliability of modern systems, including deep learning models, smart contracts, and programmable networks. Petar have co-created multiple publicly available security and reliability systems that are regularly used: = ERAN, the world's first scalable verifier for deep neural networks: https://github.com/eth-sri/eran = VerX, the world's first fully automated verifier for smart contracts: https://verx.ch = Securify, the first scalable security scanner for Ethereum smart contracts: https://securify.ch = DeGuard, de-obfuscates Android binaries: http://apk-deguard.com = SyNET, the first scalable network-wide configuration synthesis tool: https://synet.ethz.ch Petar also co-founded ChainSecurity, an ETH spin-off that within 2 years became a leader in formal smart contract audits and was acquired by PwC Switzerland in 2020. // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://latticeflow.ai/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Petar on LinkedIn: https://www.linkedin.com/in/petartsankov/

Bernie Wu is VP of Business Development for MemVerge. He has 25+ years of experience as a senior executive for data center hardware and software infrastructure companies including companies such as Conner/Seagate, Cheyenne Software, Trend Micro, FalconStor, Levyx, and MetalSoft. Boosting LLM/RAG Workflows & Scheduling w/ Composable Memory and Checkpointing // MLOps Podcast #270 with Bernie Wu, VP Strategic Partnerships/Business Development of MemVerge. // Abstract Limited memory capacity hinders the performance and potential of research and production environments utilizing Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) techniques. This discussion explores how leveraging industry-standard CXL memory can be configured as a secondary, composable memory tier to alleviate this constraint. We will highlight some recent work we’ve done in integrating of this novel class of memory into LLM/RAG/vector database frameworks and workflows. Disaggregated shared memory is envisioned to offer high performance, low latency caches for model/pipeline checkpoints of LLM models, KV caches during distributed inferencing, LORA adaptors, and in-process data for heterogeneous CPU/GPU workflows. We expect to showcase these types of use cases in the coming months. // Bio Bernie is VP of Strategic Partnerships/Business Development for MemVerge. His focus has been building partnerships in the AI/ML, Kubernetes, and CXL memory ecosystems. He has 25+ years of experience as a senior executive for data center hardware and software infrastructure companies including companies such as Conner/Seagate, Cheyenne Software, Trend Micro, FalconStor, Levyx, and MetalSoft. He is also on the Board of Directors for Cirrus Data Solutions. Bernie has a BS/MS in Engineering from UC Berkeley and an MBA from UCLA. // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: www.memverge.com Accelerating Data Retrieval in Retrieval Augmentation Generation (RAG) Pipelines using CXL: https://memverge.com/accelerating-data-retrieval-in-rag-pipelines-using-cxl/ Do Re MI for Training Metrics: Start at the Beginning // Todd Underwood // AIQCON: https://youtu.be/DxyOlRdCofo Handling Multi-Terabyte LLM Checkpoints // Simon Karasik // MLOps Podcast #228: https://youtu.be/6MY-IgqiTpg Compute Express Link (CXL) FPGA IP: https://www.intel.com/content/www/us/en/products/details/fpga/intellectual-property/interface-protocols/cxl-ip.htmlUltra Ethernet Consortium: https://ultraethernet.org/ Unified Acceleration (UXL) Foundation: https://www.intel.com/content/www/us/en/developer/articles/news/unified-acceleration-uxl-foundation.html RoCE networks for distributed AI training at scale: https://engineering.fb.com/2024/08/05/data-center-engineering/roce-network-distributed-ai-training-at-scale/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Bernie on LinkedIn: https://www.linkedin.com/in/berniewu/ Timestamps: [00:00] Bernie's preferred coffee [00:11] Takeaways [01:37] First principles thinking focus [05:02] Memory Abundance Concept Discussion [06:45] Managing load spikes [09:38] GPU checkpointing challenges [16:29] Distributed memory problem solving [18:27] Composable and Virtual Memory [21:49] Interactive chat annotation [23:46] Memory elasticity in AI [27:33] GPU networking tests [29:12] GPU Scheduling workflow optimization [32:18] Kubernetes Extensions and Tools [37:14] GPU bottleneck analysis [42:04] Economical memory strategies [45:14] Elastic memory management strategies [47:57] Problem solving approach [50:15] AI infrastructure elasticity evolution [52:33] RDMA and RoCE explained [54:14] Wrap up

Gideon Mendels is the Chief Executive Officer at Comet, the leading solution for managing machine learning workflows. How to Systematically Test and Evaluate Your LLMs Apps // MLOps Podcast #269 with Gideon Mendels, CEO of Comet. // Abstract When building LLM Applications, Developers need to take a hybrid approach from both ML and SW Engineering best practices. They need to define eval metrics and track their entire experimentation to see what is and is not working. They also need to define comprehensive unit tests for their particular use-case so they can confidently check if their LLM App is ready to be deployed. // Bio Gideon Mendels is the CEO and co-founder of Comet, the leading solution for managing machine learning workflows from experimentation to production. He is a computer scientist, ML researcher and entrepreneur at his core. Before Comet, Gideon co-founded GroupWize, where they trained and deployed NLP models processing billions of chats. His journey with NLP and Speech Recognition models began at Columbia University and Google where he worked on hate speech and deception detection. // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://www.comet.com/site/ All the Hard Stuff with LLMs in Product Development // Phillip Carter // MLOps Podcast #170: https://youtu.be/DZgXln3v85s Opik by Comet: https://www.comet.com/site/products/opik/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Gideon on LinkedIn: https://www.linkedin.com/in/gideon-mendels/ Timestamps: [00:00] Gideon's preferred coffee [00:17] Takeaways [01:50] A huge shout-out to Comet ML for sponsoring this episode! [02:09] Please like, share, leave a review, and subscribe to our MLOps channels! [03:30] Evaluation metrics in AI [06:55] LLM Evaluation in Practice [10:57] LLM testing methodologies [16:56] LLM as a judge [18:53] OPIC track function overview [20:33] Tracking user response value [26:32] Exploring AI metrics integration [29:05] Experiment tracking and LLMs [34:27] Micro Macro collaboration in AI [38:20] RAG Pipeline Reproducibility Snapshot [40:15] Collaborative experiment tracking [45:29] Feature flags in CI/CD [48:55] Labeling challenges and solutions [54:31] LLM output quality alerts [56:32] Anomaly detection in model outputs [1:01:07] Wrap up

Raj Rikhy is a Senior Product Manager at Microsoft AI + R, enabling deep reinforcement learning use cases for autonomous systems. Previously, Raj was the Group Technical Product Manager in the CDO for Data Science and Deep Learning at IBM. Prior to joining IBM, Raj has been working in product management for several years - at Bitnami, Appdirect and Salesforce. // MLOps Podcast #268 with Raj Rikhy, Principal Product Manager at Microsoft. // Abstract In this MLOps Community podcast, Demetrios chats with Raj Rikhy, Principal Product Manager at Microsoft, about deploying AI agents in production. They discuss starting with simple tools, setting clear success criteria, and deploying agents in controlled environments for better scaling. Raj highlights real-time uses like fraud detection and optimizing inference costs with LLMs, while stressing human oversight during early deployment to manage LLM randomness. The episode offers practical advice on deploying AI agents thoughtfully and efficiently, avoiding over-engineering, and integrating AI into everyday applications. // Bio Raj is a Senior Product Manager at Microsoft AI + R, enabling deep reinforcement learning use cases for autonomous systems. Previously, Raj was the Group Technical Product Manager in the CDO for Data Science and Deep Learning at IBM. Prior to joining IBM, Raj has been working in product management for several years - at Bitnami, Appdirect and Salesforce. // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://www.microsoft.com/en-us/research/focus-area/ai-and-microsoft-research/ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Raj on LinkedIn: https://www.linkedin.com/in/rajrikhy/

//Abstract If there is one thing that is true, it is data is constantly changing. How can we keep up with these changes? How can we make sure that every stakeholder has visibility? How can we create a culture of understanding around data change management? //Bio - Benjamin Rogojan: Data Science And Engineering Consultant @ Seattle Data Guy - Chad Sanderson: CEO & Co-Founder @ Gable - Christophe Blefari: CTO & Co-founder @ NAO - Maggie Hays: Founding Community Product Manager, DataHub @ Acryl Data A big thank you to our Premium Sponsors  @Databricks ,  @tecton8241 , &  @onehouseHQ for their generous support!

The AI Dream Team: Strategies for ML Recruitment and Growth // MLOps Podcast #267 with Jelmer Borst, Analytics & Machine Learning Domain Lead, and Daniela Solis, Machine Learning Product Owner, of Picnic. // Abstract Like many companies, Picnic started out with a small, central data science team. As this grows larger, focussing on more complex models, it questions the skillsets & organisational set up. Use an ML platform, or build ourselves? A central team vs. embedded? Hire data scientists vs. ML engineers vs. MLOps engineers How to foster a team culture of end-to-end ownership How to balance short-term & long-term impact // Bio Jelmer Borst Jelmer leads the analytics & machine learning teams at Picnic, an app-only online groceries company based in The Netherlands. Whilst his background is in aerospace engineering, he was looking for something faster-paced and found that at Picnic. He loves the intersection of solving business challenges using technology & data. In his free time loves to cook food and tinker with the latest AI developments. Daniela Solis Morales As a Machine Learning Lead at Picnic, I am responsible for ensuring the success of end-to-end Machine Learning systems. My work involves bringing models into production across various domains, including Personalization, Fraud Detection, and Natural Language Processing. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Jelmer on LinkedIn: https://www.linkedin.com/in/japborst Connect with Daniela on LinkedIn: https://www.linkedin.com/in/daniela-solis-morales/

Francisco Ingham, LLM consultant, NLP developer, and founder of Pampa Labs. Making Your Company LLM-native // MLOps Podcast #266 with Francisco Ingham, Founder of Pampa Labs. // Abstract Being an LLM-native is becoming one of the key differentiators among companies, in vastly different verticals. Everyone wants to use LLMs, and everyone wants to be on top of the current tech but - what does it really mean to be LLM-native? LLM-native involves two ends of a spectrum. On the one hand, we have the product or service that the company offers, which surely offers many automation opportunities. LLMs can be applied strategically to scale at a lower cost and offer a better experience for users. But being LLM-native not only involves the company's customers, it also involves each stakeholder involved in the company's operations. How can employees integrate LLMs into their daily workflows? How can we as developers leverage the advancements in the field not only as builders but as adopters? We will tackle these and other key questions for anyone looking to capitalize on the LLM wave, prioritizing real results over the hype. // Bio Currently working at Pampa Labs, where we help companies become AI-native and build AI-native products. Our expertise lies on the LLM-science side, or how to build a successful data flywheel to leverage user interactions to continuously improve the product. We also spearhead, pampa-friends - the first Spanish-speaking community of AI Engineers. Previously worked in management consulting, was a TA in fastai in SF, and led the cross-AI + dev tools team at Mercado Libre. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: pampa.ai --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Francisco on LinkedIn: https://www.linkedin.com/in/fpingham/ Timestamps: [00:00] Francisco's preferred coffee [00:13] Takeaways [00:37] Please like, share, leave a review, and subscribe to our MLOps channels! [00:51] A Literature Geek [02:41] LLM-native company [03:54] Integrating LLM in workflows [07:21] Unexpected LLM applications [10:38] LLM's in development process [14:00] Vibe check to evaluation [15:36] Experiment tracking optimizations [20:22] LLMs as judges discussion [24:43] Presentaciones automatizadas para podcast [27:48] AI operating system and agents [31:29] Importance of SEO expertise [35:33] Experimentation and evaluation [39:20] AI integration strategies [41:50] RAG approach spectrum analysis [44:40] Search vs Retrieval in AI [49:02] Recommender Systems vs RAG [52:08] LLMs in recommender systems [53:10] LLM interface design insights

Simba Khadder is the Founder & CEO of Featureform. He started his ML career in recommender systems where he architected a multi-modal personalization engine that powered 100s of millions of user’s experiences. Unpacking 3 Types of Feature Stores // MLOps Podcast #265 with Simba Khadder, Founder & CEO of Featureform. // Abstract Simba dives into how feature stores have evolved and how they now intersect with vector stores, especially in the world of machine learning and LLMs. He breaks down what embeddings are, how they power recommender systems, and why personalization is key to improving LLM prompts. Simba also sheds light on the difference between feature and vector stores, explaining how each plays its part in making ML workflows smoother. Plus, we get into the latest challenges and cool innovations happening in MLOps. // Bio Simba Khadder is the Founder & CEO of Featureform. After leaving Google, Simba founded his first company, TritonML. His startup grew quickly and Simba and his team built ML infrastructure that handled over 100M monthly active users. He instilled his learnings into Featureform’s virtual feature store. Featureform turns your existing infrastructure into a Feature Store. He’s also an avid surfer, a mixed martial artist, a published astrophysicist for his work on finding Planet 9, and he ran the SF marathon in basketball shoes. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: featureform.comBigQuery Feature Store // Nicolas Mauti // MLOps Podcast #255: https://www.youtube.com/watch?v=NtDKbGyRHXQ&ab_channel=MLOps.community --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Simba on LinkedIn: https://www.linkedin.com/in/simba-k/ Timestamps: [00:00] Simba's preferred coffee [00:08] Takeaways [02:01] Coining the term 'Embedding' [07:10] Dual Tower Recommender System [10:06] Complexity vs Reliability in AI [12:39] Vector Stores and Feature Stores [17:56] Value of Data Scientists [20:27] Scalability vs Quick Solutions [23:07] MLOps vs LLMOps Debate [24:12] Feature Stores' current landscape [32:02] ML lifecycle challenges and tools [36:16] Feature Stores bundling impact [42:13] Feature Stores and BigQuery [47:42] Virtual vs Literal Feature Store [50:13] Hadoop Community Challenges [52:46] LLM data lifecycle challenges [56:30] Personalization in prompting usage [59:09] Contextualizing company variables [1:03:10] DSPy framework adoption insights [1:05:25] Wrap up