Compressing LLM Agent context [Shopify]

In this episode, we discuss how Shopify tackled a recurring cost in its AI agent by compressing detailed system instructions into a smaller learned representation. The approach preserved prediction quality in the team’s evaluations while reducing latency and GPU requirements. As businesses expand their use of LLMs, the cost of processing tokens at scale becomes an increasingly important operational consideration. Approaches like this show how improving serving efficiency can help businesses scale their AI capabilities while keeping costs manageable.For more details, you can refer to their published tech blog, linked here for your reference: https://shopify.engineering/gisting

In this episode, we talk about how Dropbox improved the search ranking performance of one of its key products. The team used a small amount of human judgment to calibrate an LLM, then used the LLM to generate training labels at roughly 100 times the human effort and train a more efficient production ranking model. With this approach, the LLM can scale human judgment, while the human reference set keeps that scaling grounded in correctness.For more details, you can refer to their published tech blog, linked here for your reference: https://dropbox.tech/machine-learning/llm-human-labeling-improving-search-relevance-dropbox-dash 

In this episode, we discuss how Pinterest moved from three independently evolving ads engagement models toward a unified modeling framework. The team started with a simple merged model, then progressively refined it by combining complementary modeling components, adding surface-specific calibration and multi-task learning, preserving surface-specific model exports, and optimizing both model computation and serving infrastructure. The result was a system that could share learning across surfaces while still supporting the differences that mattered for each one, with the final approach delivering significant improvements in both offline and online metrics.For more details, you can refer to their published tech blog, linked here for your reference: https://medium.com/pinterest-engineering/unifying-ads-engagement-modeling-across-pinterest-surfaces-4b5cd3d99e67

In this episode, we talk about Lyft’s need to predict whether riders will convert across a wide range of highly specific situations, where the available data becomes increasingly sparse as the situations become more specific. Instead of relying entirely on a global model or trusting noisy local estimates, Lyft built a hierarchical Bayesian framework where specific models can borrow information from broader models. This allows the system to capture local behavior when enough evidence exists while relying more on stable, broader patterns when local data is limited.For more details, you can refer to their published tech blog, linked here for your reference: https://eng.lyft.com/predicting-rider-conversion-in-sparse-data-environments-with-bayesian-trees-07227ff92789

In this episode, we discuss the budget allocation problem faced by Atlassian when trying to optimize its marketing investment decisions across many channels, products, and geographic markets. The company solved this with a two-stage system: a global payout model that predicts how business outcomes change with spend, followed by a constrained optimizer that finds the best allocation across the entire portfolio. They then added an independent optimizer as a reliability check and connected the system to a continuous offline-online feedback loop. The result was a double-digit percentage reduction in cost at equivalent spending levels, showing that significant gains can come simply from allocating the same money more intelligently.For more details, you can refer to their published tech blog, linked here for your reference: https://www.atlassian.com/blog/how-we-build/budget-allocation-at-scale-building-ml-optimization-system

In this episode, we discuss how LinkedIn moves beyond keyword matching to find candidates based on actual qualification fit. The team first created a scalable relevance definition using LLM-based supervision, distilled that knowledge into customized embeddings, and then designed the retrieval, ranking, infrastructure, and evaluation system around that signal. The result was a search system that improved both candidate relevance and recruiter engagement while actually returning fewer candidates.For more details, you can refer to their published tech blog, linked here for your reference: https://www.linkedin.com/blog/engineering/ai/semantic-search-for-ai-agents-at-scale-retrieval-and-ranking-for-linkedins-hiring-assistant

In this episode, we discuss how Meta measures the business value of shared ML infrastructure when that value appears indirectly through downstream model launches. Instead of relying on adoption, usage, or traditional infrastructure metrics, the team built an observational causal framework around individual model launches. By comparing feasible adopters with comparable non-adopters and controlling for major confounding factors, the team estimated the incremental revenue associated with infrastructure adoption. This produced a very different picture from simply looking at which platforms were used the most.For more details, you can refer to their published tech blog, linked here for your reference: https://medium.com/@AnalyticsAtMeta/measuring-what-matters-how-causal-inference-turned-ai-infrastructure-into-a-quantifiable-business-bbc49bddaa2b

In this episode, we discuss two LLM-assisted pipelines for turning unstructured text into a structured taxonomy with minimal manual annotation. The bottom-up approach lets semantic representations and clustering discover categories from the data, making it useful for open-ended exploration and relatively efficient at scale. The top-down approach starts with LLM-generated categories and recursively classifies and subdivides the corpus, making it more controllable and easier to align with business concepts, but at a higher computational cost. What makes both approaches effective is not simply the use of an LLM, but the way the LLM is combined with conventional techniques to create a systematic workflow for a problem that was previously highly manual.For more details, you can refer to their published tech blog, linked here for your reference: https://medium.com/data-science-at-microsoft/from-chaos-to-clarity-building-taxonomies-from-unstructured-text-using-large-language-models-c1303db3adb1

In this episode, we talk about how Instacart identified a limitation of a standard taxonomy: it could classify products, but it couldn't connect them the way real shoppers do. The team developed a solution around Semantic IDs, which compress product embeddings into hierarchical discrete codes, while contrastive learning and taxonomy-based supervision make those codes more semantically meaningful. The result is a representation that can support better product understanding, catalog quality, and related business needs.For more details, you can refer to their published tech blog, linked here for your reference: https://tech.instacart.com/semantic-ids-product-understanding-at-scale-5283e0288f5a 

In this episode, we discuss how Faire addresses a fundamental bottleneck in machine learning development: there are more potential experiments than human scientists can realistically run. They developed Autoscience, giving an AI agent a structured environment where it can plan, execute, evaluate, remember, and continuously refine experiments. What makes the approach effective is not the agent alone, but the combination of a reliable experimentation harness, integration with the existing ML stack, and the context needed to explore like an experienced Applied Scientist.For more details, you can refer to their published tech blog, linked here for your reference: https://craft.faire.com/autoscience-how-we-built-agent-driven-ml-exploration-at-faire-f3183e7deb29