State Machine Replication, and Why You Should Care with Doug Patti

Andrew Hunter makes code really, really fast. Before joining Jane Street, he worked for seven years at Google on multithreaded architecture, and was a tech lead for tcmalloc, Google’s world-class scalable malloc implementation. In this episode, Andrew and Ron discuss how, paradoxically, it can be easier to optimize systems at hyperscale because of the impact that even miniscule changes can have. Finding performance wins in trading systems—which operate at a smaller scale, but which have bursty, low-latency workloads—is often trickier. Andrew explains how he approaches the problem, including his favorite profiling techniques and tools for visualizing traces; the unique challenges of optimizing OCaml versus C++; and when you should and shouldn’t care about nanoseconds. They also touch on the joys of musical theater, and how to pass an interview when you’re sleep-deprived.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:“Profiling a warehouse-scale computer”Magic-traceOODA loop

Peter Bogart-Johnson was one of Jane Street’s first program managers, and helped bring the art of PMing—where that “P” variously stands for “project,” “product,” or some blend of the two—to the company at large. He’s also a poet and the editor of a literary magazine. In this episode, Peter and Ron discuss the challenge of gaining trust as an outsider: how do you teach teams a new way of doing things while preserving what’s already working? The key, Peter says, is you listen; a good PM is an anthropologist. They also discuss how paying down technical debt isn’t something you do instead of serving customers; what Jane Street looks for in PM candidates; and how to help teams coordinate in times of great change.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:LIT Magazine (more recently here)How to be a PM that engineers don’t hate and How to be an engineer that PMs don’t hate

Richard Eisenberg is one of the core maintainers of Haskell. He recently joined Jane Street’s Tools and Compilers team, where he hacks on the OCaml compiler. He and Ron discuss the powerful language feature that got him into PL design in the first place—dependent types—and its role in a world where AIs can (somewhat) competently write your code for you. They also discuss the differences between Haskell and OCaml; the perils of trying to make a language that works for everybody; and how best a company like Jane Street can collaborate with the open source community.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:Dependent typesGHCUnboxed types in OCamlLanguage extensions in Haskell

Ella Ehrlich has been a developer at Jane Street for close to a decade. During much of that time, she’s worked on Gord, one of Jane Street’s oldest and most critical systems, which is responsible for normalizing and distributing the firm’s trading data. Ella and Ron talk about how to grow and modernize a legacy system without compromising uptime, why game developers are the “musicians of software,” and some of the work Jane Street has done to try to hire a more diverse set of software engineers.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:EG, The League of Legends team that Ella is a huge fan of.Apache Kafka, the message bus that Gord migrated to.Some of the various sources of symbology you have to deal with when normalizing trading data. (Really, there are too many sources to list here!)A list of Jane Street’s recruiting Programs and Events, including INSIGHT, which focuses on women, and IN FOCUS, which focuses on historically underrepresented ethnic or racial minorities.

Doug Patti is a developer in Jane Street’s Client-Facing Tech team, where he works on a system called Concord that undergirds Jane Street’s client offerings. In this episode, Doug and Ron discuss how Concord, which has state-machine replication as its core abstraction, helps Jane Street achieve the reliability, scalability, and speed that the client business demands. They’ll also discuss Doug’s involvement in building a successor system called Aria, which is designed to deliver those same benefits to a much wider audience.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:Jane Street’s client-facing trading platformsA Signals and Threads episode on market data and multicast which discusses some of the history of state-machine replication in the markets.The FIX protocolUDP multicastReliable multicastKafka

Stephen Dolan works on Jane Street’s Tools and Compilers team where he focuses on the OCaml compiler. In this episode, Stephen and Ron take a trip down memory lane, discussing how to manage computer memory efficiently and safely. They consider trade-offs between reference counting and garbage collection, the surprising gains achieved by prefetching, and how new language features like local allocation and unboxed types could give OCaml users more control over their memory.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:Stephen’s command-line JSON processor, jqStephen’s Cambridge dissertation, “Algebraic Subtyping”, and a protoype implementation of mlsub, a language based on those ideas.A post from Stephen on how to benchmark different memory allocators.A Jane Street tech talk on “Unboxed Types for OCaml”, and an RFC in the OCaml RFC repo.A paper from Stephen and KC Sivaramakrishnan called “Bounding Data Races in Space and Time”, which is all about a new and better memory model for Multicore OCaml.Another paper describing the design of OCaml’s multicore GC.The Rust RFC for Higher-ranked trait bounds.

Anil Madhavapeddy is an academic, author, engineer, entrepreneur, and OCaml aficionado. In this episode, Anil and Ron consider the evolving role of operating systems, security on the internet, and the pending arrival (at last!) of OCaml 5.0. They also discuss using Raspberry Pis to fight climate change; the programming inspiration found in British pubs and on Moroccan beaches; and the time Anil went to a party, got drunk, and woke up with a job working on the Mars Polar Lander.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:Ron, Anil, and Jason Hickey’s book, “Real World OCaml”Anil’s personal website and Google Scholar pageThe MirageOS library operating systemCambridge University’s OCaml LabsNASA’s Mars Polar LanderThe Xen Project, home to the hypervisorThe Tezos proof-of-stake blockchainThe Coq Proof Assistant system

Ty Overby is a programmer in Jane Street’s web platform group where he works on Bonsai, our OCaml library for building interactive browser-based UI. In this episode, Ty and Ron consider the functional approach to building user interfaces. They also discuss Ty’s programming roots in Neopets, what development features they crave on the web, the unfairly maligned CSS, and why Excel is “arguably the greatest programming language ever developed.”You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:Jane Street’s Bonsai libraryThe 3D design system OpenSCADMatt Keeter’s libfive design toolsTry .NET in-browser replJane Street’s Incr_dom libraryThe Elm Architecture “pattern for architecting interactive programs”React JavaScript libraryThe Houdini proposalSvelte UI toolkit

James Somers is Jane Street’s writer-in-residence, splitting his time between English and OCaml, and helping to push forward all sorts of efforts around knowledge-sharing at Jane Street. In this episode, James and Ron talk about the role of technical writing in an organization like Jane Street, and how engineering software relates to editing prose.You can find the transcript for this episode  on our website.Some links to topics that came up in the discussion:mdx, the modified Markdown format that supports executing OCaml code blocksMore on the 4 types of technical writing that James referencesDonald Knuth’s original book on Literate ProgrammingMore on John McPhee’s use of KEDITPeter Seibel’s Coders at WorkDavid Goodsell’s The Machinery of LifeScott Huler’s Defining the WindSome of James’s writing on our tech blog

Signals & Threads is back, and we have a fun season of topics lined up, including: Building better abstractions for design and user interfaces, the role of writing in a technical organization, the approach that different languages take to memory management...and more. We hope you’ll join us. The first episode drops September 1st.