User Research for Programming Languages (featuring Michael Coblenz)

When thinking about how to create a language where little models can be combined into bigger more complex system models, Marianne struggles to understand why not to take the completely straight forward approach of importing files. While searching for a good explanation she comes across the official blog of Futhark and decides to interview its lead on their design decisions.

Part of Marianne’s flash of inspiration came from a talk she’d seen about translating programs to a form Z3, a popular SMT solver, can run as proofs. Full of enthusiasm she invites the software engineer who gave that talk, Tikhon Jelvis, to elaborate on the similarities between SMT and code normally generated by the compiler.

Still struggling to understand how to implement uncertain types, Marianne calls on two friends to sit down with her and brainstorm different approaches. It looks more and more like adding uncertainty will cause the language to scale to impractical levels of computational complexity… then suddenly Marianne has a stroke of inspiration that changes everything.

Equipped with new knowledge about type systems, Marianne begins speccing out plans to implement uncertain types and inject probabilistic programming into Fault models. She picks the brain of Maria Gorinova—researcher and probabilistic programming evangelist—and wonders if she’s finally bitten off more than she can chew.

Marianne completely underestimates the complexity of type systems and struggled to figure out how inference, checking and conversion work together. After a month of scraping together a smattering of facts, she calls on expert Ron Garcia to help her figure it out.

A few weeks of user testing has revealed specific problems with the syntax of Fault. Marianne ponders various approaches to solving those problems and talks to James Houghton about the intersection between programming and system dynamic modeling.

Marianne has a working prototype of Fault, but still no idea if anyone will understand the design or find it useful. She needs to test it with some users and see whether it has the right features and syntax. To do this she talks with programming language researcher Michael Coblenz, who specializes in techniques for testing the usability of programming languages.

Inevitably the best way to make design decisions is to practice implementation over and over again. Marianne experiments with writing different types of specs and considers various ways they could be executed under the hood. Midway through she realizes she doesn't actually understand how to think about errors and attempts to do more research while her cat tries to distract her.

A parser generator like ANTLR won't handle evaluating programs written in your new language. You need to write the code for that yourself. Marianne explains how to build an AST from a parse tree, create an evaluation loop, and how esolangs (esoteric programming languages) make for great learning tools.

Writing a programming language starts with its grammar. But what is a grammar in the context of computation? And how does one go about writing one? This week Marianne talks with computational linguist Jeff Heinz about Alan Turing, Noam Chomsky and what context-free grammar actually means.