The Downtime Project podcast

Auth0 Silently Loses Some Indexes

19.4.2021

The Downtime Project

0:00

47:00

Auth0 experienced multiple hours of degraded performance and increased error rates in November of 2018 after several unexpected events, including a migration that dropped some indexes from their database. The published post-mortem has a full timeline and a great list of action items, though it is curiously missing a few details, like exactly what database […]

Flere episoder fra "The Downtime Project"

7 Lessons From 10 Outages
22.6.2021
46:20
After 10 post-mortems in their first season, Tom and Jamie reflect on the common issues they’ve seen. Click through for details! Summing Up Downtime We’re just about through our inaugural season of The Downtime Project podcast, and to celebrate, we’re reflecting back on recurring themes we’ve noticed in many of the ten outages we’ve poured […]
Salesforce Publishes a Controversial Postmortem (and breaks their DNS)
31.5.2021
40:51
On May 11, 2021, Salesforce had a multi hour outage that affected numerous services. Their public writeup was somewhat controversial — it’s the first one we’ve done on this show that called out the actions of a single individual in a negative light. The latest SRE Weekly has a good list of some different articles […]
Gå ikke glip af nogen episoder af “The Downtime Project” - abonnér på podcasten med gratisapp GetPodcast.
Kinesis Hits the Thread Limit
25.5.2021
44:42
During a routine addition of some servers to the Kinesis front end cluster in US-East-1 in November 2020, AWS ran into an OS limit on the max number of threads. That resulted in a multi hour outage that affected a number of other AWS servers, including ECS, EKS, Cognito, and Cloudwatch. We probably won’t do […]
How Coinbase Unleashed a Thundering Herd
17.5.2021
38:16
In November 2020, Coinbase had a problem while rotating their internal TLS certificates and accidentally unleashed a huge amount of traffic on some internal services. This was a refreshingly non-database related incident that led to an interesting discussion about the future of infrastructure as code, the limits of human code review, and how many load […]
Auth0’s Seriously Congested Database
10.5.2021
1:00
Just one day after we released Episode 5 about Auth0’s 2018 outage, Auth0 suffered a 4 hour, 20 minute outage that was caused by a combination of several large queries and a series of database cache misses. This was a very serious outage, as many users were unable to log in to sites across the […]
Talkin’ Testing with Sujay Jayakar
3.5.2021
29:37
Tom was feeling under the weather after joining Team Pfizer last week, so today we have a special guest episode with Sujay Jayakar, Jamie’s co-founder and engineer extraordinaire. While it’s great to respond well to an outage, it’s even better to design and test systems in such a way that outages don’t happen. As we […]
GitHub’s 43 Second Network Partition
26.4.2021
53:14
In 2018, after 43 seconds of connectivity issues between their East and West coast datacenters and a rapid promotion of a new primary, GitHub ended up with unique data written to two different databases. As detailed in the postmortem, this resulted in 24 hours of degraded service. This episode spends a lot of time on […]
Auth0 Silently Loses Some Indexes
19.4.2021
47:00
Auth0 experienced multiple hours of degraded performance and increased error rates in November of 2018 after several unexpected events, including a migration that dropped some indexes from their database. The published post-mortem has a full timeline and a great list of action items, though it is curiously missing a few details, like exactly what database […]
One Subtle Regex Takes Down Cloudflare
12.4.2021
54:51
On July 2, 2019, a subtle issue in a regular expression took down Cloudflare (and with it, a large portion of the internet) for 30 minutes.
Monzo’s 2019 Cassandra Outage
5.4.2021
43:26
Monzo experienced some issues while adding servers to their Cassandra cluster on July 29th, 2019. Thanks to some good practices, the team recovered quickly and no data was permanently lost.

Få adgang til hele det store podcastunivers med gratisappen GetPodcast.

Abonnér på dine favoritpodcasts, lyt til episoder offline, og få spændende anbefalinger.

© radio.de GmbH 2026

En virksomhed fra

MADSACK