I went down a rabbit hole reading the post-mortems and status logs from the Oct 21 Railway breach.
On the systems side, I don't have much to say re: the engineering response.
On the comms side, it's a different story...
🧵
1/10
0 Boosts
0 Likes
At 09:25 UTC, Railway knew enough to ask Stripe and Anthropic to revoke customer keys. Yet they kept calling it an "outage" on the status page until 21:59 UTC.
For over 12 hours, customers sat waiting when they should have been rotating production secrets.
Because Railway told vendors 12 hours before telling customers, users found out from third parties first.
3/10
Imagine finding out your infrastructure host got popped because Stripe and Anthropic suddenly nuked your production API keys, 40 minutes before your host said a word.
Worse: on-chain researchers tied wallet drains starting at 19:36 UTC to keys stored on Railway. Over two hours before the public disclosure.
The Catch-22: Railway told every customer to urgently rotate every secret...while builds were still paused.
4/10
You can't deploy new environment variables if you can't build. Free and Hobby users were locked out for over 27 hours.
If builds are down, tell users to revoke at the provider level first and give an ETA for deploys. Don't tell them to run while tying their shoelaces together.
5/10
A few other unforced errors in the comms stack:
⚬ Two timelines: Saying you spotted it Oct 21, then later saying Oct 20 without explaining the gap (letting HN/X assume the worst).
⚬ Anchoring: Calling 100,000 workspaces "a subset of users."
⚬ The email: Sending a breach notice at 22:37 UTC that literally cut off mid-sentence.
6/10
Why does this happen to good engineering teams?
Because during an incident, a hundred people pile into the Slack channel and nobody owns the comms response.
Legal wants to say nothing. Engineering wants to explain everything. Meanwhile, an outsider's theory is upvoted to the top of Hacker News while the team rewrites a Google Doc again and again (and again)
7/10
First, a golden distinction for any // all tech crisis comms:
When you announce an outage, customers wait.
When you announce a breach, customers act.
2/10