I went down a rabbit hole reading the post-mortems and status logs from the Oct 21 Railway breach.
On the systems side, I don't have much to say re: the engineering response.
On the comms side, it's a different story...
🧵
1/10
0 Boosts
0 Likes
At 09:25 UTC, Railway knew enough to ask Stripe and Anthropic to revoke customer keys. Yet they kept calling it an "outage" on the status page until 21:59 UTC.
For over 12 hours, customers sat waiting when they should have been rotating production secrets.
Because Railway told vendors 12 hours before telling customers, users found out from third parties first.
3/10
Imagine finding out your infrastructure host got popped because Stripe and Anthropic suddenly nuked your production API keys, 40 minutes before your host said a word.
Worse: on-chain researchers tied wallet drains starting at 19:36 UTC to keys stored on Railway. Over two hours before the public disclosure.
The Catch-22: Railway told every customer to urgently rotate every secret...while builds were still paused.
4/10
You can't deploy new environment variables if you can't build. Free and Hobby users were locked out for over 27 hours.
If builds are down, tell users to revoke at the provider level first and give an ETA for deploys. Don't tell them to run while tying their shoelaces together.
5/10
First, a golden distinction for any // all tech crisis comms:
When you announce an outage, customers wait.
When you announce a breach, customers act.
2/10