@ai6yr I’ve definitely had trouble fixing servers when, for instance, sshd keeps getting oomkilled. But you’d think they could bring a server up and watch it get unhealthy. Unless they’re just failing instantly but that doesn’t seem to be the case based on the status report. I just end up feeling for their ops team. #HugOps. This stuff is hard. I hope there’s a good postmortem published. AI or not, there’s a team of really stressed-out people somewhere staring at their screens.

@ai6yr Wrong question. The correct question I think should be: Do we ever want a time where 90% of our personnel should have to know how our system operates?

@ai6yr I thought this would’ve been resolved within an hour or two, but now that it’s dragged on this long, I was thinking about it and I’m guessing it’s hard to bring the system up quickly enough to not implode in on itself as backlogged demand hammers away at the first nodes to come up.

We might be in “this has never been attempted” territory. As in the system has never been cold started with the same order of magnitude of subscribers as they have now.

…

@ai6yr “We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.”

I read this as “we’re scrambling to get emergency contracts in place with all the people we laid off who actually know how to diagnose the problems”.

@ai6yr it's also probably a matter of having enough servers for the usual load then having them under perform means they are in ddos territory

@ai6yr