Looks like #Microsoft365 is melting down.
They're telling people to look in the admin console for details, but I can't log into the admin console either (and their status page is also broken).
#M365 #outage
3 media files
0 Boosts
0 Likes
This federated comment is not yet available here.
Here is my best guess of their current theory:
- Health monitor malfunctions, starts taking out servers that weren't really unhealthy.
- Remaining servers are overwhelmed by load shunted from taken-down servers.
- This provokes a vicious cycle they can't get out of because whenever a server comes back online it is immediately overwhelmed and goes red again.
- Only way to recover is to throttle inbound traffic to servers, get them back online, gradually unthrottle traffic.
#Microsoft #outage
As for how they'll reintroduce full production traffic, that's obvious…
#Microsoft #outage
- I just noticed that Outlook on my phone received a single email message at 5:58pm US/Eastern, a bit over an hour ago.
- Still no new emails in Outlook desktop. Message in my Outbox there is still stuck. - Still can't get to https://outlook.office365.com/. But! It loads slightly more than before, before crapping out.
So, faint, occasional signs of life, but still widespread debility.
#Microsoft #outage
@jik The world we're building with current IT policy for orgs is really, really stupid. We're becoming so reliant on tech we don't own.
My work server is moving to a data centre next month. All I need is 10tb and a NAS drive but no, that's not company policy.
So our capacity to do work is now at the whim of someone else and everything's slower. For which we pay a monthly fee.
We're promised a 99. 5% (or whatever) up time, which sounds great but we have like, 10 SaaS services. So the chances of one of them going down in say, a month, is actually pretty high. And then we're screwed.
/rant
This federated comment is not yet available here.
I Picked the Wrong Week to Stop Sniffing Glue
YouTube
Here, let me translate this new update from #Microsoft: "We still don't know what the fuck is wrong or how to fix it."
#outage #M365 #Microsoft365
This federated comment is not yet available here.
Good thing I'm not a stock prognosticator because there was no sell-off before the close of the market. In fact, the price bounced back. I imagine there are a lot of automated trading algorithms which consider $MSFT a guaranteed bounce-back after any drop, and they drove the stock price back up when it started to fall.
Previously: https://federate.social/@jik/117605274414544168
This federated comment is not yet available here.
I'm not much of a stock market prognosticator, but it seems likely to me that there's going to be a $MSFT selloff right before the market closes if the root cause of this outage isn't found and fixed in the next half hour or so.
Microsoft theorizes that the servers that are claiming to be unhealthy and therefore getting taken out of service and sending more load to other servers, thus overwhelming those servers, aren't _actually_ unhealthy, it's actually _the health monitor itself_ that is malfunctioning. So they're going to bypass it and see if that helps.
Look, I've been in exactly this situation. I'm not saying it's _impossible_ that the health monitor is the root cause… but it usually isn't.
#Microsoft #outage
It appears that #Microsoft has now published at least three different service health articles about this outage in the admin center—EX1550543, TM1550488, and MO1550821—and told people to look at them, and then made two of them disappear. It appears EX1550543 is the only one left standing at this point.
Followup: see https://federate.social/@jik/117605259128123177
