It seems blowing up things for progress doesn't work all so well outside the rocket industry. I guess military, but only certain things should blow up there.
It doesn’t even work that well in the Rocket Industry, honestly. And Elon knows that! The Falcon 9 rocket is a stable ‘release’ that’s not experimented on. Sure, Starship isn’t, but Starship isn’t yet a production rocket. He can ask Vulcan Centaur what happens when you have a glitch in a production model rocket. That’s gonna be grounded until they find out what broke on the launch yesterday and how to fix it going forward!
The thing missing was that with new rocket designs you expect some booms, but you take that data and you find out why so the next boom isn't from the same thing. Which in the case of anything they've touched in the FAA or other places, is not what's happening. It's a free for all sponsored by the prevailing administration. There was not a goal to fix and improve, but to make a show of it (at a minimum, I'm sure conspiracies abound on the real purposes).
I often see SpaceX as a separate thing from Musk, mainly because he's been smart enough to let them run themselves more than the other companies. And it shows.
I work at a Fortune 500 company and we have CAB meetings all the time. It’s a bit of a pain but a necessary part of maintaining stability in Production systems.
I work for a health insurance company. We kept old systems going way longer than we should because costs were predictable. We reduced volatility and could throw more money at patient care. (Due to how the company was built, there’s no shareholders so we can’t hide the money there, and in our state excesses get refunded to payers every year).
Change of guard, new boss says everything must be modern. The legacy maintenance costs are killing us. Not wrong but missing the point. They were predictable. We can manage predictable.
They move everything to cloud shit with Microsoft dynamics and it fucking sucks. Shit doesn’t work half the time, calls keep dropping, systems are slow and unreliable, AI is everywhere it doesn’t need to be and work is of lower quality, higher cost, and takes more time/money. costs WILL explode once it’s finally working because we have willingly entered into a hostage situation with the vendor and what are we gonna do about it, move again? Security is horrifyingly bad. If you mention the words “packet capture” IT stops talking to you. They just keep buying new products to staple onto this mess that promise to solve their problems instead of hiring people who know what they are fucking doing and can say the word “no” to the c-suites.
Spoke to someone the other day from a similar company, griping about the upgrades going very badly. They said they had upgrades they had to do too when the edict everything must be win11 came down. His companies IT dept actually fought the board and refused. They said they absolutely will not service win11 machines handling patient data due to security concerns. The board caved and funded switching that entire hospital system to Mac and upgrading on-prem systems where possible instead of clouding everything.
Remember how everything Apple does is on a time delay? The ram cost surge didn’t clobber then as soon as everyone else. They got a killer deal on those minis.
Once the migration was done, they spent about 30% less than expected and their downtime was minimal. IT was saying no because of the security implications. The board saw lower costs and not having to replace the entire IT dept. everybody is happy.
I mentioned this to a supervisor. They were interested and asked what industry they were in. When I said healthcare, her eyes went wide.
We wasted so much money on needless risk. That could have gone to fucking insulin or cancer treatments.
Sounds a bit like my company (thankfully we rarely hold someone’s life in our hands though). New leadership is obsessed with removing redundancies. In theory it’s a nice idea but it often falls apart when the rubber meets the road. We’re going all in on Microsoft products (started before the latest outage) and we’re not just annoyed with the migrations we’re quite pissed off at the features we’re losing from our previous products. There’s also data, propriety company information, that’s straight up going to be purged from existence because most of us have neither the time or motivation to convert artifacts into the Microsoft equivalent. 6 months later when we get asked about that research effort from 2 years ago we’ll just have to say, “What research?”
I’m going to put on my conspiracy theorist hat to say someone is lying because this looks like an attack.
Microsoft and a bunch of 911 systems went down at this time too
Microsoft and a bunch of 911 systems went down at this time too
Wouldn’t an attack be a good thing, from a narcissistic sycophant point of view? Look how quickly we responded! Buy all our stuff!
One would think that very few people have access to the master update files and even less have full access to direct it to every computer. They’re probably all based on sccm, so I’m kinda flabbergasted that enough computers took the plunge at once to close all traffic. Seems like maybe more market manipulation as a much easier answer as to how and why.
That’s not what happened at all.
Going by what they said, since that’s all we have, a new router can easily be misconfigured to take out a network, whether it’s publishing bad routes, flooding the net with verbose data, or not throttling the fiber side down to what the copper side could accept. Thats why you have redundancy and very conservative process.
Wasn’t it even on a Friday afternoon?
Definitely FNGs. Nobody that’s been in IT for 10 minutes updates shit on a Friday afternoon.
You would also probably want to advise the pilots currently in the air before takeoff…
We love read only Fridays
Given the global nature of the industry, is it even realistic to wait for a ‘down’ period? I suppose for local traffic, which would be the majority at international hubs. Would be interesting to see the graphs.
If you don’t run your change on Friday, at the very least you have people available to deal with any issues, without paying overtime.
But yes, I’m sure sometimes are quieter than ithers
Lol. Only fools test in production.
Everyone has a test environment. Some people are lucky enough to have a separate environment for production…
Much like the saying, “any machine can be a smoke machine if you use it wrong enough.”
“There’s no such thing as inefficiency if all you’re doing is generating heat.”
I mean, isn’t visibility better during the day?
Seems like you’d rather it go down on a clear day than at night or during a winter storm.
You’d rather it go down with as few flights in the air as possible. Ideally no flights in the air…but ideally you can do this without taking the system down entirely.
I would say the number of flights at nightis not lower enough to matter, especially because other countries with different time zones are sending planes at different times.
You would be wrong. Go look at flights out of any national airport between 11pm - 5am. They’re significantly cheaper and have less options for a reason. Because demand is much lower during those hours and it provides a window for other work to happen to keep the airport safe and operational.
For passenger flights, even major airports can shut down their terminals.
We don’t really have any details but presumably they were able to continue with manual processes. We used to do that all the time and it still works. Just doesn’t scale, so you’d want as little traffic as possible.
He’s another one of the know-nothings maga loves, since they sound as misinformed and clueless as they are, very tribal.
Maybe conservatives shouldn’t be making important decisions for the public.
Also reality TV stars dont belong in politics
I’m sure Claude thought it was a good idea. Everyone who knew better has likely been fired or “retired” long ago.
Grok i bet, but yes.
‘yes, the new system needs a stress test, so completing it while its busy allows you to complete both at the same time’
I’m tired of everything being broken or about to be broken boss.
As democracy is perfected, the office of president represents, more and more closely, the inner soul of the people. On some great and glorious day the plain folks of the land will reach their heart's desire at last and the White House will be adorned by a downright moron.
H.L. Mencken, On Politics: A Carnival of Buncombe
“Democracy is by the people, of the people, for the people.”
Does that mean people support stupidity and pedophilia for electing Trump?
So. One of the things DOGE got rid of was the CAB - Change Advisory Board. I’m pretty sure there were established Change Windows with approvals required and a documented test plan and rollback plan. Don’t need those! That just gets in the way of business! Let’s get rid of it and see how that goes!
Yeah. We saw how it went, alright.
Edit: I should have said it’s apparent that DOGE got rid of CAB. Because this isn’t the kind of thing you do with an Information System if you have a CAB.