This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
21 Boosts
2 Likes
Haelwenn /элвэн/ :triskell:
@lanodan@queer.hacktivis.me
·
activity timestamp
16 hr. ago
@hzulla Oh wow, I now even more love how most of my timers are based on elapsed time (monotonic or realtime).
@hzulla systemd timer units have just such a randomising setting but the default is zero so this is in essence a demonstration of the Power of The Default Setting
@hzulla I’ve always found it frustrating that the task scheduler defaults this way, knowing that jobs should be spread out… I’d love a randomize function to switch things up in the UI.
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting may have on the global electricity grid.
@hzulla more for constraints on local compute capacity than the global power grid, but I can't be the only sysadmin who routinely staggers recurring jobs likely updates or backups to kick off at semi-random times like 02:17, 03:43, etc., based on exactly this sort of reasoning… 🤔
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational load
Software often outlives all sorts of assumptions!
@mattblaze @hzulla I doubt cron is a relevant part of it.
From the article: "In Ireland, for example, where data centres with large-scale battery storage systems currently account for around a fifth of the country’s electricity demand, the amplitude of the 1-minute signal quadrupled between 2016 and 2021"
@mattblaze @hzulla Which Cron? I mean there are, for good reasons, many different versions of Cron out there. There are Crons that target specific use cases the original writers didn't consider (like having computers that are turned off or in standby from time to time).
That's the wonderful thing about Free Software. It's not a "product" but instead a chorus of competing ideas and forks which explore new things.
@casandro @hzulla Um, if open source or free software or whatever buzzword the kids are calling it these days solves this problem, why wasn’t it solved already? As you pointed out, cron’s code has been re-implemented many times over the years.
Open source is nice and all, but it didn’t anticipate or solve this.
@mattblaze I suspect the full text of the 'law' would be
"Software outlives the assumptions about what is 'true' when it is written."
I remember being told not to consider 'residential to corp' use of SunRPC because that would presume *everyone* had at least a T-1 line's worth of bandwidth.
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
@SteveBellovin @hzulla Yeah, but back then, you mostly only had to worry about conflicts on the same machine.
UK tv broadcasters had to deliberately stagger their commercial breaks as everyone put the kettle on to make tea during the commercials. And none of that non-British 1kW kettle nonsense, proper British kettles are 3kW!
@maya_b And this is why most americans don't bother with kettles -- I don't know if our outlets can even supply 3kW -- and just heat up a cup of water in the microwave instead
@psymar @maya_b I really don't think that is why: Canadiens do use electric kettles and they still are safer and work better then the microwave and we are in the same power grid as you.
(You should never boil water in the microwave, there is a small chance if you use a smooth enough glass that it won't boil when it pushes one the BP due to a lack of nucleation sites... until you touch the glass and disturb the water and it suddenly boils all at once)
@hzulla Similar problem for servers and clusters with on the hour spikes in compute. It was a big issue in my previous job. Especially around 14:00 when most data sources we used had been updated.
@amethyst We had the additional fun result that since our VMs ground to a halt at 14:00, people moved to scheduled CI jobs instead, so then between 14 and 16 each day, no one could get a PR through because all the runners were queued up for the next 2h. 😅
@hzulla new 500-mile email just dropped
