This is FASCINATING. Researchers were able to pin down the cause of measurable global 60 second power spikes to software that does regular checks on the minute mark. They found large scale battery installations as one cause, but obviously this is true for any other computerized system that does cron jobs and has a synchronized clock. As weird as it sounds, adding random 0-59.999s delays to cron jobs in Linux distributions and other software would actually help the power grid.
16 Boosts
1 Like
Haelwenn /элвэн/ :triskell:
@lanodan@queer.hacktivis.me
·
activity timestamp
48 sec. ago
@hzulla Oh wow, I now even more love how most of my timers are based on elapsed time (monotonic or realtime).
@hzulla systemd timer units have just such a randomising setting but the default is zero so this is in essence a demonstration of the Power of The Default Setting
It is such an obvious cause once you think about it, and yet before you hear about this, no software developer or maintainer ever considers the possible effects of e.g. a default Debian cron job setting may have on the global electricity grid.
@hzulla Cron, of course, dates back to a world in which:
- computers were a significantly smaller part of the load on the power grid
- we didn't have tightly synchronized clocks everywhere (no NTP yet)
- we didn't have architectures in which power consumption was as tightly linked to computational load
Software often outlives all sorts of assumptions!
@mattblaze I suspect the full text of the 'law' would be
"Software outlives the assumptions about what is 'true' when it is written."
I remember being told not to consider 'residential to corp' use of SunRPC because that would presume *everyone* had at least a T-1 line's worth of bandwidth.
@mattblaze @hzulla Even way back when, I generally offset my cron jobs by a few minutes, just to avoid CPU and RAM (“core storage”) conflicts with other cron jobs.
@SteveBellovin @hzulla Yeah, but back then, you mostly only had to worry about conflicts on the same machine.
@hzulla Similar problem for servers and clusters with on the hour spikes in compute. It was a big issue in my previous job. Especially around 14:00 when most data sources we used had been updated.
@amethyst We had the additional fun result that since our VMs ground to a halt at 14:00, people moved to scheduled CI jobs instead, so then between 14 and 16 each day, no one could get a PR through because all the runners were queued up for the next 2h. 😅
@hzulla new 500-mile email just dropped
