Hmm, learning what "break glass" means. (IT systems... "break glass" is a way to access systems if you are unable to get to other systems, i.e. if you cannot receive or send email, you still need a way to login to other systems). #breakglass
0 Boosts
0 Likes
@ai6yr yeah I work in networking and one thing I always explain when we are designing management and monitoring systems is that if there is a severe network outage every other app goes down and they can say "so sorry but the network is down, nothing we can do" _except_ for the networking department. We have to be careful how we take dependencies on central auth or other services like VM hosting so that our ability to login and recover isn't cut off if our management network is cut off by an outage. HA fail over can only get you so far, a bad condition that is exposed to both units can take down both, like bad config or bugs in the HA system itself (they add a lot of complexity which causes more problems).
@raven667 @ai6yr Could you imagine a telco, where nobody in leadership or technical capacities, had a sim on some other carrier, for when the (predictably) unthinkable happens?
Hint: you don't have to imagine. you need to get in a car and drive to Best Buy. Like RIGHT now. Because that's what they did.
@ai6yr I only learned the IT connotation recently, because I've spent my whole career in Fire Alarm Systems, where "break glass" is a type of pull station / call point
@ai6yr I was just discussing this in the context of admin access to our remote access vpn appliance. If my entire infrastructure is broken so enterprise-level auth is kaput, we would need to get to the vpn to deploy some other method for authenticating remote users who need to connect in order to rebuild things so regular auth works. I’m a 15 minute drive away so that’s not too bad but that won’t work for a coworker in Florida.
In hospital electronic records, you'd get the "break glass" warning if accessing patient records you aren't authorized to view.
When they started using that, a lot of employees who were accustomed to be able to look at their own medical records were suddenly surprised to find they were denied access.
@ai6yr in a system i used to operate, we ran a full incident postmortem every time someone had to break glass. Normally no one had access to production systems, but everyone on call was empowered to break glass and become root. Key is it's not quiet - everyone /else/ would get paged by the smash and we would get together as a team to do a full audit log review after.
@ai6yr one FinTech I worked for literally had a set of YubiKeys behind glass that contained the AWS root account passwords.
Using said break-glass accounts caused no less than 20 people to get paged.
@kajer @ai6yr this is not a bad approach, although you’d need geographic diversity in case you lost access to one location. For a while we did Shamir key splitting (k, n) of those keys, as well as root CA keys, and they were distributed to enough people to ensure diversity to recover from just about anything short of a nuclear war.
All comes down to your threat model.
@petrillic @kajer @ai6yr Shamir keys are so good (I posted the same, before your comment had backfilled on my instance)
@scott @kajer @ai6yr it solves the trust and collusion problem quite neatly. If I recall, ours was (3, 9). We figured if you can get 3 people to agree, that’s good enough. We favored uptime over security in these cases.
But as in the original posts comments, making this visible is critical. We did struggle with solving that and never did in a suitably fool-proof (damn the universe for making better fools!) manner. This is just an extension of the general idea for JIT access. Make it “easy”, but make it appropriately noisy as well.
Sadly, too many people are still sitting around trying to think of ways to make it all harder.
@ai6yr Usually it's that one random cable modem they got screwed to the plywood on the wall of the fiber room, the one we have to check on every walkthrough. It goes to a small router device. And if the router is down, there's a phone line and serial modem plugged into its terminal port. From there the network admins can poke to find out what's wrong.