A critical mistake/misframing of the free software movement was deeming the bulk of software something desirable to make free from proprietary restrictions.
The real missing freedom was freedom *from* software. Not being forced to use any software to interact with the world.
That is to say, the majority of the harm from proprietary software is not that it being non-free keeps folks who want to use and understand it from doing so, but that it's systems of opaque rules people are forced to interact with in ways where not understanding those rules puts them at a disadvantage.
Making it so a few specialists can understand the rules (by having the source) only helps minimally.