2.5 years ago we bought 8 NVMe SSDs as OS disks for devices with next to no disk churn. The filesystem stores less than 5 GB, almost no logging, no swap partition. The systems have just been sitting there for 20k hours with the occasional debian upgrade.
The wear indicator now shows >60 % of lifetime reached, with 36 TB written to disk, on all of them.
I'm certain the OS didn't issue that many writes. My only theory left is that the drives have a firmware bug and are writing themselves to death. I hate this, I don't want to replace 8 SSDs in this economy 
0 Boosts
0 Likes
@benjojo yeah, still a bit much? maybe the nand is exceptionally bad and it constantly needs to rewrite all blocks to fight entropy
@fionafokus Lots of QLC NAND do need to re-write their contents back on after some time, but likely not that often. If you still long term metrics it would be interesting to see what the difference is between the kernel claimed writes and the drives counters.
I spent a load of time staring at this problem at a previous job found that fsync and small append-ish writes can amplify out massively due to internals on how the drive controllers are implemented. But also it sounds like you have a small-ish consumer drive, so maybe the rules are just very different on these units, My dad had a nasty case of this before: https://benjojo.co.uk/u/benjojo/h/m9kdQ15cfZPw23CcZN
@benjojo@benjojo.co.uk @fionafokus@mystical.garden I have sometimes seen some massive write amplification when the drive is "full". Is trim actually getting all the way to the disk? Disk encryption and some LVM type setups can eat it.
@fionafokus Write amplification can really get you! What were the TBW lifetime of the drive according to the spec sheets?
While it obviously wont apply to all drives, I've been hammering some MX500's that have a spec sheet TBW of ~300TB way beyond 2000TBW and they have been error free the whole time (other than being quite slow), some of these drives can clearly exceed expectations