Yep. Samsung is already researching this.
Also, I feel Bitnet would be useful here (so it would just be a few adders, no huge multipliers, much more doable on memory ICs). They could even take a hybrid approch, processing attention on the GPU ‘traditionally’ while the memory handles the huge MLPs.