I think you’re citing the wrong person. I cited the Othello-GPT line of research making the point transformers develop a world model. The other user (or you?) was mistaken about their interpretation of what a world model was representing and got caught up on the accuracy of a toy model exclusively trained on board notations like ‘a4’ and thought that the model making 95% accurate moves as a tiny parameter model trained on nothing else was somehow evidence that it didn’t form a world model from the data because it wasn’t 100% accurate, completely missing the point of the toy model scale allowing for interpretability methods of the time to identify geometric board reproduction and things like tracking game state on the board with flags for ‘my’ vs ‘opponent’ pieces, which was the data answering researcher questions about world modeling vs surface statistics in transformers.
1 Boost
0 Likes
No replies yet
Be the first to share your thoughts.