Something weird is happening with LLMs and chess

There are lots of people on the internet who have tried to get LLMs to play chess. The history seems to go something like this:

• Before September 2023: Wow, recent LLMs can sort of play chess! They fall apart after the early game, but they can do something! Amazing!
• September-October 2023: Wow! LLMs can now play chess at an advanced amateur level! Amazing!
• (Year of silence.)
• Recently: Wow, recent LLMs can sort of play chess! They fall apart after the early game, but they can do something! Amazing!

I can only assume that lots of other people are experimenting with recent models, getting terrible results, and then mostly not saying anything. I haven’t seen anyone say explicitly that only gpt-3.5-turbo-instruct is good at chess. No other LLM is remotely close.

via https://dynomight.net/chess/