An Evaluation of the Chess Models

Magnus Carlsen wins Game 11 to claim the match and remains the World Champion. Congratulations!

Before the match started, we created a prediction model based on each player’s rating to calculate the likelihood of Magnus Carlsen defending his throne, and we also broke it down into likely outcomes of when each player would win the match. After the first game we made an adjusted model that also considered the uniqueness of the match situation is the world championships and how that has played in historically, as well as considering the which player had the white pieces. Now that the match is over, it is time to see how each model fared.

Rating model:

The expected match outcome distribution of the World Chess Championships before the first game.

The expected match outcome distribution of the World Chess Championships before the first game.

Before the first game, the ratings only model had Carlsen as a huge favorite to win, and predicted that he would win it in 10 games (with 11 games being the second most likely outcome). In retrospect, it seems that the model hit very well since he won it in 11 games. That, however, does not necessarily mean that the model is correct or that the model would have been wrong had Anand won it in 11 games. It just tells us something about the likelihood based on our initial assumptions. Further, it can give us some insight into why Carlsen’s rating is down 1.0 point since the start of the match and Anand’s rating has increased accordingly. Since this model is based on those ratings, and the most likely outcome was a Carlsen win in 10 games, it makes sense that his rating is down when he used an additional game to secure the win. It can also be interesting to see the variation in the model throughout the game. In the figure below I have shown the progression of each player’s win chance as well as the chance of a tie-break, and I have indicated which game the model predict Carlsen to win the match in:

A summary of the predictions made by the Ratings only model throughout the match.

A summary of the predictions made by the Ratings only model throughout the match.

As can be seen, the model predicted Carlsen to win it in 11 games most of the time, and had him as a clear favorite the entire match. His lowest chance was after Game 5, when he had a 69.3% chance of winning outright and the most likely scenario was that he would win in 11 games. All in all I would say the model fared pretty well from its initial predictions, but that it is a bit too general when updating after each game. Which is why we made the adjusted model.

Adjusted model:

Fordeling

The expected match outcome distribution of the World Chess Championships before the first game using adjusted model.

Following the first game, there were some questions as to why Carlsen’s chances had decreased using the ratings model, when he had managed a draw with the black pieces in the first game. There was also some discussion about the fact that since this was a match between two players, that changes a lot and especially the draw rate. Which led to the adjusted model, which also had Carlsen as a huge favorite to win but in 11 games instead of in 10. Which ended up being the case. Again, just because the most likely outcome happened does not mean that the model is perfect, but in this case I do feel that the adjustments made benefited the model. Both because it allowed for better game-to-game updates, but also because it incorporated some of the unique aspects of a match while retaining the influence made by the ratings. It then also gave bigger variations when someone won a game, while varying slightly up and down after a draw based on who played with white:

A summary of the predictions made by the Adjusted model throughout the match.

A summary of the predictions made by the Adjusted model throughout the match.

This model as well predicted Carlsen to win it in 11 games most of the time, and had him as a clear favorite the entire match. His lowest chance was after Game 4, when he had a 56.4% chance of winning outright and the most likely scenario was that he would win in 12 games. Following the draw in Game 5, his chances remained mostly the same, while Anand’s chances dropped a bit. Which made sense since Carlsen was to play the majority of the remaining games with white.

Summary:

When all is said and done, I prefer the adjusted model for these types of matches. Especially since the ratings are mostly based on tournament games, where it is a higher emphasis on winning games where in a match you want to avoid losing one. So the ratings model serve as a good starting point, but should be adjusted based on the circumstances of what is to be predicted. So for the 2016 World Chess Championships we will probably use the adjusted model.

The Future:

Will we try to make similar models for other events? Possibly. There may be tournaments where it could be interesting to try to model the outcomes. However, those would not be as straight-forward as a championship match, so it would largely depend on the tournament, how much spare time there is to set up a model in advance and to do the follow-up, and if there is a demand to read about the models. There will certainly be one for the next championship match though.

In addition, we are working on trying to find a way to incorporate/adjust for player ratings with the calculations and evaluations of chess engines such as Stockfish or Houdini. We have some ideas on how to do this, which could certainly be an interesting addition to how we follow chess today.

This entry was posted in Chess, Models and tagged , , . Bookmark the permalink.

2 Responses to An Evaluation of the Chess Models

  1. Pingback: Predicting the Grenke Chess Classic | Analytic Minds

  2. Pingback: Win Chances of Grenke Chess Classic After Round 3 | Analytic Minds

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s