game compression v3

[niklasf]

• tune relative importance of criteria
• better piece-square tables
• arithmetic coding (https://www.reddit.com/r/chess/comments/83xmhl/lichess_developer_update_275_improved_game/dvlsijp/) or grouping symbols in the huffman code (https://lichess.org/forum/general-chess-discussion/blog-Wqa7GiAAAOIpBLoY?page=5#44) (<1%)
• use different huffman codes depending on the position (e.g. number of legal moves, or even game phase) (6%)
• try if ideas from https://lichess.org/@/marcusbuffett/blog/compressing-chess-moves-even-further-to-37-bits-per-move/YgugQc42 generalize to non-openings
• use padding instead of storing ply count

[cyanfish]

Faster alternative to arithmetic coding:
https://en.wikipedia.org/wiki/Asymmetric_numeral_systems

[ErdoganSeref]

I'd like to work on the issue but could you explain the approach for the arithmetic coding in detail? Is our alphabet the codes of the Huffman Tree?

[niklasf]

There's no predetermined approach. This issue is basically a research project, trying to improve upon the compression we have now, and everything above is just as collection of rough ideas.

There are theoretical limits on what can be achieved with better entropy coding alone (I believe the maximum is saving 0.5 more bits per position, about 10%). Major gains would have to come from better modelling. That is, to find a better way to assign probabilities to each move in a given position, minimizing cross entropy.

[ErdoganSeref]

Which script and data did you use to generate the Huffman codes?

Using different Huffman codes depending on the game phase would be easy to implement. Just pass the read and write methods of the Huffman class a move parameter. Then the correct tree according to the game phase is selected.

But we'd have to divide each game into opening, middlegame and endgame to generate them.

I suggest move 0 to 20 is the opening, move 20 to 40 is the middlegame and move 40 upwards is the endgame.

[niklasf]

Awesome, sounds good!

The data was a sample from https://database.lichess.org/. The tools and scripts are linked in the "Ordering moves" section of https://lichess.org/blog/Wqa7GiAAAOIpBLoY/developer-update-275-improved-game-compression.

[ErdoganSeref]

Based on the number of games I guess you used December 2017 as the sample.

I couldn't find a script which performs the exact task. I guess you build a private script based on rust-pgn-reader.

Still I don't understand how you translate the probability of thousands of legal moves from the pgn to 256 Huffman codes.

[niklasf]

The legal moves in any position are first sorted based on an ordering heuristic. Then the word to encode is the index in the list of ordered moves. If the move ordering heuristic is good, index 0 is most likely to have been played, followed by 1, etc. And no chess position has more than 256 moves. See https://lichess.org/blog/Wqa7GiAAAOIpBLoY/developer-update-275-improved-game-compression for a bit more details.

One weakness of this approach is that the probability distribution is fixed across all positions, without context. For example index 7 still has some implied probability, even if we know there are only 4 legal moves. And another bit of unused context is the game phase, that you've prepared a branch for.

The linked Rust and Python scripts are not the cleanest, but they should be complete.

[niklasf]

Nice work! And sorry, looks like a major part was missing from the Rust version after all.

What were the overall results in terms of average bytes per game or move?

[ErdoganSeref]

Opening: 21.518720819986676 bytes per game
Middlegame: 12.454941800918728 bytes per game
Endgame: 3.736622027554465 bytes per game

How much did the modification improve the algorithm?

[niklasf]

It's a slight regression, with the status quo being 37.03 bytes per game. There must be a subtle mistake somewhere. Will review more closely, soon.