Chess-Ray Vision: Multi-Object Detection and Classification for Resolving Gamestate in Chessboard Photographs

We aim to solve the problem of detecting and classifying the types, colors, and positions of multiple chess pieces from photographs of chessboards. We frame this as a multi-class, multi- label task. Our results can be interpreted in terms of three distinct outcomes. First, we take 500 aerial photographs to create a new hand-crafted dataset of chess game images labeled in accordance with Forsyth-Edwards notation. Next, we devise an image preprocessing procedure that standardizes photos taken from different angles into consistent, cropped squares; this greatly reduces the complexity of the learning problem. Finally, and most importantly, we design several models, the best of which is a convolutional neural network that correctly classifies game states with over 99% accuracy.

Our training set was relatively small with a lot of noise (angle of photos, background, etc.). Using Hough Transforms, we were able to find the corners of the boards and warp every image into a perfect square, which significantly reduced the noise. With these simpler images, we were able to use a relatively straightforward CNN to classify the piece on each of the 64 spaces in the board images with near perfect accuracy.

Our paper can be found here.

Contributors

• Samuel Ryan (samryan@seas)
• Mukund Venkateswaran (mukundv@seas)
• Michael Deng (michdeng@wharton)
• Kurt Convey (kconvey@seas)

How to Use

To run on the real dataset:

1. Follow instructions below to preprocess labeled images
2. Upload the most up to real dataset notebook date notebook from clean_notebooks/ into Google Colab and update path to the preprocessed dataset.

To Preprocess Images for a Real Dataset :

Only necessary for a new dataset. There is a preprocessed dataset available here which is referenced in the current notebook.

$ git clone https://github.com/samryan18/chess-ray-vision.git
$ cd chess-ray-vision
$ pip install -e .
$ preprocess [OPTIONS] # or run with no options to be prompted for inputs

# Example run with inputs
$ preprocess --verbose --glob_path="path_to_images/*.jpeg" --dest_path="./preprocessed"

Options

Glob Path [--glob_path]

• Include --glob_path="path_to_directory/*.jpeg" to specify where the images to preprocess are stored and what the file format is
• Prompted for if not included

Destination Path [--dest_path]

• Include --dest_path="path_to_destination_directory" to specify where to put preprocessed images
• Prompted for if not included

Verbose [--verbose]

• Flag for whether to print info about the run
• Include --verbose to run in this mode

Help [--help]

• Run $ preprocess --help for help.

Results

We are nearing 100% validation accuracy.

First Perfect Classification!

Queen's Gambit :)

Loss Function

Results on Easier Dataset

As a proof of concept, we ran our models on a much easier dataset of online chessboard imgaes.

How to run on the easy dataset:

1. Upload the most up to date easy dataset notebook from clean_notebooks/ into Google Colab and run all cells.

Learning curves for a few different CNN models:

Preprocessing

Preprocessing code adapted from here. (Thanks to @Elucidation)

How Preprocessing Works:

1. Convert image to binary bitmap
2. Blur the image
3. Sobol Filter
4. Canny Edge Detectors
5. Finding Contours with cv2
6. "Prune the contours"—Ramer–Douglas–Peucker Algorithm to reconcile "almost lines"
7. Find line intersections
8. Sanity checks
   • convex hull
   • correct num points
   • check angles between lines
9. Warp images
10. Write to new files

Main Libraries Used

• PyTorch (for training models)
• opencv-python (for image preprocessing)