Result files are now available from the minizinc.github.io repo

Author: Michael Marte

.gitignore

@@ -1,89 +1 @@
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
-env/
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-*.egg-info/
-.installed.cfg
-*.egg
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*,cover
-.hypothesis/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-target/
-
-# IPython Notebook
-.ipynb_checkpoints
-
-# pyenv
-.python-version
-
-# celery beat schedule file
-celerybeat-schedule
-
-# dotenv
-.env
-
-# virtualenv
-venv/
-ENV/
-
-# Spyder project settings
-.spyderproject
-
-# Rope project settings
-.ropeproject
+results.db

README.md

@@ -1,68 +1,9 @@
 # minizinc-challenge-results
-Tools to build and query a database of MiniZinc challenge results
 
 The aim of this project is to provide tools to investigate the results of the [MiniZinc challenges](http://www.minizinc.org/challenge.html).
 
 Currently there are two Python scripts:
 
-* `create-results-db.py` creates an Sqlite database of results from MiniZinc challenges (`results.db`). To import results, call the script with a list of JSON result files (see below) on the command line. When the database already exists, the given results will be added unless they are already in the database.
+* `import-results.py` creates an Sqlite database of results from MiniZinc challenges (`results.db`). To import results, call the script with the year and the JSON result file on the command line. (Result files are available from https://github.com/MiniZinc/minizinc.github.io/tree/main/public/challenge). When the database already exists, the given results will be added unless they are already in the database.
 
-* `compare-solvers.py` compares the performance of given solvers in a given challenge by looking at the objective values the competitors achieved. (Runtime and proof of optimality do not play a role here.) For each solver and each instance, the script computes a penalty; the worse a result in comparison to the best result, the higher the penalty. (Please see the script for the details of penalization.) In the end, the script prints the penalties in terms of their mean, standard deviation, and median.
-
-The `results` folder contains the MiniZinc challenge results from 2011 to 2024; for each year, there is a JSON file. (This data was extracted from the JavaScript files used by the MiniZinc web site.) The files contain all the data except for the scores.
-
-To generate a JSON file for import into the database, proceed as follows:
-
-* Download the results from the MiniZinc web site (e.g. http://www.minizinc.org/challenge2016/results.js) or from the web site's repo (https://github.com/MiniZinc/minizinc.github.io/tree/main/public/challenge).
-* Append this snippet:
-
-```
-function writeJson(results) {
-    var fs = require('fs');
-    fs.writeFile(
-        "results.json",
-        JSON.stringify(results, null, 2),
-        function(err, result) {
-            if(err) console.log('error', err);
-        });
-}
-
-var year = 2024
-
-if (year < 2021) {
-    writeJson({
-        "year": year,
-        "solvers": solvers,
-        "fd_solvers": fd_solvers,
-        "free_solvers": free_solvers,
-        "par_solvers": par_solvers,
-        "open_solvers": open_solvers,
-        "problems": problems,
-        "kind": kind,
-        "instances": instances,
-        "benchmarks": benchmarks,
-        "results": results,
-        "times": times,
-        "objectives": objectives
-    });
-} else {
-    with (json.results) writeJson({
-        "year": year,
-        "solvers": solvers,
-        "fd_solvers": fd_solvers,
-        "free_solvers": free_solvers,
-        "par_solvers": par_solvers,
-        "open_solvers": open_solvers,
-        "problems": problems,
-        "kind": kind,
-        "instances": instances,
-        "benchmarks": benchmarks,
-        "results": results,
-        "times": times,
-        "objectives": objectives
-    });
-}
-```
-
-* Update the year.
-* Run the resulting script with `nodejs`.
+* `eval-solvers.py` evaluates the performance of given solvers in a given challenge by looking at the objective values the competitors achieved. (Runtime and proof of optimality do not play a role here.) For each solver and each instance, the script computes a penalty; the worse a result in comparison to the best result, the higher the penalty. (Please see the script for the details of penalization.) In the end, the script prints the penalties in terms of their mean, standard deviation, and median.

create-results-db.py import-results.pycreate-results-db.py renamed to import-results.py

@@ -1,7 +1,7 @@
 #! /usr/bin/python3
 
 # This script creates a database of results from MiniZinc challenges (results.db).
-# To import results, call the script with a list of JSON result files on the command line.
+# To import results, call the script with the year and the JSON result file on the command line.
 # When the database already exists, the given results will be added unless they are
 # already in the database.
 
@@ -43,9 +43,8 @@ def fixedSolverName(solver):
     'INC': (False, False)
 }
 
-def importResults(file, cursor):
-    results = json.load(file)
-    year = results['year']
+def importResults(year, file, cursor):
+    results = json.load(file)['results']
     solvers = list(map(fixedSolverName, results['solvers']))
     problems = results['problems']
     benchmarks = results['benchmarks']
@@ -70,15 +69,15 @@ def importResults(file, cursor):
 
 def main():
     parser = argparse.ArgumentParser(description = 'Puts MiniZinc challenge results into database')
-    parser.add_argument('filenames', metavar = 'json-result-file', nargs = '+')
+    parser.add_argument('year', metavar = 'year')
+    parser.add_argument('filename', metavar = 'json-result-file')
     args = parser.parse_args()
     with sqlite3.connect("results.db") as conn:
         cursor = conn.cursor()
         createDb(cursor)
         cursor.execute('PRAGMA foreign_keys = ON');
-        for filename in args.filenames:
-            with open(filename) as file:
-                importResults(file, cursor)
-                conn.commit()
+        with open(args.filename) as file:
+            importResults(args.year, file, cursor)
+            conn.commit()
 
 main()