Remove dependencies on python2, move to python3

ghamerly · ghamerly · commit 47277d51ce64 · 2020-06-10T23:01:54.000Z
Also replace libmozjs with nodejs.
diff --git a/.travis.yml b/.travis.yml
@@ -1,6 +1,5 @@
 language: python
 python:
-  - 2.7
   - 3.7
 
 script: py.test
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM ubuntu:18.04
+FROM ubuntu:20.04
 
 MAINTAINER austrin@kattis.com
 
@@ -14,10 +14,10 @@ RUN apt-get update && \
             libgmp10 \
             libgmpxx4ldbl \
             openjdk-8-jdk \
-            python-minimal \
-            python-pip \
-            python-plastex \
-            python-yaml \
+            python3-minimal \
+            python3-pip \
+            python3-plastex \
+            python3-yaml \
             sudo \
             texlive-fonts-recommended \
             texlive-lang-cyrillic \
@@ -26,4 +26,4 @@ RUN apt-get update && \
             tidy \
             vim
 
-RUN pip install git+https://github.com/kattis/problemtools
+RUN pip3 install git+https://github.com/ghamerly/problemtools@move-to-py3
diff --git a/README.md b/README.md
@@ -35,12 +35,12 @@ There are four recommended ways of installing and running problemtools.
 
 Run
 ```
-pip install git+https://github.com/kattis/problemtools
+pip3 install git+https://github.com/kattis/problemtools
 ```
 
 Or if you don't want a system-wide installation,
 ```
-pip install --user git+https://github.com/kattis/problemtools
+pip3 install --user git+https://github.com/kattis/problemtools
 ```
 With this second option, in order to get the command line scripts, you need
 to make sure that the local user bin path used (e.g., on Linux,
@@ -153,18 +153,18 @@ problemtools' configuration:
 
 1. `languages.yaml`.  Use it to override problemtools' default
    programming language configuration.  For instance, while the
-   problemtools default is to use the CPython `/usr/bin/python2`
-   interpreter for Python 2, many contests, as well as the Kattis
-   online judge, use Pypy as the interpreter for Python 2.  To change
+   problemtools default is to use the CPython `/usr/bin/python3`
+   interpreter for Python 3, many contests, as well as the Kattis
+   online judge, use Pypy as the interpreter for Python 3.  To change
    this on your machine, you can simply place a file
    `/etc/kattis/problemtools/languages.yaml` (or
    `~/.config/problemtools/languages.yaml` if you only want to make the
    change for your user) containing the following:
 
    ```yaml
-   python2:
-      name: 'Python 2 w/Pypy'
-      run: '/usr/bin/pypy "{mainfile}"'
+   python3:
+      name: 'Python 3 w/Pypy'
+      run: '/usr/bin/pypy3 "{mainfile}"'
    ```
    Here, overriding the name of the language is not strictly
    necessary, but it is often helpful to clearly indicate that Pypy is
@@ -196,14 +196,14 @@ problemtools' configuration:
 
 ## Requirements and compatibility
 
-To build and run the tools, you need Python 2 with the YAML and PlasTeX libraries,
+To build and run the tools, you need Python 3 with the YAML and PlasTeX libraries,
 and a LaTeX installation.
 
 ### Ubuntu
 
 The dependencies needed to *build/install* problemtools can be installed with:
 
-    sudo apt install automake g++ make libboost-regex-dev libgmp-dev libgmp10 libgmpxx4ldbl python python-pytest python-setuptools python-yaml
+    sudo apt install automake g++ make libboost-regex-dev libgmp-dev libgmp10 libgmpxx4ldbl python3 python3-pytest python3-setuptools python3-yaml python3-plastex
 
 And the dependencies needed to *run* problemtools can be installed with:
 
@@ -213,11 +213,11 @@ And the dependencies needed to *run* problemtools can be installed with:
 
 On Fedora, these dependencies can be installed with:
 
-    sudo dnf install boost-regex gcc gmp-devel gmp-c++ python2 python2-pyyaml texlive-latex texlive-collection-fontsrecommended texlive-fancyhdr texlive-subfigure texlive-wrapfig texlive-import texlive-ulem texlive-xifthen texlive-overpic texlive-pbox tidy ghostscript
+    sudo dnf install boost-regex gcc gmp-devel gmp-c++ python3 python3-pyyaml texlive-latex texlive-collection-fontsrecommended texlive-fancyhdr texlive-subfigure texlive-wrapfig texlive-import texlive-ulem texlive-xifthen texlive-overpic texlive-pbox tidy ghostscript
 
 Followed by:
 
-    pip2 install --user plastex
+    pip3 install --user plastex
 
 ### Other platforms
 
diff --git a/admin/docker/Dockerfile.build b/admin/docker/Dockerfile.build
@@ -5,7 +5,7 @@
 # version of problemtools to be built (default is latest version of
 # develop branch on GitHub)
 
-FROM ubuntu:18.04
+FROM ubuntu:20.04
 
 LABEL maintainer="austrin@kattis.com"
 
@@ -29,10 +29,7 @@ RUN apt update && \
         python3-pytest \
         python3-setuptools \
         python3-yaml \
-        python \
-        python-pytest \
-        python-setuptools \
-        python-yaml
+        python3-setuptools
 
 RUN mkdir -p /usr/local/problemtools_build
 
diff --git a/admin/docker/Dockerfile.full b/admin/docker/Dockerfile.full
@@ -20,6 +20,7 @@ RUN apt-get update && \
             swi-prolog \
             scala \
             sbcl \
+            nodejs \
             ocaml-nox \
             rustc \
-   ;
+   ;
diff --git a/admin/docker/Dockerfile.icpc b/admin/docker/Dockerfile.icpc
@@ -18,7 +18,7 @@ RUN apt update && \
         g++ \
         openjdk-11-jdk \
         pypy \
-        python3
+        pypy3
 
 # Install Kotlin
 WORKDIR /usr/local
@@ -28,15 +28,20 @@ RUN ln -s /usr/local/kotlinc/bin/* bin/
 
 # Reconfigure problemtools:
 # - Use PyPy for Python 2
+# - Use PyPy for Python 3
 # - Use /usr/local/bin rather than /usr/bin for Kotlin
 RUN mkdir -p /etc/kattis/problemtools
 RUN echo " \n\
 python2: \n\
     name: 'Python 2 w/PyPy'\n\
     run: '/usr/bin/pypy \"{mainfile}\"'\n\
  \n\
+python3: \n\
+    name: 'Python 3 w/PyPy'\n\
+    run: '/usr/bin/pypy3 \"{mainfile}\"'\n\
+ \n\
 kotlin: \n\
     compile: '/usr/local/bin/kotlinc -d {path}/ -- {files}' \n\
     run: '/usr/local/bin/kotlin -Dfile.encoding=UTF-8 -J-XX:+UseSerialGC -J-Xss64m -J-Xms{memlim}m -J-Xmx{memlim}m -cp {path}/ {Mainclass}Kt' " > /etc/kattis/problemtools/languages.yaml
 
-WORKDIR /
+WORKDIR /
diff --git a/admin/docker/Dockerfile.minimal b/admin/docker/Dockerfile.minimal
@@ -1,6 +1,6 @@
 # Minimalistic problemtools docker image, containing only problemtools
 # and its dependencies, no languages (except whichever are
-# dependencies of problemtools, e.g. Python 2)
+# dependencies of problemtools, e.g. Python 3)
 #
 # Build requirements:
 # - The problemtools .deb package must be available from the host file
@@ -10,7 +10,7 @@
 #    PROBLEMTOOLS_VERSION but this is not checked.)
 
 ARG PROBLEMTOOLS_VERSION=develop
-FROM ubuntu:18.04
+FROM ubuntu:20.04
 
 LABEL maintainer="austrin@kattis.com"
 
@@ -20,12 +20,10 @@ RUN apt update && \
     apt install -y \
         ghostscript \
         libgmpxx4ldbl \
-        python-minimal \
         python-pkg-resources \
-        python-plastex \
-        python-yaml \
         python3-minimal \
         python3-yaml \
+        python3-plastex \
         texlive-fonts-recommended \
         texlive-lang-cyrillic \
         texlive-latex-extra \
diff --git a/bin/problem2html.sh b/bin/problem2html.sh
@@ -7,4 +7,4 @@
 
 export PYTHONPATH
 PYTHONPATH="$(dirname "$(dirname "$(readlink -f "$0")")"):$PYTHONPATH"
-exec python2 -m problemtools.problem2html "$@"
+exec python3 -m problemtools.problem2html "$@"
diff --git a/bin/problem2pdf.sh b/bin/problem2pdf.sh
@@ -7,4 +7,4 @@
 
 export PYTHONPATH
 PYTHONPATH="$(dirname "$(dirname "$(readlink -f "$0")")"):$PYTHONPATH"
-exec python2 -m problemtools.problem2pdf "$@"
+exec python3 -m problemtools.problem2pdf "$@"
diff --git a/bin/verifyproblem.sh b/bin/verifyproblem.sh
@@ -7,4 +7,4 @@
 
 export PYTHONPATH
 PYTHONPATH="$(dirname "$(dirname "$(readlink -f "$0")")"):$PYTHONPATH"
-exec python2 -m problemtools.verifyproblem "$@"
+exec python3 -m problemtools.verifyproblem "$@"
diff --git a/debian/control b/debian/control
@@ -2,13 +2,13 @@ Source: kattis-problemtools
 Section: devel
 Priority: optional
 Maintainer: Per Austrin <austrin@kattis.com>
-Build-Depends: debhelper (>= 8.0.0), g++ (>= 4.8), dh-python, python3, python3-setuptools, python3-pytest, python3-yaml, python (>= 2.6.6-3~), python-setuptools, python-pytest, python-yaml, libboost-regex-dev, libgmp-dev, automake, autoconf
+Build-Depends: debhelper (>= 8.0.0), g++ (>= 4.8), dh-python, python3, python3-setuptools, python3-pytest, python3-yaml, python3-setuptools, python3-pytest, libboost-regex-dev, libgmp-dev, automake, autoconf
 Standards-Version: 3.9.4
 Homepage: https://github.com/Kattis/problemtools
 
 Package: kattis-problemtools
 Architecture: any
-Depends: ${shlibs:Depends}, ${python:Depends}, ${python3:Depends}, ${misc:Depends}, python-yaml, plastex, python3-plastex, python-pkg-resources, texlive-plain-generic, texlive-fonts-recommended, texlive-latex-extra, texlive-lang-cyrillic, tidy, ghostscript
+Depends: ${shlibs:Depends}, ${python3:Depends}, ${misc:Depends}, python3-plastex, python-pkg-resources, texlive-plain-generic, texlive-fonts-recommended, texlive-latex-extra, texlive-lang-cyrillic, tidy, ghostscript
 Recommends: gcc, g++
 Description: Kattis Problem Tools
  These are tools to manage and verify problem packages in the
diff --git a/debian/rules b/debian/rules
@@ -11,12 +11,5 @@ export PYBUILD_AFTER_CLEAN=make -C support distclean
 export PYBUILD_TEST_PYTEST=1
 export no_proxy=github.com
 
-# FIXME: while we're building for both py2 and py3, we get a clash
-# with the CLI scripts.  The following Workaround puts the py3 CLI
-# scripts somewhere else where they don't do any harm, so that the
-# main CLI scripts are the py2 ones for now.  Once we drop py2 support
-# we also drop this line.
-export PYBUILD_INSTALL_ARGS_python3=--install-scripts=/usr/lib/problemtools/bin/
-
 %:
-	dh $@ --with python3,python2 --buildsystem=pybuild
+	dh $@ --with python3 --buildsystem=pybuild
diff --git a/examples/different/input_format_validators/validate.py b/examples/different/input_format_validators/validate.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 from sys import stdin
 import sys
 import re
diff --git a/examples/different/submissions/accepted/different.js b/examples/different/submissions/accepted/different.js
@@ -1,9 +1,14 @@
-var line;
+const readline = require('readline');
 
-while (line = readline()) {
+const rl = readline.createInterface({
+    input: process.stdin,
+    output: process.stdout
+});
+
+rl.on('line', (line) => {
     var nums = line.split(' ');
     var a = parseInt(nums[0]);
     var b = parseInt(nums[1]);
     var res = Math.abs(a - b);
-    print(res);
-}
+    console.log(res);
+});
diff --git a/examples/different/submissions/accepted/different.py b/examples/different/submissions/accepted/different.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python2
+
 import sys
 
 for line in sys.stdin:
diff --git a/examples/different/submissions/accepted/different_py3.py b/examples/different/submissions/accepted/different_py3.py
@@ -1,4 +1,4 @@
-#! /usr/bin/python3
+#!/usr/bin/env python3
 
 import sys
 
diff --git a/examples/different/submissions/slow_accepted/different_slow.py b/examples/different/submissions/slow_accepted/different_slow.py
@@ -0,0 +1,15 @@
+#! /usr/bin/python3
+
+import sys
+
+for line in sys.stdin:
+    ab = line.split()
+    a = int(ab[0])
+    b = int(ab[1])
+    # needless loop just to be slow
+    x = a
+    for _ in range(100000000):
+        x += 1
+    diff = abs(a-b) + x - x
+
+    print(diff)
diff --git a/examples/guess/input_format_validators/validate.py b/examples/guess/input_format_validators/validate.py
@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 from sys import stdin
 import sys
 import re
diff --git a/examples/guess/submissions/wrong_answer/guess.py b/examples/guess/submissions/wrong_answer/guess.py
@@ -1,3 +1,3 @@
-#!/usr/bin/python
+#!/usr/bin/env python3
 
-print "500"
+print("500")
diff --git a/examples/hello/input_format_validators/validate.py b/examples/hello/input_format_validators/validate.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 from sys import stdin
 import sys
 
diff --git a/examples/hello/submissions/accepted/hello.py b/examples/hello/submissions/accepted/hello.py
@@ -1 +1,3 @@
-print 'Hello World!'
+#!/usr/bin/env python3
+
+print('Hello World!')
diff --git a/problemtools/config/languages.yaml b/problemtools/config/languages.yaml
@@ -154,8 +154,8 @@ javascript:
     name: 'JavaScript'
     priority: 500
     files: '*.js'
-    compile: '/usr/bin/js24 -c {files}'
-    run: '/usr/bin/js24 "{mainfile}"'
+    compile: '/usr/bin/nodejs -c {files}'
+    run: '/usr/bin/nodejs "{mainfile}"'
 
 kotlin:
     name: 'Kotlin'
diff --git a/problemtools/problem2html.py b/problemtools/problem2html.py
@@ -1,4 +1,4 @@
-#! /usr/bin/env python2
+#! /usr/bin/env python3
 # -*- coding: utf-8 -*-
 from __future__ import print_function
 import re
@@ -39,11 +39,7 @@ def convert(problem, options=None):
 
         # Setup parser and renderer etc
 
-        # Python 2 compatibility: the second keyword argument for th TeX
-        # class changed name from file to myfile in Python 3 version.  When
-        # Python 2 compatibility is dropped, change this to "tex =
-        # TeX(myfile=texfile)".
-        tex = TeX(None, texfile)
+        tex = TeX(myfile=texfile)
 
         ProblemsetMacros.init(tex)
 
diff --git a/problemtools/problem2pdf.py b/problemtools/problem2pdf.py
@@ -1,4 +1,4 @@
-#! /usr/bin/env python2
+#! /usr/bin/env python3
 # -*- coding: utf-8 -*-
 import re
 import os.path
diff --git a/problemtools/tests/test_languages.py b/problemtools/tests/test_languages.py
@@ -145,7 +145,7 @@ def test_missing_run(self):
 
     def test_invalid_run(self):
         vals = self.__language_dict()
-        vals['run'] = ['python2', '{mainfile}']
+        vals['run'] = ['python3', '{mainfile}']
         with pytest.raises(languages.LanguageConfigError):
             languages.Language('id', vals)
         vals['run'] = 'echo {nonexistent}'
diff --git a/problemtools/verifyproblem.py b/problemtools/verifyproblem.py
@@ -1,4 +1,4 @@
-#! /usr/bin/env python2
+#! /usr/bin/env python3
 # -*- coding: utf-8 -*-
 from __future__ import print_function
 import glob
diff --git a/setup.py b/setup.py
@@ -1,4 +1,4 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 
 from setuptools import setup, find_packages
 from setuptools.command.bdist_egg import bdist_egg as _bdist_egg
@@ -81,7 +81,6 @@ def get_version():
       include_package_data=True,
       install_requires=[
           'PyYAML',
-          'plasTeX<=1.0;python_version<"3"',
           'plasTeX>=2.0;python_version>="3"'
       ],
 #      Temporarily disabled, see setup.cfg
diff --git a/support/default_grader/default_grader b/support/default_grader/default_grader

-Original file line number
+Diff line change
@@ @@ -1,6 +1,5 @@ @@
 language: python
 python:
 -  - 2.7
   - 3.7
 script: py.test