Markdown support

Dockerfile

@@ -13,7 +13,9 @@ RUN apt-get update && \
             libgmp-dev \
             libgmp10 \
             libgmpxx4ldbl \
+            librsvg2-bin \
             openjdk-8-jdk \
+            pandoc \
             python3-minimal \
             python3-pip \
             python3-plastex \

README.md

@@ -207,13 +207,13 @@ The dependencies needed to *build/install* problemtools can be installed with:
 
 And the dependencies needed to *run* problemtools can be installed with:
 
-    sudo apt install ghostscript libgmpxx4ldbl python3-minimal python-pkg-resources python3-plastex python3-yaml texlive-fonts-recommended texlive-lang-cyrillic texlive-latex-extra texlive-plain-generic tidy
+    sudo apt install ghostscript libgmpxx4ldbl librsvg2-bin pandoc python3-minimal python-pkg-resources python3-plastex python3-yaml texlive-fonts-recommended texlive-lang-cyrillic texlive-latex-extra texlive-plain-generic tidy
 
 ### Fedora
 
 On Fedora, these dependencies can be installed with:
 
-    sudo dnf install boost-regex gcc gmp-devel gmp-c++ python3 python3-pyyaml texlive-latex texlive-collection-fontsrecommended texlive-fancyhdr texlive-subfigure texlive-wrapfig texlive-import texlive-ulem texlive-xifthen texlive-overpic texlive-pbox tidy ghostscript
+    sudo dnf install boost-regex gcc gmp-devel gmp-c++ librsvg pandoc python3 python3-pyyaml texlive-latex texlive-collection-fontsrecommended texlive-fancyhdr texlive-subfigure texlive-wrapfig texlive-import texlive-ulem texlive-xifthen texlive-overpic texlive-pbox tidy ghostscript
 
 Followed by:
 

admin/docker/Dockerfile.build

@@ -25,6 +25,8 @@ RUN apt update && \
         libgmp-dev \
         libgmp10 \
         libgmpxx4ldbl \
+        librsvg2-bin \
+        pandoc \
         python3 \
         python3-pytest \
         python3-setuptools \

admin/docker/Dockerfile.full

@@ -19,10 +19,12 @@ RUN apt-get update && \
             gnustep-devel gnustep gnustep-make gnustep-common gobjc \
             libgmp3-dev \
             libmozjs-78-dev \
+            librsvg2-bin \
             lua5.4 \
             mono-complete \
             nodejs \
             ocaml-nox \
+            pandoc \
             php-cli \
             pypy \
             rustc \

admin/docker/Dockerfile.minimal

@@ -20,6 +20,8 @@ RUN apt update && \
     apt install -y \
         ghostscript \
         libgmpxx4ldbl \
+        librsvg2-bin \
+        pandoc \
         python-pkg-resources \
         python3-minimal \
         python3-yaml \

debian/control

@@ -8,7 +8,7 @@ Homepage: https://github.com/Kattis/problemtools
 
 Package: kattis-problemtools
 Architecture: any
-Depends: ${shlibs:Depends}, ${python3:Depends}, ${misc:Depends}, python3-plastex, python3-pkg-resources, texlive-plain-generic, texlive-fonts-recommended, texlive-latex-extra, texlive-lang-cyrillic, tidy, ghostscript, dvisvgm
+Depends: ${shlibs:Depends}, ${python3:Depends}, ${misc:Depends}, pandoc, python3-plastex, python3-pkg-resources, texlive-plain-generic, texlive-fonts-recommended, texlive-latex-extra, texlive-lang-cyrillic, tidy, ghostscript, dvisvgm
 Recommends: gcc, g++
 Description: Kattis Problem Tools
  These are tools to manage and verify problem packages in the

examples/README.md

@@ -24,4 +24,5 @@ more than one language.
 ## oddecho
 
 This is an example of a *scoring* problem where submissions can get
-different scores depending on which test groups they solve. It also demonstrates how an input validator might check different constraints for different test groups.
+different scores depending on which test groups they solve. It also demonstrates how an input validator might check different constraints for different test groups. The swedish statement showcases how to use images, footnotes
+and tables in Markdown.

examples/different/problem.yaml

@@ -5,6 +5,11 @@
 ## Author of the problem (default: null)
 # author:
 
+# The problem name
+# "en" may be omitted, as there is only one language
+name:
+    en: A Different Problem
+
 ## Where the problem was first used (default: null)
 source: Kattis
 # source_url:

examples/guess/problem.yaml

@@ -2,6 +2,9 @@ source: Kattis
 license: cc by-sa
 
 validation: custom interactive
+name:
+  sv: Gissa talet
+  en: Guess the Number
 
 # Override standard limits: say that the TLE solutions provided should
 # be at least 4 times above the time limit in order for us to be

examples/guess/problem_statement/problem.sv.md

@@ -0,0 +1,20 @@
+Jag tänker på ett hemligt tal mellan $1$ and $100$, kan du gissa vilket?
+Givet en gissning kommer jag att berätta om din gissning 
+var för stor, för liten eller rätt. Du får bara $10$ gissningar, använd 
+dem klokt!
+
+
+## Interaktion
+Ditt program ska skriva ut gissningar om talet.
+En gissning är en rad som enbart innehåller ett heltal mellan $1$ och $1000$.
+Efter varje gissning måste du flusha standard out.
+
+Efter varje gissning kan du läs svaret på standard in.
+Detta svar är ett av tre ord:
+
+- `lower` om talet jag tänker på är lägre än din gissning,
+- `higher` om talet jag tänker på är högre än din gissning, eller
+- `correct` om din gissning är korrekt.
+
+Efter att ha gissat rätt ska du avsluta ditt program.
+Om du gissar fel $10$ gånger får du inga fler chanser och ditt program kommer avbrytas.

examples/hello/problem.yaml

@@ -1,5 +1,8 @@
 source: Kattis
 license: public domain
+name:
+    sv: Hej Världen!
+    en: Hello World!
 
 # Fix memory limit at 512 MB.  (Note that for most problems, this
 # should not be done.  It is only done in this case because we include

examples/oddecho/problem.yaml

@@ -2,6 +2,8 @@ license: cc by-sa
 author: Johan Sannemo
 source: Principles of Algorithmic Problem Solving
 type: scoring
-name: Echo
+name:
+    en: Odd Echo
+    sv: Udda Eko
 grading:
     show_test_data_groups: true

examples/oddecho/problem_statement/problem.sv.md

@@ -0,0 +1,33 @@
+**EKO! Eko! Ek...**
+
+![CC-BY-SA 2.0 By William Craig on wikimedia.org](cave.jpg)
+
+Du älskar att skrika i grottor för att höra dina ord ekade tillbaka till dig. Tyvärr, som en hårt arbetande mjukvaruingenjör, har du
+inte tid för att komma ut och skrika i grottor så ofta. Istället skulle du vilja implementera ett program som fungerar som en ersättning för en grotta.
+
+Ibland vill du mata in några ord i programmet och få dem ekade tillbaka till dig. Men, som det är välkänt, om du skriker för snabbt i en grotta kan ekot störa de nya ord du säger. [^1] Mer specifikt, vartannat ord du säger kommer att störa ekot av ditt tidigare ord. Därför kommer endast det första, tredje, femte och så vidare ordet faktiskt att producera ett eko.
+
+Din uppgift är att skriva ett program som simulerar detta beteende.
+
+## Indata
+
+Den första raden av indata innehåller ett heltal $N$ ($1 \le N \le 10$).
+
+De följande $N$ raderna innehåller vardera ett ord. Varje ord är högst $100$ bokstäver långt och innehåller endast bokstäverna `a-z`.
+
+## Utdata
+
+Skriv ut de ord som har udda index (dvs. första, tredje, femte och så vidare) i inmatningen.
+
+
+## Poängsättning
+
+Din lösning kommer att testas på en mängd testfallsgrupper.
+För att få poäng för en grupp så måste du klara alla testfall i gruppen.
+
+| Grupp | Poäng | Begränsningar            |
+|-------|-------|--------------------------|
+| 1     | 1     | $N$ är alltid $5$        |
+| 2     | 1     | Inga ytterligare begränsningar |
+
+[^1]: [https://sv.wikipedia.org/wiki/Interferens](https://sv.wikipedia.org/wiki/Interferens)

problemtools/md2html.py

@@ -0,0 +1,148 @@
+#! /usr/bin/env python3
+# -*- coding: utf-8 -*-
+import argparse
+import html
+import os
+import re
+import shutil
+import string
+import subprocess
+
+import nh3
+
+from . import statement_common
+
+
+FOOTNOTES_STRING = '<section class="footnotes">'
+
+def convert(problem: str, options: argparse.Namespace) -> bool:
+    """Convert a Markdown statement to HTML
+
+    Args:
+        problem: path to problem directory
+        options: command-line arguments. See problem2html.py
+    """
+    problembase = os.path.splitext(os.path.basename(problem))[0]
+    destfile = string.Template(options.destfile).safe_substitute(problem=problembase)
+
+    statement_path = statement_common.find_statement(problem, extension="md",
+                                                     language=options.language)
+
+    if statement_path is None:
+        raise FileNotFoundError('No markdown statement found')
+
+    if not os.path.isfile(statement_path):
+        raise FileNotFoundError(f"Error! {statement_path} does not exist")
+
+
+    command = ["pandoc", statement_path, "-t" , "html", "--mathjax"]
+    statement_html = subprocess.run(command, capture_output=True, text=True,
+        shell=False, check=True).stdout
+
+    statement_html = sanitize_html(problem, statement_html)
+
+    templatepaths = [os.path.join(os.path.dirname(__file__), 'templates/markdown_html'),
+                     '/usr/lib/problemtools/templates/markdown_html']
+    templatepath = next((p for p in templatepaths
+        if os.path.isdir(p) and os.path.isfile(os.path.join(p, "default-layout.html"))),
+        None)
+
+    if templatepath is None:
+        raise FileNotFoundError('Could not find directory with markdown templates')
+
+    problem_name = statement_common.get_yaml_problem_name(problem, options.language)
+
+    statement_html = _substitute_template(templatepath, "default-layout.html",
+           statement_html=statement_html,
+           language=options.language,
+           title=html.escape(problem_name) if problem_name else "Missing problem name",
+           problemid=html.escape(problembase))
+
+    samples = statement_common.format_samples(problem, to_pdf=False)
+
+    # Insert samples at {{nextsample}} and {{remainingsamples}}
+    statement_html, remaining_samples = statement_common.inject_samples(statement_html, samples, "")
+
+    # Insert the remaining samples at the bottom
+    # However, footnotes should be below samples
+    if FOOTNOTES_STRING in statement_html:
+        pos = statement_html.find(FOOTNOTES_STRING)
+    else:
+        pos = statement_html.rfind("</body>")
+    statement_html = statement_html[:pos] + "".join(remaining_samples) + statement_html[pos:]
+
+    with open(destfile, "w", encoding="utf-8", errors="xmlcharrefreplace") as output_file:
+        output_file.write(statement_html)
+
+    if options.css:
+        shutil.copyfile(os.path.join(templatepath, "problem.css"), "problem.css")
+
+    return True
+
+def sanitize_html(problem: str, statement_html: str):
+    # Allow footnote ids (the anchor points you jump to)
+    def is_fn_id(s):
+        pattern_id_top = r'^fn\d+$'
+        pattern_id_bottom = r'^fnref\d+$'
+        return bool(re.fullmatch(pattern_id_top, s)) or bool(re.fullmatch(pattern_id_bottom, s))
+
+    allowed_classes = ("sample", "problemheader", "problembody",
+                    "sampleinteractionwrite", "sampleinteractionread",
+                    "footnotes")
+
+    # Annoying: nh3 will ignore exceptions in attribute_filter
+    image_fail_reason: str|None = None
+    def attribute_filter(tag, attribute, value):
+        if attribute == "class" and value in allowed_classes:
+            return value
+        if tag == "a" and attribute == "href":
+            return value
+        if tag in ("li", "a") and attribute == "id" and is_fn_id(value):
+            return value
+        if tag == "img" and attribute == "src":
+            fail = statement_common.is_image_valid(problem, value)
+            if fail:
+                nonlocal image_fail_reason
+                image_fail_reason = fail
+                return None
+            copy_image(problem, value)
+            return value
+        return None
+
+    statement_html = nh3.clean(statement_html,
+        link_rel="noopener nofollow noreferrer",
+        attribute_filter=attribute_filter,
+        tags=nh3.ALLOWED_TAGS | {"img", "a", "section"},
+        attributes={"table": {"class"}, "div": {"class"}, "section": {"class"}, "img": {"src"},
+                    "a": {"href", "id"}, "li": {"id"}},
+    )
+
+    if image_fail_reason:
+        assert isinstance(image_fail_reason, str)
+        if "Unsupported" in image_fail_reason:
+            raise ValueError(image_fail_reason)
+        raise FileNotFoundError(image_fail_reason)
+
+    return statement_html
+
+def copy_image(problem_root: str, img_src: str) -> None:
+    """Copy image to output directory
+
+    Args:
+        problem_root: the root of the problem directory
+        img_src: the image source as in the Markdown statement
+    """
+
+    source_name = os.path.join(problem_root, "statement", img_src)
+
+    if os.path.isfile(img_src): # already copied
+        return
+    shutil.copyfile(source_name, img_src)
+
+def _substitute_template(templatepath: str, templatefile: str, **params) -> str:
+    """Read the markdown template and substitute in things such as problem name,
+    statement etc using python's format syntax.
+    """
+    with open(os.path.join(templatepath, templatefile), "r", encoding="utf-8") as template_file:
+        html_template = template_file.read() % params
+    return html_template