Release 2.2

This release fixes two security issues, all users are encouraged to update:

• Since BenchExec 2.1, the setup of the container for the tool-info module (which was added in BenchExec 1.20) could silently fail, for example if user namespaces are disabled on the system. In this case the tool-info module would be executed outside of the container.
  Run execution was not affected.
• The kernel offers a keyring feature for storage of keys related to features like Kerberos and ecryptfs. Before Linux 5.2, there existed one keyring per user, and BenchExec did not prevent access from the tool inside the container to the kernel keyring of the user who started BenchExec.
  Now such accesses are forbidden (on all kernel versions) using seccomp if libseccomp2 is installed, which should the case on any standard distribution.
  Note that seccomp filters do have a slight performance impact and could prevent some binaries on exotic architectures from working. In such a case please file a bug report.

Release 2.1

benchexec can now partition the Level 3 cache of the CPU for parallel runs and measure cache usage and memory bandwidth, at least on some Intel CPUs and if the pqos and pqos_wrapper are installed. More information is in the documentation.

Furthermore, some error messages for systems without container support were improved.

Release 2.0

This release does not add new features compared to BenchExec 1.22, but removes several deprecated features and brings several other backwards-incompatible changes to make BenchExec more consistent and user-friendly:

• Support for Python 3.2 and 3.3 is removed, the minimal Python version is now 3.4.
  Additionally, runexec/RunExecutor continue to support Python 2.7 until end of 2019.
• Support for running benchmarks as a different user with sudo is removed (parameters --user/--users).
  Use container mode as better method for isolating runs.
• Container mode is enabled by default.
  It can be disabled with --no-container, but this decreases reliability of benchmarking.
• If the cpuacct cgroup is not available, CPU-time measurements and limits are not supported.
• Either container mode or the freezer cgroup are required to ensure protection against fork bombs.
• Niceness of benchmarked process is not changed, previously it was increased by 5.
• Changes to input of benchexec:
  • The memory limit given to benchexec requires an explicitly specified unit.
  • Support for <test> tags, <sourcefiles> tags, and variables named ${sourcefile_*} removed from benchmark definitions.
    Use <rundefinition>, <tasks>, and ${inputfile_*} instead.
  • Variables named ${taskdef_*} are defined only if task-definition files are used, and variables named ${inputfile_*} only otherwise.
• Changes to table-generator:
  • A column named memUsage is automatically renamed to memory.
  • A column named memory is automatically converted to Megabytes.
    Both conversions are only applied if no <column> tags are used.
• Changes to run-result data:
  • In case of aborted or failed runs, no dummy results (e.g., cputime of 0s) are present.
  • The memory results of benchexec are named memory, not memUsage.
  • Memory results have the unit B explicitly specified.
    Furthermore, units are present in all attributes of the result XML files where they were still missing.
  • Result item exitcode is removed, only RunExecutor.execute_run() still returns it, but as an object instance instead of an int.
    Use returnvalue and exitsignal instead.
• Module benchexec.test_tool_wrapper is removed, use benchexec.test_tool_info instead.
• BenchExec (both benchexec and runexec) terminates itself cleanly after aborting all runs if it receives one of the signals SIGTERM, SIGINT (Ctrl+C), or SIGQUIT.

Additionally, this release adds a fix for the container that is used since BenchExec 1.20 for the tool-info module. In this container, the environment variable HOME did not point to /home/benchexec as expected but to the user's real home directory. This broke tools like Ultimate if the /home was configured to be hidden or read-only.

Furthermore, we declare the following features deprecated and plan on removing them for BenchExec 3.0, which is expected to be released in January 2020:

• Support for Python 2.7 and 3.4 (cf. #438)
• Support for checking correctness of run results and computing scores if task-definition files are not used (cf. #439)

Please respond in the respective issue if one of these deprecations is a problem for you.

Release 1.22

• More robust handling of Ctrl+C in benchexec.
  For example, output files are now always fully written, whereas previously pressing Ctrl+C at the wrong time could result in truncated files. A side effect of this is that if you call benchexec.benchexec.BenchExec().start() in own Python code, you must now add a signal handler for SIGINT. The same was already true for users of RunExecutor, this is now documented.
• Fix Ctrl+C for benchexec in container mode.
  In BenchExec 1.21, one would need to press Ctrl+C twice to stop benchexec.
• Fix unreliable container mode on Python 3.7.
• Some robustness improvements and fixes of rare deadlocks.
• Decreased overhead of benchexec while runs are executing.

Release 1.21

This release contains only a few bug fixes:

• Forwarding signals to the benchmarked process (and thus, stopping runs via Ctrl+C), was broken on Python 2.
• If the freezer cgroup was available but mounted in a separate hierarchy, it was not used reliably as protection against fork bombs when killing processes.
• Since BenchExec 1.19, an exception would occur if a non-existing command was started in container mode.
• Since BenchExec 1.19, copying output files from a container would occur while subprocesses are still running and would be counted towards the walltime limit. This is fixed, although subprocesses will still be running if the freezer cgroup is not available (cf. #433).

Release 1.20

• If benchexec --container is used, all code that is part of the tool-info module (as well as all processes started by it) are now run in a separate container with the same layout and restrictions as the run container.
  Note, however, that it is not the same container, so any modifications made by the tool-info module to files on disk are not visible in the runs!
  The test_tool_info utility also has gained a parameter --container for testing how a tool-info module behaves in a container.
• Nested containers are now supported.
  Due to a change to the internal implementation of the container mode, commands like the following succeed now:
  containerexec -- containerexec --hidden-dir /sys -- /bin/bash.
  (Some parts of /sys need to be excluded because of kernel limitations.)
  Note that nesting runexec or benchexec is still not supported, because nested cgroups are not implemented, so any cgroup-related features (resource limitations and measurements) are missing. But nesting containerexec and runexec --container (or vice-versa) now works.
• /etc/hostname in container now also shows the container's host name that exists since BenchExec 1.19.
• Change how CPUs with several NUMA nodes per CPU are handled:
  BenchExec will now treat each NUMA node like a separate CPU package and avoid creating runs that span several NUMA nodes. Thanks @alohamora!

Release 1.19

• In container mode, all temp directories are now on a tmpfs "RAM disk".
  This affects everything written to directories in the hidden or overlay modes. Files written there are now included in the memory measurements and the memory limit! The advantage is that performance should be more deterministic, especially if several runs use much I/O in parallel. This feature can be disabled with --no-tmpfs.
• /dev/shm and /run/shm are now available inside the container and provide a tmpfs instance (even with --no-tmpfs) as required by some tools for shared memory.
• Container mode now recommends LXCFS and automatically uses it if available for a better container isolation (e.g., uptime measures container uptime, not host uptime). On Debian/Ubuntu, just use sudo apt install lxcfs.
• Several small bug fixes and other improvements of isolation for container mode (e.g., host name in container is no longer the real host name).
• Add benchexec --no-hyperthreading, which restricts core assignments to a single virtual core per physical CPU core (all other sibling cores will stay unused). Thanks @alohamora!

Release 1.18

• Add result done that tools can output if the standard results true/false/unknown
  are not applicable (for example because no property was checked),
  and the run completed successfully.
• In container mode, --keep-system-config is no longer necessary if overlayfs
  is not used for /etc, and thus it is is no longer automatically implied in such cases.
• Benchmark definitions support a new attribute displayName with a human-readable name
  that will be shown in tables.
• A new variable ${taskdef_name} can now be used in places where variable substitution is supported.
• Table-generator supports % as unit for numerical values.
• Some improvements for score handling outside of SV-COMP (i.e., if scores are not calculated by BenchExec).
• New tool-info modules for Test-Comp'19
• Several small bug fixes and improvements

Release 1.17

• Tasks can now be defined in a YAML-based format, cf. the documentation.
  This supports tasks with several input files, and allows providing metadata such as expected verdicts
  in a structured format instead of encoded in the file name.
  The format will be extended to handle more information in the future.
• The wall-time limit can now be specified separately from the CPU-time limit for benchexec as command-line parameter or in the benchmark definition.
• Support for SV-COMP'19 property memcleanup.
• In containers, properly handle /run/systemd/resolve, which is necessary for DNS resolution on systems with systemd-resolved.
• Avoid warnings for mountpoints below inaccessible directories in containers.
• Improvements for handling NaN and Inf values in table-generator.
• Log output of BenchExec will now have colors if coloredlogs is installed.
• New tool-info modules and updates for SV-COMP'19.

Release 1.16

• Support for energy measurements if cpu-energy-meter is installed.
• Several small bug fixes and improvements