Skip to content

Releases: DataBiosphere/toil

9.3.0

05 Mar 21:07
@adamnovak adamnovak
releases/9.3.0
b1c1536

Choose a tag to compare

View all tags
9.3.0 Latest
Latest

Highlighted Features Added

  • Toil now supports singularity:// and file:// protocols in container in WDL, to load SIF image files by path. (#5435)

CWL

  • CWL conformance badge in the README should no longer be a broken image link (#5417)
  • CWL conformance page now has working badges (#5417)
  • toil-cwl-runner now has the --rm-tmpdir and --leave-tmpdir options that feed through to cwltool. (Note that when not bypassing the file store, CWL temp files in per-job directories will still be subject to Toil-level cleanup.) (#5226)

Dependencies

  • Update kubernetes requirement from <34,>=12.0.1 to >=12.0.1,<36 (#5437)
  • Update dill requirement from <0.4,>=0.3.2 to >=0.3.2,<0.5 (#5438)

Misc

  • threading: use psutil to calculate CPU affinitity size, which adds Windows & FreeBSD support (#5416)
    • Note that Toil overall still likely does not work on Windows and is not under test on FreeBSD.
  • --slurmTime now understands all time formats Slurm does, and doesn't silently mistake days for hours. (#5433)
  • Python workflows can now rename Toil's --config option by passing config_option to getDefaultArgumentParser() and friends. (#5420)
  • Toil no longer (repeatedly) duplicates PATH entries when applying environment variables for jobs (#5425)
  • Toil server mode should no longer be able to read partially-written state data. Workflows will no longer be reported as RUNNINGIZING, as fun as that sounds for them. (#5447)
  • Toil retry job logic now has an exponential backoff delay, controlled by --retryBackoffSeconds (default 2) and --retryBackoffFactor (default 3). (#5452)
  • Updating files in the job store by stream is now atomic: either the stream writer succeeds and the new content is visible, or something goes wrong and the old content is visible (#5456)
  • job.py now has (mostly synthetic) type hints (#5462)

Thank you to our contributors:
@mr-c, @mnijhuis-tos, and @adamnovak

Contributors

mr-c, adamnovak, and mnijhuis-tos
Assets 2
Loading

9.2.0

03 Feb 16:02
@adamnovak adamnovak
releases/9.2.0
03ecbc5

Choose a tag to compare

View all tags
9.2.0

Highlighted Features Added

  • Toil now supports Python 3.14 (#5412)

Breaking Changes

  • Toil no longer works on Python 3.9 (#5412)

CWL

  • Bump cwltool from 3.1.20250925164626 to 3.1.20251031082601 (#5407)
  • CWL file import now handles file:/path/to/file URIs, and other cases where the input filename is not the same as the normalized candidate URI, when doing import on workers. (#5392)
  • CWL file import on workers will no longer symlink tool-associated files. (#5410)

WDL

  • Toil now runs (some of) the WDL 1.2 spec tests (#5379, #5375)
  • WDL import on workers now works (again?) (#5410)

Dependencies

  • Bump github/codeql-action from 2 to 4 (#5399)
  • Update pytest-subtests requirement from <0.15,>=0.14.2 to >=0.14.2,<0.16 (#5404)
  • Update astroid requirement from <4,>=3 to >=3,<5 (#5424)
  • Update gunicorn requirement from <24,>=23.0.0 to >=23.0.0,<26 (#5443)

Misc

  • Fixed some typos found while updating the Debian package for Toil (#5403)
  • Toil no longer crashes trying to search AWS when the Flatcar release feed is unavailable (#5411)
  • Quay.io login now has automatically-generated exponential backoff code (#5440)
  • Toil Slurm cluster Docker tests are now using Python 3.12 and a new Slurm container image (#5440)
  • Toil Cactus integration test now uses a newer version of Cactus that can shepherd sonLib through the wheel build process (#5440)
  • Update AWS inventories for Toil release (#5444)
  • Use subtests from pytest 9.x (#5430)
  • Randomized deadlock detection tests replaced with static example cases that should definitively pass or fail. (#5354)

Thank you to our contributors:
@mr-c, Alexandre Detiste, and @adamnovak

Contributors

mr-c and adamnovak
Loading

9.1.2

13 Oct 14:48
@adamnovak adamnovak
releases/9.1.2
355b72b

Choose a tag to compare

View all tags
9.1.2

Misc

  • Toil workflows will no longer touch and then delete (or race each other to use) succeeded.log or failed.log in the current directory on completion. (#5086)

Thank you to our contributors:

Contributors

adamnovak
Loading

9.1.1

29 Sep 21:19
@adamnovak adamnovak
releases/9.1.1
f654a3c

Choose a tag to compare

View all tags
9.1.1

Misc

  • Using files from multiple iterations of the same task inside a scatter in WDL will no longer cause file collisions on download (#5393)

Thank you to our contributors:

Contributors

adamnovak
Loading

9.1.0

16 Sep 14:59
@adamnovak adamnovak
releases/9.1.0
e341bb6

Choose a tag to compare

View all tags
9.1.0

Highlighted Features Added

  • Toil's AWS job store now depends only on the S3 API; Amazon SimpleDB is no longer required. (#5123)
  • Toil's autoscaler should again be able to add AWS spot market nodes. (#5368)
  • Toil now checks scontrol first for Slurm jobs, and then pages through constrained time ranges of data from sacct, instead of first asking sacct about all of history. (#5381)

Breaking Changes

Because of the new AWS job store implementation (#5123), Toil 9.1 can no longer read, delete, or resume AWS job stores created with Toil 9.0 or earlier. If you need to destroy an old job store, use toil clean from toil[aws]==9.0. In general, Toil doesn't consider losing compatibility with an older Toil's job store data on disk to be a major-version-worthy change, since it might happen any time Toil revises its internal data structures, but in this case the lack of delete support for the old format is a tripping hazard that we want to call out.

Misc

  • Toil now has limited patience when registering with Mesos and will fail the workflow if Mesos never responds. (#5345)
  • CI tests that sniff logs will no longer crash when a Toil job log containing % is logged (#5353)
  • File staging tutorial script now runs from any location (#5305)
  • Node launch times in the provisioner are now always timezone-aware (#5365)
  • Toil doctests are now actually run in CI and via make test or make doctest (#5367)
  • Toil AWS spot provisioning and integrated autoscaler are now under periodic integration testing (#5370)
  • Toil now logs Slurm job numbers for debugging workflows (#5359)
  • toil debug-file should be able to fetch files again (#5382)
  • Toil will now detect and warn about jobs that use more CPU than they requested. (#5366)
  • toil stats will no longer report negative CPU Wait values when jobs use more CPU than they requested. (#5366)

Thank you to our contributors:

Contributors

mr-c, adamnovak, and DailyDreaming
Loading

9.0.0

15 Jul 18:19
@adamnovak adamnovak
releases/9.0.0
fa1e35a

Choose a tag to compare

View all tags
9.0.0

Highlighted Features Added

  • Added new unified plugin system (#5302)
  • Toil now supports URL plugins. To provide a plugin, add toil_url_access_ at beginning of python package name. To register a new URL scheme use register_plugin("url_access", scheme, implementation_type_factory). (#5302)
  • Updated real world example in quick start (#3534)
  • Toil Docker image now supports Mesos on Pythons newer than 3.10 (#5344)

Breaking Changes

  • Refactor URL access functions to use new URLAccess class instead of AbstractJobStore (#5301)
  • Toil Docker container no longer ships s3am by default (#5321)
  • Deleted deprecated addBatchSystemFactory, BATCH_SYSTEM_FACTORY_REGISTRY, and BATCH_SYSTEMS. Use add_batch_system_factory instead. (#5323)
  • --importWorkersThreshold has been renamed to --importWorkersBatchsize (#5325)

CWL

  • Fix a bug in Toil server mode CWL YAML parsing (#5252)
  • Fix missing CWL badges (#5304)
  • CWL worker import works again (#5256)
  • CWL worker import test (and other non-conformance CWL tests) now run on CI again (#5256)
  • Fixed a regression with tmpdir_prefix and nonexistent diectories (#5310)
  • Mesos CI tests no longer reference CWL conformance test class by old name (#5329)
  • Remove redundant listings when loading inputs to a job (#5099)
  • Added --cwl-min-ram to CWL. This specifies a minimum memory allocation for all jobs. This is different than --defaultMemory, which only applies to jobs without explicitly set memory requirements. (#2862)

WDL

  • Toil now deletes files that go out of scope at the ends of workflows and tasks. (#5222)
  • toil-wdl-runner can again successfully finish a workflow and emit outputs when using --restart (#5247)
  • wdl tests: use pytest fixtures to support parallel execution (#5235)
  • Added lint checking to toil-wdl-runner (#4630)
    • Use --strict=True to error on lint warnings
  • Toil can now handle coercing input Files to Strings and back in WDL workflows. (#5265)
  • Allow relaxing quantifier checks with --quantCheck (#5297)
  • toil-wdl-runner will bump container memory if below 4MiB (#5325)

Kubernetes

  • Added command line options --kubernetesPodSecurityContext and --kubernetesSecurityContext to change the security context of pods and containers (#5179)

Dependencies

  • Update to connexion 3 (#4927)

Misc

  • Toil metrics published to Dockstore now are marked as coming from TOIL and not OTHER. (#5237)
  • Toil metrics published to Dockstore are now "authenticated" with a public, non-secret token. (#5239)
  • Dockstore publication tkinter dialog now opens over other windows on Mac. (#5239)
  • Publishing metrics for the current workflow will no longer send an empty list of tasks also. (#5239)
  • needs_aws_s3: also skip tests if there is a proxy error (#5250)
  • Speed up CWL badge generation on CI (#5246)
  • gitlab-ci: add collapsable sections (with recorded durations) (#5235)
  • gitlab-ci: increase test threads to 4; standalone tests only need one test thread; use worksteal more often; upgrade cwltest, run conformance tests in parallel; enable caching of pip & mypy (#5235)
  • Toil can now build correctly for pip installs from Git again.
  • Toil now tests linting on the oldest supported Python for every PR (#5276)
  • Added a way to turn Toil history database on and off with TOIL_HISTORY. (#5254)
  • Added TOIL_JOB_HISTORY to force-enable job history tracking even though it's not ready for production use. (#5258)
  • Toil tests no longer record history to the Toil history database. (#5254)
  • Suppress spurious nvidia warnings (#5306)
  • Use different Google Cloud example file since the Broad data is now requester-pays (#5307)
  • Improve Google cloud download speed by polling for metadata once (#5311)
  • Toil Docker build is now willing to hide system Python packages (#5320)
  • Remove separate plugin test job since quick_test_offline runs them (#5334)
  • Toil server mode CI tests now run on PRs. Toil server mode integration tests now run on PRs that touch the server code. Toil server mode CI tests are now updated for Connexion 3. (#5332)
  • Toil web URL tests now run against local servers instead of httpstat.us (#5347)
  • Prepare for adamchainz/ec2-metadata#562 and in the meantime hide SSH key name in instance tags. (#5349)
  • Deprecated the Toil built-in autoscaler for Mesos (#4623)
  • Toil now supports not having lsof installed (#5236)
  • Make cleaning more idempotent (#5352)

Thank you to our contributors:

Contributors

mr-c, adamnovak, and 3 other contributors
Loading

8.2.0

30 Apr 18:56
@DailyDreaming DailyDreaming
releases/8.2.0
3efbe6f

Choose a tag to compare

View all tags
8.2.0 
releases/8.2.0

release 8.2.0
Loading

8.1.0b1

12 Mar 19:00
@adamnovak adamnovak
releases/8.1.0b1
4bb0534

Choose a tag to compare

View all tags
8.1.0b1 Pre-release
Pre-release

Note: this is a beta release with new functionality immediately needed by Cactus.

Highlighted Features Added

  • Toil can now publish workflow execution metrics to Dockstore for workflows where it knows the TRS ID (i.e. workflows run by TRS ID or Dockstore URL) (#5159).
    • Toil now saves workflow execution history in ~/.toil/history.sqlite
    • Toil will prompt the user to decide whether they want to publish workflow metrics, and, if it thinks it can get ahold of the user, will not proceed until they decide or a timeout elapses.

CWL

  • Add caching to toil-cwl-runner. Use --cachedir [dir] to enable, and avoid rerunning previously cached jobs. (#4298)
  • Add CWL badges to documentation (#5183)
  • Improve error message when a file is not found in CWL (#5174)

Misc

  • Toil will no longer select GPU or high-priority Slurm partitions when it does not need to. (#5223)
  • Allow handling unlimited number of jobs on Slurm (#5064)
  • Add hack to automatically un-stick jobs that can be un-stuck by lsof. (#5214)
  • Re-added support for --time in TOIL_SLURM_ARGS (#5230)
  • Added --slurmPartition and --slurmGPUPartition options for manual Slurm partition selection. (#5231)

Thank you to our contributors:
@stxue1 @mr-c @adamnovak

Contributors

mr-c, adamnovak, and stxue1
Loading

8.0.0

12 Feb 18:34
@stxue1 stxue1
releases/8.0.0
d2ae0ea

Choose a tag to compare

View all tags
8.0.0

Highlighted Features Added

  • toil debug-job now has --retrieveTaskDirectory <dir> which will set up a job's downloaded files under <dir> and try to stop the job after doing the downloads. Jobs can call self.files_downloaded_hook() to provide a stopping point for this mode. (#4815)
  • toil debug-job can now reconstruct the inside-the-container environment for CWL and WDL tasks. (#4815)
  • Added support for caching on Slurm and other HPC schedulers (#4775)
  • Replace all instances of boto2 with boto3 for all Toil AWS code (#4718)
  • Add support for Python 3.12 (#4718)
  • Add support for Python 3.13 (#5145)
  • Ceph input/output errors from file locking functions are now tolerated. (#4874)
  • Toil now uses flock to enable directory locks to work properly (#4924)
  • Added support to get Slurm partitions and automatically send jobs to GPUs on Slurm (#4833) (supports both CWL and WDL)
  • New --symlinkJobStoreReads=False option lets you force local-node copies (possibly in the cache) even when reading directly from a FileJobStore is possible, potentially reducing shared filesystem IO. (#4673)
  • Toil now supports reading and writing MiniWDL's call cache. (#4797)
  • Toil now supports running CWL and WDL workflows from Dockstore, by using either a Dockstore page URL or TRS ID as the URL/filename of the workflow to run. Since these often contain ? or #, remember to quote them on the command line! (#5049)
  • Add support for parallel file imports (#5114)
    • New argument --importWorkersThreshold. This specifies the threshold where files will begin to be imported on individual jobs. Small files will be batched into the same import job up to this threshold.
    • --importWorkersDisk defaults to 1 MiB. Should be increased when download streaming is not possible on a worker.

Breaking Changes

CWL

  • Prevent simultaneous Singularity container pulls in toil-cwl-runner (#4990)
  • Added support to import files on workers for toil-cwl-runner (#5025)
    • --runImportsOnWorkers to enable importing files on workers
    • --importWorkersDisk to control how much disk space the import worker will use
  • Don't error when passing through input as the output (#5138)
  • CWL jobs with dynamic requirements now have input type checking properly protected by their conditionals. (#4930)
  • Fixed a LoadListing bug with CWL workflows (#5149)
  • Fix CWL Workflow Slurm memory test (#5151)
  • workDir and jobStore defaults to tmp-outdir-prefix (#5154)
  • CWL container prepull: no reason to if extensions are enabled, they are now supported by cwl-utils 0.36+ (#5188)
  • CWL container prepull: skip if --no-containers is specified (#5188)

WDL

  • Update WDL conformance tests on CI (#4875)
  • Added support to run task only WDL files (#4960)
  • Added support for the gpu field in WDL (#4949)
  • Support passing inputs into toil-wdl-runner for task only WDLs (#4977)
  • toil-wdl-runner will now carry through task exit codes (#4978)
  • toil-wdl-runner will respect explicit null values for optional inputs (#4981)
  • toil-wdl-runner will not immediately error on nonexistent coerced files until outputted (#4994)
    • File? type for string to file coercion is now supported (will be nullified)
  • WDL output files will now live in directories named after their tasks instead of UUID directories (#5008)
  • Fixed a bug with conditional statements inside a WDL scatter (#5055)
  • toil-wdl-runner now correctly finds and returns outputs from tasks in scatters and conditionals when a WDL workflow lacks an output section. (#5094)
  • toil-wdl-runner has a new --allCallOutputs option to allow including all calls' outputs in a workflow's output. (#5093)
  • toil-wdl-runner can now detect and try not to delete the outputs of a workflow that is meant to use the Cromwell Output Organizer (croo). Note that croo still can't actually work on the output of toil-wdl-runner. (#5093)
  • --allCallOutputs no longer discards WDL workflow outputs section outputs. (#5106)
  • File virtualization in toil-wdl-runner now only happens at task boundaries (#5028)
    • File to String coercion should be supported
  • Added support to import files on workers for toil-wdl-runner (#5103)
  • Support WDL 1.1 disk specification as per spec (#5001)
  • Fixed a bug with WDL file imports (#5121)

Kubernetes

Dependencies

  • Toil can now use connexion 4 (#5196)
  • Toil now uses htcondor 23.6 or 24, which are still on PyPI

Misc

  • Makefile: use isolated builds, add dist target (sdist+wheel) and deprecate the sdist target. (#4820) (#4826)
  • Toil will now wait --jobStoreTimeout seconds (default: 30) to see an update to/removal of a job that was run, and will not let the job succeed unless it is seen to make progress. (#3814)
  • Toil job descriptions no longer have a command field, and we track the link to the job body and the command to invoke the Toil worker separately. (#4811)
  • Several typos in the docs were fixed (#4889)
  • Add a test to ensure batchsystem plugins are installable (#4879)
  • Fix Toil utils to work without the AWS extra (#4953)
  • Print commit hash with toil --version when installed from source. Before: 7.1.0a1. After: 7.1.0a1-ccf57e6071e32675daabdcbacb91988e871745a9 (#4954)
  • Fixed a broken URL and an omitted variable in CI tests (#4974)
  • Generate default config correctly (#5014)
  • Use the latest setuptools when running cactus. (#5017)
  • Toil will refuse to proceed if it detects that its coordination directory or a Singularity cache directory it needs to lock is on Ceph, to prevent hanging the Ceph MDS (#4972)
  • Fix a NotImplementedError in the Grid Engine batchsystem (#5061)
    • Added basic Grid Engine CI tests
  • Update Cactus on CI to 2.9.0 (#5062)
  • Separate out create/delete iam role functions into lib.
  • Remove deprecated pipes module (#5122)
  • New --slurmTime/TOIL_SLURM_TIME setting to set the time limit on Slurm jobs in a way Toil itself understands. (#5010)
  • New --slurmPE argument to allow setting a parallel-job Slurm partition without using TOIL_SLURM_PE (#5010)
  • New --slurmArgs argument to allow specifying extra Slurm submission arguments without using TOIL_SLURM_ARGS (#5010)
  • For non-GPU jobs on Slurm, Toil will submit the job to a partition with a time limit long enough to accommodate the configured runtime (from --slurmTime). (For GPU jobs, the lowest-priority GPU partition is still always used.) (#5010)
  • Toil now has a --slurmDefaultAllMem option to run jobs lacking their own memory requirements with Slurm's --mem=0, so they get a whole node's memory. (#4971)
  • toil-cwl-runner now has --no-cwl-default-ram (and --cwl-default-ram) to control whether the CWL spec's default ramMin is applied, or Toil's own default memory logic is used. (#4971)
  • The --dont_allocate_mem and --allocate_mem options have been deprecated and replaced with --slurmAllocateMem, which can be True or False. (#4971)
  • Added WDL unit tests to CI (#5110)
  • Mesos build updated. (#5049)
  • CWL and WDL argument parsing revised for Python 3.12. (#5049)
  • Organize stats and logging files into stats/inbox and stats/archive and avoid a circular rename. (#1727)
  • Added proper FTP support for jobstores (#5134)
  • URL existence and size gets/checks are now done with HEAD requests (#5134)
  • Dependabot configuration should now pass schema validation and is itself under CI (#5175)
  • Toil now tests a version of Cactus that ought to run on Python 3.13. (#5184)
  • WDL conformance tests on Kubernetes may now run for 30 minutes. (#5185)
  • When importing files on workers, fall back to importing on the leader when file sizes are not obtainable (#5135)

Thank you to our contributors: @stxue1, @DailyDreaming, @adamnovak, @mr-c, @gmloose, @davidjsherman!

Contributors

mr-c, adamnovak, and 4 other contributors
Loading

7.0.0

21 May 22:25
@DailyDreaming DailyDreaming
releases/7.0.0
d569ea5

Choose a tag to compare

View all tags
7.0.0

What's Changed

Contributors

mr-c, adamnovak, and 3 other contributors
Loading