Skip to content

CHANGELOG.md

Latest commit

 

History

History
1306 lines (1004 loc) · 105 KB

CHANGELOG.md

File metadata and controls

1306 lines (1004 loc) · 105 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Unreleased

Breaking Changes

  • Update HelmConnector to v4 API (#1046)

Added

  • Add RISC-V runners from the RISE project (#998)
  • Enforce CHANGELOG.md updates in PRs (#1066)
  • Add MPI application tutorial and docs agent skills (#1042)
  • Add fault tolerance paper reference in the documentation (#1049)

Fixed

  • Fix double-counting of occupied disk space (#1103)
  • Fix NodeJS retrieval when Docker is unavailable (#1054)
  • Fix permission propagation when transferring files between locations (#1085)
  • Fix provenance of empty unbound inputs (#968)
  • Fix bind mount inspection in SingularityConnector (#1058)
  • Fix --wait flag in Helm4Connector for Cobra parser (#1050)
  • Fix shell reuse collision across execution locations (#1045)
  • Fix operators in Hardware and Storage (#1044)
  • Fix Docker Compose NDJSON location parsing (#1043)

Removed

  • Remove OccamConnector and all associated files (#1041)

Dependencies

  • Bump aiohttp from 3.13.5 to 3.14.3 (#1079, #1088, #1118, #1120)
  • Bump asyncssh from 2.22.0 to 2.24.0 (#1064, #1089, #1106)
  • Bump cachebox from 5.2.3 to 6.2.1 (#1078, #1083, #1092, #1094, #1119, #1122)
  • Bump cryptography from 46.0.7 to 48.0.1 (#1096)
  • Bump cwltool from 3.2.20260413085819 to 3.2.20260720092025 (#1117)
  • Bump cwl-utils from 0.41 to 0.42 (#1080)
  • Bump idna from 3.11 to 3.15 (#1074)
  • Bump kaleido from 1.2.0 to 1.3.0 (#1055)
  • Bump kubernetes-asyncio from 35.0.1 to 36.1.0 (#1090)
  • Bump mistune from 3.1.4 to 3.2.1 (#1060)
  • Bump msgpack from 1.1.2 to 1.2.1 (#1101)
  • Bump plotly from 6.7.0 to 6.9.0 (#1081, #1110)
  • Bump typing-extensions from 4.15.0 to 4.16.0 (#1107)
  • Bump urllib3 from 2.6.3 to 2.7.0 (#1063)

Dev Dependencies

  • Bump actions/checkout from 6 to 7 (#1097)
  • Bump actions/setup-node from 6 to 7 (#1113)
  • Bump actions/setup-python from 6 to 7 (#1116)
  • Bump azure/setup-helm from 4 to 5 (#1051)
  • Bump black from 26.3.1 to 26.5.1 (#1073)
  • Bump codecov/codecov-action from 6 to 7 (#1087)
  • Bump codespell from 2.4.2 to 2.4.3 (#1114)
  • Bump mypy from 1.20.2 to 2.3.0 (#1059, #1065, #1108, #1111)
  • Bump pytest from 9.0.3 to 9.1.1 (#1095, #1098)
  • Bump pytest-asyncio from 1.3.0 to 1.4.0 (#1076)
  • Bump ruff from 0.15.11 to 0.16.0 (#1047, #1067, #1075, #1077, #1082, #1093, #1099, #1104, #1105, #1109, #1115, #1121)
  • Bump sphinxcontrib-bibtex from 2.6.5 to 2.7.0 (#1057)
  • Bump types-antlr4-python3-runtime from 4.13.0.20260408 to 4.13.0.20260518 (#1070)
  • Bump types-cachetools from 6.2.0.20260408 to 7.0.0.20260713 (#1052, #1071, #1112)
  • Bump types-jsonschema from 4.26.0.20260408 to 4.26.0.20260518 (#1062, #1069)
  • Bump types-psutil from 7.2.2.20260408 to 7.2.2.20260518 (#1061, #1072)

0.2.0rc2 - 2026-04-23

Added

  • Add CHANGELOG.md and changelog update skill (#1037)
  • Add AI agent guidelines via AGENTS.md and task-specific skills (#937)

Changed

  • Improve StreamFlowExecutor termination protocol (#1000)
  • Improve save protocol and SQLite pragmas (#1036)
  • Refactor load API to drop the context parameter (#1035)
  • Refactor save API to take a Database argument (#1034)
  • Migrate linting from flake8+isort to ruff (#1033)
  • Make dependabot ignore setuptools and wheel (#1013)

Fixed

  • Fix root folder always added to hardware storage (#1040)
  • Correctly skip nested subworkflows (#1019)
  • Fix container mount parsing (#1032)
  • Fix lock releasing logic in scheduler (#1031)

Dependencies

  • Bump aiohttp from 3.13.4 to 3.13.5 (#1015)
  • Bump cachebox from 5.2.2 to 5.2.3 (#1027)
  • Bump cryptography from 46.0.6 to 46.0.7 (#1025)
  • Bump cwl-utils from 0.40 to 0.41 (#1012)
  • Bump lxml from 6.0.2 to 6.1.0 (#1038)
  • Bump plotly from 6.6.0 to 6.7.0 (#1026)

Dev Dependencies

  • Bump cwltool from 3.1.20260315121657 to 3.2.20260413085819 (#1028, #1030)
  • Bump mypy from 1.19.1 to 1.20.2 (#1014, #1029, #1039
  • Bump pytest from 9.0.2 to 9.0.3 (#1021)
  • Bump types-antlr4-python3-runtime (#1017, #1020)
  • Bump types-cachetools from 6.2.0.20260317 to 6.2.0.20260408 (#1022)
  • Bump types-jsonschema from 4.26.0.20260325 to 4.26.0.20260408 (#1016, #1024)
  • Bump types-psutil from 7.2.2.20260130 to 7.2.2.20260408 (#1018, #1023)

0.2.0rc1 - 2026-03-31

Added

  • Implement __hash__ method in the ExecutionLocation (#933)
  • Added restore method (#864)
  • Added type checking with mypy (#938)
  • Add UnrecoverableWorkflowException (#954)
  • Implement persistent shells for improved performance (#960)
  • Support files deduplication in RunCrate (#978)
  • Added --validate flag to streamflow run (#995)
  • Add tag to JobToken (#999)
  • Add Status.RECOVERED (#958)
  • Implemented template validation in the CommandTemplateMap class (#916)
  • Add FaultTolerance documentation (#1004)
  • Validate CWL when condition dependencies (#825)

Changed

  • Refactor DirectedGraph class (#919)
  • Migrate from cachetools to cachebox (#945)
  • Substitute manual quoting with shlex and mslex (#946)
  • Improved InterWorkflowPort (#951)
  • Use official Docker setup action (#956)
  • Improve LoopCombinator (#952)
  • Improve RemotePathMapper performance (#970)
  • Adjust the get_path_processor method (#973)
  • Optimize SQLite performance and concurrency (#974)
  • Improve LocalStreamFlowPath.checksum() performance (#975)
  • Change propagation condition of InterWorkflowPort (#1001)
  • Propagate mounts to wrapped locations in QueueManagerConnector (#947)

Fixed

  • Fix SSHContextFactory retrieval logic (#931)
  • Fix uv pip uninstall in cwl conformance tests script (#934)
  • Fix trivial mypy errors (#943)
  • Fix trivial mypy [var-annotated] errors (#944)
  • Fix StreamFlowPath comparison (#948)
  • Fix optional empty list output in CWL (#959)
  • Close Shell if an error occurs (#962)
  • Fix database relative path (#969)
  • Fix provenance with multiple equal files (#988)
  • Fix empty glob (#989)
  • Fixed recovery of loops (#826)
  • Fixed synchronization of multiple recovery workflows (#822)
  • Fix notify of stacked locations (#1006)
  • Fixed retrieving disk information in SingularityConnector (#894)

Dependencies

  • Bump aiohttp from 3.13.3 to 3.13.4 (#1010)
  • Bump cachetools from 6.2.4 to 7.0.0 (#932, #936)
  • Bump cryptography from 46.0.3 to 46.0.6 (#949, #1009)
  • Bump importlib-metadata from 8.7.1 to 9.0.0 (#996, #997)
  • Bump kubernetes-asyncio from 34.3.3 to 35.0.1 (#953, #967)
  • Bump orjson from 3.11.5 to 3.11.6 (#991)
  • Bump plotly from 6.5.2 to 6.6.0 (#980)
  • Bump psutil from 7.2.1 to 7.2.2 (#935)
  • Bump pygments from 2.19.2 to 2.20.0 (#1011)
  • Bump rdflib from 7.5.0 to 7.6.0 (#993)
  • Bump requests from 2.32.5 to 2.33.0 (#1007)

Dev Dependencies

  • Bump actions/download-artifact from 7 to 8 (#971)
  • Bump actions/upload-artifact from 6 to 7 (#972)
  • Bump bandit from 1.9.3 to 1.9.4 (#966)
  • Bump black from 26.1.0 to 26.3.1 (#987, #990)
  • Bump codecov/codecov-action from 5 to 6 (#1008)
  • Bump codespell from 2.4.1 to 2.4.2 (#986)
  • Bump cwltool from 3.1.20260108082145 to 3.1.20260315121657 (#992)
  • Bump docker/build-push-action from 6 to 7 (#985)
  • Bump docker/login-action from 3 to 4 (#983)
  • Bump docker/setup-buildx-action from 3 to 4 (#984)
  • Bump docker/setup-compose-action from 1 to 2 (#976)
  • Bump docker/setup-docker-action from 4 to 5 (#979)
  • Bump docker/setup-qemu-action from 3 to 4 (#982)
  • Bump helm/kind-action from 1.13.0 to 1.14.0 (#957)
  • Bump isort from 7.0.0 to 8.0.1 (#961, #977)
  • Bump mypy from 1.19.0 to 1.19.1 (#940)
  • Bump pandas-stubs from 2.3.3.251201 to 2.3.3.260113 (#942)
  • Bump pytest-cov from 7.0.0 to 7.1.0 (#1002)
  • Bump types-cachetools from 6.2.0.20251022 to 6.2.0.20260317 (#994)
  • Bump types-jsonschema from 4.25.1.20251009 to 4.26.0.20260325 (#941, #1003, #1005)
  • Bump types-psutil from 7.1.3.20251202 to 7.2.2.20260130 (#939)

0.2.0.dev14 - 2026-01-24

Breaking Changes

  • Drop support for Python 3.9 (#827)

Added

  • Add support for CWL v1.3 (#802)
  • Add support for Python 3.14 (#820)
  • Added checks for CWL feature requirement (#866)
  • Added MatchingBindingFilter (#872)

Changed

  • Refactor processors' logic for async execution (#789)
  • Make CommandOutputProcessor handle Future objs (#790)
  • Refactor TokenProcessor and CommandOutputProcessor classes (#793)
  • Update versions in CI/CD pipeline (#806)
  • Refactor exit code evaluation (#799)
  • Added a check for the chosen deployment in the tests (#814)
  • Refactor handling of File and Directory literals (#811)
  • Remove disable-content-trust in DockerConnector (#856)
  • Remove Lima version from MacOS CI (#859)
  • Remove TransferStep from CWL ExpressionTool (#860)
  • Adjust Python cache in GitHub CI/CD (#865)
  • Update MacOS X CI/CD to version 15 (#819)
  • Reduced overhead of the transfer_data method (#883)
  • Revert sphinx to v8.1.3 (#888)
  • Migrate to uv package manager (#889)
  • Respect available CPUs in LocalConnector (#896)

Fixed

  • Fixed issues related to default values in CWL (#808)
  • Fixed recoverable attribute in the ListToken and ObjectToken classes (#752)
  • Fixed issue on multiple exampleOfWork on RunCrate provenance (#841)
  • Fixed issue in DependencyResolver (#839)
  • Fix Lima version to v1.2.2 (#850)
  • Fixed loadContents in the collector step (#846)
  • Fix race condition with output unique paths (#863)
  • Fix minor issues on recovery (#862)
  • Fixed CWLObjectCommandOutputProcessor to handle arrays of records (#858)
  • Fixed eval method of the DependencyResolver (#867)
  • Handle location storage retrieval errors (#871)
  • Fix StreamFlowContext (#880)
  • Fixed directory registration (#881)
  • Fixed type checking in CWLCommandTokenProcessor (#895)
  • Fix issues in the ValueFromTransformer and LoopValueFromTransformer classes (#882)
  • Fixed StreamFlowContext creation (#901)
  • Fixed transfer of hardlink (#891)
  • Fixed ExpressionTool outputs processing (#902)
  • Fix cwl.output.json behaviour (#851)
  • Fix ext show command (#927)

Dependencies

  • Bump aiohttp from 3.12.15 to 3.13.2 (#821, #832, #840)
  • Bump aiosqlite from 0.21.0 to 0.22.1 (#898, #906)
  • Bump asyncssh from 2.21.0 to 2.22.0 (#815, #918)
  • Bump bcrypt from 4.3.0 to 5.0.0 (#812)
  • Bump cachetools from 6.1.0 to 6.2.3 (#792, #828, #861, #897)
  • Bump cwl-utils from 0.39 to 0.40 (#803)
  • Bump filelock from 3.20.0 to 3.20.3 (#917)
  • Bump importlib-metadata from 8.7.0 to 8.7.1 (#907)
  • Bump jsonschema from 4.25.0 to 4.26.0 (#786, #923)
  • Bump kaleido from 1.0.0 to 1.2.0 (#804, #848)
  • Bump kubernetes-asyncio from 32.3.2 to 34.3.3 (#783, #915)
  • Bump pandas from 2.3.1 to 2.3.3 (#788, #816)
  • Bump plotly from 6.2.0 to 6.5.2 (#784, #818, #847, #869, #905, #920)
  • Bump psutil from 7.0.0 to 7.2.1 (#807, #833, #837, #844, #914)
  • Bump rdflib from 7.1.4 to 7.5.0 (#809, #842, #877)
  • Bump referencing from 0.36.2 to 0.37.0 (#830)
  • Bump typing-extensions from 4.14.1 to 4.15.0 (#791)
  • Bump urllib3 from 2.6.1 to 2.6.3 (#921)

Dev Dependencies

  • Bump actions/checkout from 4 to 6 (#782, #874)
  • Bump actions/download-artifact from 4 to 7 (#780, #836, #899)
  • Bump actions/setup-node from 4 to 6 (#796, #831)
  • Bump actions/setup-python from 5 to 6 (#795)
  • Bump actions/upload-artifact from 4 to 6 (#838, #900)
  • Bump bandit from 1.8.6 to 1.9.3 (#868, #875, #924)
  • Bump black from 25.1.0 to 26.1.0 (#810, #854, #886, #925)
  • Bump cwltest from 2.6.20250314152537 to 2.6.20251216093331 (#785, #909)
  • Bump cwltool from 3.1.20250715140722 to 3.1.20260108082145 (#813, #843, #911)
  • Bump flake8-bugbear from 24.12.12 to 25.11.29 (#834, #879)
  • Bump github/codeql-action from 3 to 4 (#823)
  • Bump helm/kind-action from 1.12.0 to 1.13.0 (#845)
  • Bump isort from 6.0.1 to 7.0.0 (#817, #829)
  • Bump mukunku/tag-exists-action from 1.6.0 to 1.7.0 (#890)
  • Bump pytest from 8.4.1 to 9.0.2 (#797, #887)
  • Bump pytest-asyncio from 1.1.0 to 1.3.0 (#805, #855)
  • Bump pytest-cov from 6.2.1 to 7.0.0 (#801)
  • Bump pyupgrade from 3.20.0 to 3.21.2 (#824, #852, #870)
  • Bump sphinx-jsonschema from 1.19.1 to 1.19.2 (#878)
  • Bump sphinx-rtd-theme from 3.0.2 to 3.1.0 (#928)
  • Bump sphinx-llms-txt from 0.3.1 to 0.7.1 (#929)

0.2.0.dev13 - 2025-08-07

Added

  • Added ROLLBACK status (#697)
  • Added recoverable attribute to the Token object in the FailureManager (#699)
  • Implemented fault tolerance based on Database module (#169)
  • Check if bindings exist in the workflow (#708)
  • Added llms.txt to StreamFlow documentation (#754)

Changed

  • Refactor private functions (#684)
  • Always wait for DataLocation to be available (#690)
  • Refactor unit tests (#691)
  • Improved WorkflowBuilder (#692)
  • Improved is_available method of the FileToken class (#696)
  • Refactored TransferStep (#698)
  • Refactored the no-cycles check for stacked deployment definitions (#705)
  • Improved handling of File literals (#693)
  • Improve JSONSchema specification (#745)
  • Add a check to verify object instantiation in persistence tests (#741)
  • Refactor the invocation of FailureManager within the Step (#753)
  • Improved token processing logic in CWLObjectTokenProcessor (#758)
  • Make Scheduler helper methods raise exceptions (#759)
  • Improve StreamFlow type system (#762)
  • Make StreamFlowContext constructor self-contained (#767)
  • Improved Combinator persistence tests (#764)
  • Improved provenance of the task executions (#772)
  • Remove support for nested Token objects (#777)

Fixed

  • Fix requirement propagation in CWLTranslator (#688)
  • Fix concurrent output retrieval (#689)
  • Fixed cleanup in unit tests (#695)
  • Fixed invalidate_location method (#694)
  • Fixed CloneTransformer persistence (#719)
  • Fix InitialWorkDirRequirement for nested dirs (#735)
  • Capture TimeoutError when retrieving disk usage (#737)
  • Fixed persistence of CWLCommandOutputProcessor classes (#740)
  • Fixed preparation of workdir for list and object inputs (#732)
  • Fixed tag sorting in the GatherStep (#761)
  • Fixed CWLMapCommandOutputProcessor persistence (#765)
  • Fixed CWLCommand persistence (#771)
  • Fixed the transferBufferSize parameter in the FutureConnector (#775)
  • Fixed subworkflows case in the RunCrateProvenanceManager (#774)
  • Fixed DefaultRetagTransformer within scatter (#742)

Removed

  • Removed comparison operators from Hardware class (#711)
  • Remove the StreamWrapperContextManager class (#760)

Dependencies

  • Bump aiohttp from 3.11.13 to 3.12.15 (#700, #712, #715, #724, #729, #731, #733, #738, #744, #757, #773)
  • Bump asyncssh from 2.20.0 to 2.21.0 (#718)
  • Bump bcrypt from 4.2.1 to 4.3.0 (#683)
  • Bump cachetools from 5.5.2 to 6.1.0 (#722, #746)
  • Bump cwl-utils from 0.37 to 0.39 (#726, #770)
  • Bump importlib-metadata from 8.6.1 to 8.7.0 (#716)
  • Bump jinja2 from 3.1.5 to 3.1.6 (#687)
  • Bump jsonschema from 4.23.0 to 4.25.0 (#723, #766, #768)
  • Bump kaleido from 0.4.2 to 1.0.0 (#749)
  • Bump kubernetes-asyncio from 32.0.0 to 32.3.2 (#707, #717)
  • Bump pandas from 2.2.3 to 2.3.1 (#734, #756)
  • Bump plotly from 6.0.0 to 6.2.0 (#702, #727, #750)
  • Bump rdflib from 7.1.3 to 7.1.4 (#706)

Dev Dependencies

  • Bump bandit from 1.8.3 to 1.8.6 (#747, #755)
  • Bump cwltest from 2.5.20241122133319 to 2.6.20250314152537 (#701)
  • Bump cwltool from 3.1.20250110105449 to 3.1.20250715140722 (#769)
  • Bump pytest from 8.3.4 to 8.4.1 (#685, #730, #748)
  • Bump pytest-asyncio from 0.25.3 to 1.1.0 (#704, #743, #763)
  • Bump pytest-cov from 6.0.0 to 6.2.1 (#709, #713, #739)
  • Bump pytest-xdist from 3.6.1 to 3.8.0 (#725, #751)
  • Bump pyupgrade from 3.19.1 to 3.20.0 (#721)
  • Bump sphinx from 8.2.1 to 8.2.3 (#686)
  • Bump sphinx-llms-txt from 0.3.0 to 0.3.1 (#776)

0.2.0.dev12 - 2025-02-28

Breaking Changes

  • Drop support for Python 3.8 (#561)

Added

  • Added NoContainerCWLDockerTranslator (#453)
  • Check job status when it terminates (#452)
  • Added get_stream_writer method to Connector (#477)
  • Implementing stacked locations scheduling (#480)
  • Created CWLWorkflow class (#504)
  • Enable StreamFlow JSON Schema dump (#543)
  • Add support for Python 3.13 (#539)
  • Support stacked storage in scheduling (#552)
  • Enable multi-workflow reports (#581)
  • Add local field to ExecutionLocation (#593)
  • Add isort to the lint pipeline (#621)
  • Added timeout to SSHConnector (#644)

Changed

  • Refactor command tokens (#469)
  • Optimize scheduling logic (#474)
  • Improve performance of local copies (#476)
  • Update Apptainer version in CI pipeline (#482)
  • Make ContainerConnector extend ConnectorWrapper (#481)
  • Removed duplicate package import (#492)
  • Moved Graph reference from CWLTokenProcessor to CWLWorkflow (#516)
  • Changed propagation of failure termination (#373)
  • Improved description about location storage in the Hardware class (#423)
  • Temporarily disable MacOS X GitHub Actions (#459)
  • Improved job propagation after scheduling (#580)
  • Simplify DataManager logic for local transfers (#582)
  • Simplify data movement logic (#583)
  • Improve LocalConnector performance (#585)
  • Refactor DataManager class and fix local copy (#587)
  • Stop loading contents on Directory objects (#594)
  • Improve logging of copy methods (#599)
  • Modernize pytest suite (#600)
  • Improve mount handling on SingularityConnector (#601)
  • Refactored unit tests (#615)
  • Improve MacOS CI performance (#609)
  • Propagate DataLocation relations to stacked envs (#608)
  • Migrate CWL parsing from cwltool to cwl-utils (#58)
  • Refactored remotepath module (#618)
  • Update coverage command and add badge (#625)
  • Improved ssh retry mechanism (#501)
  • Optimize remote commands through StreamFlowPath (#626)
  • Modernize base Docker image and Helm version (#655)
  • Migrate Docker container to Debian (#658)
  • Refactor SSHConnector class (#661)
  • Improve CWL parsing performance (#660)
  • Remove useless LocalConnector deps (#665)
  • Uniform StreamFlow output format to CWL ecosystem (#667)
  • Improved remote file creation (#679)
  • Update Linux GitHub runners to 24.04 (#680)
  • Update PostgreSQL plugin documentation (#681)

Fixed

  • Fix PyPI release pipeline (#450)
  • Moved the _retrieve_output calls inside try-except clauses (#451)
  • Fix workdir handling in SlurmConnector (#456)
  • Re-introduced log level option in DockerComposeConnector (#457)
  • Fixed StreamFlow name (#458)
  • Fixed prefix of input union types (#455)
  • Fixed record field name in CWLObjectTokenProcessor (#460)
  • Fixed workdir of wrapped deployments (#461)
  • Fix remote write (#471)
  • Fix SingularityConnector environment variables (#470)
  • Fixed output step when it is an empty list (#472)
  • Fix input directory path (#475)
  • Fixed CONTENT_LIMIT checks (#491)
  • Added explicit return in _copy_remote_to_remote (#515)
  • Fixed minor bugs (#535)
  • Fix MacOS X CI on GitHub Actions (#547)
  • Fixed job input for the CWL InitialWorkDirRequirement (#536)
  • Fix follow_symlink function return value (#542)
  • Fix ValueFromTransformer without input deps (#553)
  • Handle lists in CWL glob field (#555)
  • Fixed optional CWL parameter (#466)
  • Fix report feature in case of empty workflow (#568)
  • Improved SSHConnector error message (#574)
  • Fix requirement eval in CWL WorkflowStep (#578)
  • Fixed InputInjectorStep in remote location data case (#566)
  • Fix concurrency in DefaultScheduler (#584)
  • Fixed workdir inheritance in wrapped deployments (#588)
  • Added check to mkdir return code (#595)
  • Fix PBSConnector behaviour (#606)
  • Fixed Hardware operation methods (#616)
  • Fixed some CWL translator issues (#628)
  • Fixed remote ports (#643)
  • Fix remote files handling (#648)
  • Fixed ChannelOpenError handling on the SSHConnector (#664)
  • Fixed tag hierarchy check (#670)
  • Fix ANTLR4 script generator (#674)
  • Handle single-element output lists in CWL (#672)
  • Fix CWL pickValue with all_non_null option (#677)

Removed

  • Remove unused code (#673)

Dependencies

Dev Dependencies

0.2.0.dev11 - 2024-05-07

Added

  • Add support for Python 3.12 (#256)
  • Added possibility to load ports and steps in a different workflow (#274)
  • Add warning messages for default connector params (#294)
  • Add optional param to composite processors (#295)
  • Added missing task creation in some gathers (#327)
  • Added StreamWrapper documentation (#329)
  • Added new queries to the Database (#356)
  • Added retries and retryDelay parameters in the SSHConnector configuration (#361)
  • Support for nested Location objects (#383)

Changed

  • Update Node.js version in CI pipeline (#286)
  • Update CWL v1.2 conformance tests (#288)
  • Refactor conftest.py (#287)
  • Removed deprecated pkg_resources API (#291)
  • Replace cgi and logging.warn with new APIs (#292)
  • Improve test_download logic (#293)
  • Improved error message of batch systems when job fails (#332)
  • Add no regression test for Sphinx documentation (#337)
  • Reduce size of database metadata (#338)
  • Move CWL 1.2 conformance to main branch (#346)
  • Improved scatter/gather performance (#348)
  • Fix Codecov GitHub action (#374)
  • Fix CWL Docker Requirement config example (#375)
  • Remove SSH deployment from MacOS CI (#376)
  • Fix Codecov GitHub Action (#379)
  • Fix Codecov GitHub Action (#380)
  • Add codecov.yml configuration file (#382)
  • Add __slots__ attribute to Token objects (#385)
  • Remove __eq__ and __hash__ from Location (#389)
  • Refactor the Location class hierarchy (#390)
  • Update MacOS X CI/CD pipeline (#408)
  • Removed scheduling_policy attribute in Target class (#421)
  • Removed scheduling_groups attribute (#422)
  • Changed retrieval of location hardware in the SSHConnector class (#424)
  • Refactor test_scheduler (#425)

Fixed

  • Fixed minor issue with port-targets documentation (#276)
  • Fixed Config persistence (#289)
  • Fixed loop termination combinator persistence (#290)
  • Fix copy remote-to-remote using ConnectorWrapper (#301)
  • Adjusted local-to-remote copy with folders (#308)
  • Fixed remote-to-local copy of nested directories (#309)
  • Fixed template environment variables (#310)
  • Fixed SSHClientConnection closing (#311)
  • Fixed incomplete stream reading (#314)
  • Fix valueFrom type checking behaviour (#323)
  • Fixed data location invalidation (#324)
  • Fixed job name (#325)
  • Fixed typo errors (#326)
  • Fixed symbolic link remote input data error (#328)
  • Fixed CWLConditionalStep persistence (#330)
  • Fixed provenance issues (#331)
  • Fixed some bugs (#333)
  • Added check after the creation data (#336)
  • Fixed get method of RemotePathMapper (#343)
  • Fixed provenance for the size tokens (#370)
  • Fixed asyncio.Event synchronizations when an exception occurs (#395)
  • Fixed input ports of DotProductSizeTransformer (#396)
  • Fixed wrapped connector calls for QueueManagerConnector classes (#403)
  • Fix ListMergeCombinator port names (#407)
  • Fixed glob on symbolic link path (#417)
  • Fixed ValueFromTransformer (#446)
  • Fixed scheduler timeout in Python<3.11 (#449)

Dependencies

Dev Dependencies

0.2.0.dev10 - 2023-10-10

Added

  • Added parameter to choose the deployments to use in pytests (#232)

Fixed

  • Fix _get_existing_parent behaviour (#253)

Dependencies

  • Bump aiohttp from 3.8.5 to 3.8.6 (#255)
  • Bump asyncssh from 2.13.2 to 2.14.0 (#250)
  • Bump jsonschema from 4.19.0 to 4.19.1 (#244)
  • Bump kubernetes-asyncio from 25.11.0 to 28.2.0 (#247, #249, #254)
  • Bump pandas from 2.1.0 to 2.1.1 (#245)
  • Bump plotly from 5.16.1 to 5.17.0 (#240)

Dev Dependencies

  • Bump black from 23.7.0 to 23.9.1 (#233)
  • Bump codespell from 2.2.5 to 2.2.6 (#252)
  • Bump docker/build-push-action from 4 to 5 (#237)
  • Bump docker/login-action from 2 to 3 (#235)
  • Bump docker/setup-buildx-action from 2 to 3 (#236)
  • Bump docker/setup-qemu-action from 2 to 3 (#234)
  • Bump flake8-bugbear from 23.7.10 to 23.9.16 (#239)
  • Bump mukunku/tag-exists-action from 1.3.0 to 1.4.0 (#242)
  • Bump pyupgrade from 3.10.1 to 3.14.0 (#246, #248, #251)
  • Bump sphinx from 7.2.5 to 7.2.6 (#238)

0.2.0.dev9 - 2023-09-10

Added

  • Add configurable time option in HPC connectors (#227)

Changed

  • Avoid opening too many SSH sessions (#229)
  • Update CWL v1.2 conformance tests (#231)

Fixed

  • Fix Python dependencies conflict in Mac OS CI (#214)
  • Connector and CWL Runner fixes (#216)
  • Fix template_map error when service has no file (#223)
  • Fix scheduler behaviour when retry_delay is set (#224)
  • Fix potential race condition on sqlite row_factory (#225)
  • Fix Flux Docker version to 0.28.0 (#230)

Dependencies

  • Bump antlr4-python3-runtime from 4.13.0 to 4.13.1 (#222)
  • Bump cwltool from 3.1.20230719185429 to 3.1.20230906142556 (#226)
  • Bump cwl-utils from 0.28 to 0.29 (#220)
  • Bump pandas from 2.0.3 to 2.1.0 (#217)

Dev Dependencies

  • Bump actions/checkout from 3 to 4 (#221)
  • Bump pytest from 7.4.0 to 7.4.2 (#219, #228)
  • Bump sphinx from 7.2.3 to 7.2.5 (#215, #218)

0.2.0.dev8 - 2023-08-28

Added

  • Enable distributed inheritance for plugins (#208)

Changed

  • Updated MacOS CI in GitHub Actions (#204)
  • Change jsonschema loader (#205)
  • Adjust JSONSchema ids (#211)
  • Update docs to v0.2.0 (#213)

Fixed

  • Fix Flux and Mac OS CI steps (#209)

Dependencies

  • Bump jsonschema from 4.18.6 to 4.19.0 (#197)
  • Bump kubernetes-asyncio from 24.2.3 to 25.11.0 (#206)
  • Bump plotly from 5.15.0 to 5.16.1 (#199, #201)

Dev Dependencies

  • Bump cwltest from 2.3.20230527113600 to 2.3.20230825125225 (#212)
  • Bump mukunku/tag-exists-action from 1.2.0 to 1.3.0 (#198)
  • Bump sphinx from 7.1.2 to 7.2.3 (#203, #210)
  • Bump sphinx-rtd-theme from 1.2.2 to 1.3.0 (#207)

0.2.0.dev7 - 2023-08-06

Added

  • Added type definitions to ext CLI (#180)

Changed

  • Updated CWL v1.2 conformance tests (#168)
  • Migrated JSONSchema to Draft 2019-09 (#185)
  • Moved black configuration to pyproject.toml (#196)

Fixed

  • Fixed type checking in conftest (#195)

Dependencies

  • Bump aiohttp from 3.8.4 to 3.8.5 (#187)
  • Bump asyncssh from 2.13.1 to 2.13.2 (#170)
  • Bump cwltool from 3.1.20230601100705 to 3.1.20230719185429 (#172, #189)
  • Bump importlib-metadata from 6.6.0 to 6.8.0 (#167, #177)
  • Bump jsonschema from 4.17.3 to 4.18.6 (#175, #184, #186, #194)
  • Bump pandas from 2.0.2 to 2.0.3 (#173)

Dev Dependencies

  • Bump black from 23.3.0 to 23.7.0 (#178)
  • Bump codespell from 2.2.4 to 2.2.5 (#165)
  • Bump flake8-bugbear from 23.6.5 to 23.7.10 (#179)
  • Bump pytest from 7.3.2 to 7.4.0 (#171)
  • Bump pytest-asyncio from 0.21.0 to 0.21.1 (#182)
  • Bump pyupgrade from 3.6.0 to 3.10.1 (#166, #174, #176, #191)
  • Bump sphinx from 7.0.1 to 7.1.2 (#188, #190, #192)

0.2.0.dev6 - 2023-06-15

Added

  • Added streamflow ext subcommand (#145)
  • Added no regression tests for provenance (#150)
  • Adding explicit options to queue connectors (#151)
  • Added curl flags in Dockerfile to retry server connection when it fails (#158)
  • Handling allOf directives in config schemas (#164)

Fixed

  • Fix additional files in RunCrate provenance (#136)
  • Fix list and report subcommands (#144)

Dependencies

  • Bump antlr4-python3-runtime from 4.12.0 to 4.13.0 (#147)
  • Bump cachetools from 5.3.0 to 5.3.1 (#152)
  • Bump cwltool from 3.1.20230425144158 to 3.1.20230601100705 (#140, #154, #156)
  • Bump cwl-utils from 0.25 to 0.28 (#138, #142, #157)
  • Bump pandas from 2.0.1 to 2.0.2 (#153)
  • Bump plotly from 5.14.1 to 5.15.0 (#161)

Dev Dependencies

  • Bump cwltest from 2.3.20230108193615 to 2.3.20230527113600 (#155)
  • Bump flake8-bugbear from 23.3.23 to 23.6.5 (#137, #159)
  • Bump pytest from 7.3.1 to 7.3.2 (#162)
  • Bump pytest-cov from 4.0.0 to 4.1.0 (#149)
  • Bump pytest-xdist from 3.2.1 to 3.3.1 (#141, #143)
  • Bump pyupgrade from 3.3.2 to 3.6.0 (#135, #163)
  • Bump sphinx from 7.0.0 to 7.0.1 (#139)
  • Bump sphinx-rtd-theme from 1.2.0 to 1.2.2 (#148, #160)

0.2.0.dev5 - 2023-05-06

Added

  • Added package and version to plugin description (#134)

Changed

  • Improved StreamFlow provenance support (#122)
  • Improved CLI for plugin inspection (#130)

Fixed

  • Fix InitialWorkDirRequirement with subdirs (#118)

Dependencies

  • Bump aiosqlite from 0.18.0 to 0.19.0 (#115)
  • Bump cwltool from 3.1.20230325110543 to 3.1.20230425144158 (#129)
  • Bump cwl-utils from 0.23 to 0.25 (#114, #133)
  • Bump importlib-metadata from 6.3.0 to 6.6.0 (#119, #124)
  • Bump kubernetes-asyncio from 24.2.2 to 24.2.3 (#132)
  • Bump pandas from 2.0.0 to 2.0.1 (#125)
  • Bump psutil from 5.9.4 to 5.9.5 (#120)

Dev Dependencies

  • Bump pytest from 7.3.0 to 7.3.1 (#117)
  • Bump pyupgrade from 3.3.1 to 3.3.2 (#128)
  • Bump sphinx from 6.1.3 to 7.0.0 (#123, #126, #131)

0.1.6 - 2023-04-20

Changed

  • Updated JS imports from cwltool (#121)

0.2.0.dev4 - 2023-04-14

Added

  • Added explicit version check for DockerCompose (#96)
  • Added flux framework connector (#107)
  • Added configurable DockerRequirement conversion (#111)

Fixed

  • Fix StreamFlow MPI example (#95)
  • Updated and fixed StreamFlow Dockerfile (#97)
  • Fixed Docker and Helm features (#109)

Dependencies

  • Bump cwltool from 3.1.20230302145532 to 3.1.20230325110543 (#103)
  • Bump importlib-metadata from 6.0.0 to 6.3.0 (#100, #112)
  • Bump pandas from 1.5.2 to 2.0.0 (#92, #108)
  • Bump plotly from 5.11.0 to 5.14.1 (#93, #106, #110)
  • Bump rdflib from 6.2.0 to 6.3.2 (#104)

Dev Dependencies

  • Bump black from 23.1.0 to 23.3.0 (#105)
  • Bump flake8-bugbear from 23.2.13 to 23.3.23 (#91, #102)
  • Bump pytest from 7.2.2 to 7.3.0 (#113)
  • Bump pytest-asyncio from 0.20.3 to 0.21.0 (#101)
  • Bump pytest-xdist from 3.2.0 to 3.2.1 (#90)

0.2.0.dev3 - 2023-03-13

Added

  • Colored output for Streamflow Run (#17)
  • Added regression test scheduler (#39)
  • Workflow Run RO-Crate support (#45)
  • Added extension points to StreamFlowPlugin (#46)
  • Added nested files to main entity (#53)
  • Added no regression tests for persistence (#56)
  • Added persistence of Command class structure (#74)
  • Added support for nested connectors (#86)

Changed

  • Changed test pipeline (#33)
  • Prepend isEnabledFor to logger calls (#37)
  • Code reformat with pyupgrade (#40)
  • Revised setup.py classifiers (#49)
  • Better job naming strategy for CWL (#51)
  • Use JSON format for PBS output (#63)
  • Moving aiotarstream to the deployment module (#64)
  • Moving some logic from Translator to utils (#66)
  • Stricter type checking for CWL outputs (#70)
  • Migrate from setup.py to pyproject.toml (#89)

Fixed

  • Fix scheduler validation path (#50)
  • Fixed cwl-runner entrypoint with no workflows entry (#52)
  • Avoid Sqlite race conditions during tests (#57)
  • Register job directories after creating them (#65)
  • Fixed MacOS compatibility (#77)
  • Fix CWL map processors behaviour (#81)

Dependencies

  • Bump aiohttp from 3.8.3 to 3.8.4 (#72)
  • Bump antlr4-python3-runtime from 4.10 to 4.12.0 (#23, #76)
  • Bump asyncssh from 2.13.0 to 2.13.1 (#75)
  • Bump cachetools from 5.2.0 to 5.3.0 (#35, #47)
  • Bump cwltool from 3.1.20221008225030 to 3.1.20230302145532 (#25, #55, #61, #69, #71, #79)
  • Bump cwl-utils from 0.20 to 0.23 (#26, #54, #62)
  • Bump importlib-metadata from 5.2.0 to 6.0.0 (#30)
  • Bump jsonref from 1.0.1 to 1.1.0 (#43)
  • Bump yattag from 1.15.0 to 1.15.1 (#78)

Dev Dependencies

  • Bump actions/checkout from 2 to 3 (#29)
  • Bump actions/setup-node from 2 to 3 (#22)
  • Bump actions/setup-python from 2 to 4 (#20)
  • Bump actions/upload-artifact from 2 to 3 (#36)
  • Bump bandit from 1.7.4 to 1.7.5 (#85)
  • Bump black from 22.12.0 to 23.1.0 (#60)
  • Bump codespell from 2.2.2 to 2.2.4 (#84)
  • Bump docker/build-push-action from 2 to 4 (#21, #59)
  • Bump docker/login-action from 1 to 2 (#19)
  • Bump docker/setup-buildx-action from 1 to 2 (#24)
  • Bump docker/setup-qemu-action from 1 to 2 (#28)
  • Bump flake8-bugbear from 22.12.6 to 23.2.13 (#44, #48, #73)
  • Bump mukunku/tag-exists-action from 1.0.0 to 1.2.0 (#27)
  • Bump pytest from 7.2.0 to 7.2.2 (#41, #80)
  • Bump pytest-xdist from 3.1.0 to 3.2.0 (#67)
  • Bump sphinx from 6.0.0 to 6.1.3 (#31, #34, #38)
  • Bump sphinx-rtd-theme from 1.1.1 to 1.2.0 (#68)

0.2.0.dev2 - 2022-10-09

Added

  • Added services for SSH and QueueManagerConnector connectors (b7203d8)
  • Added CITATION file (2baa905)

Changed

  • Updated CWL toolchain compatibility (7041950)
  • Updated GitHub pipelines (3a5de2a)
  • Improved performance by removing remote commands (5c5b457)
  • Moved helper file to inline encoded command (3814f1d)
  • Updated Loop support (ed1056b)

Fixed

  • Fixed ListMergeCombinator behaviour (0695732)

Dependencies

  • Added psutil to requirements (f3dbba7)

0.2.0.dev1 - 2022-08-29

Added

  • Added port targets (6712a6d)
  • Added persistence loading logic (2240ef5)
  • Extension points implementation (7d3a937)
  • Enriched persistence layer (49c727e)
  • Added cwltool:Loop support (8a3adf7)
  • Implemented cross-connector streaming copy (0e5221b)
  • Created the aiotarstream library for async tar (43fab68)
  • Added aiotar library for remote copy (7647e5f)

Changed

  • Moved loop_source and loop_when to camelCase (1a509f2)
  • Modified valueFrom behaviour in loops (55b281a)
  • Improved SSH connector performances (04751e8)
  • Added file-system traversal to find existing locations (b81bf6a)
  • Changed DataLocation creation strategy (f09df93)
  • Modified GitHub Actions to run on success (39ef379)

Fixed

  • Fixed multiple errors on remote executions (ff5b641)
  • Fixed remote-to-remote copy behaviour (48bfd13)
  • Fix directory transfer bug and improve connector supports (#13)

0.2.0.dev0 - 2022-04-24

Changed

  • Heavy refactor of workflow management logic (d4b7b24)
  • Step class refactor (eda8963)
  • Moved ports outside steps (980c65e)
  • Moved token processors to steps (78c0cf5)
  • Substituted resource term with location (191b5b5)
  • Substituted model term with deployment (e939114)
  • Moved most log messages to DEBUG level (d0f5d60)
  • Updated CWL-v1.2 conformance test commit (8ad4db5)
  • Removed token deepcopy in Port.put method (aabdb70)

Fixed

  • Fixed the sorting of parameters in commands (a0a1aa0)
  • Fixed loadContents in CWL (ff5b9e6)
  • Fixed bug in checking available resources (e3b7086)
  • Fixed error in input token grouping (23e42fe)

Removed

  • Removed Helm2Connector from source (53284af)

Dependencies

  • Updated ANTLR version to 4.10 (3de01f2)
  • Updated kubernetes-asyncio to version 22.6.3 (4ffb543)

0.1.5 - 2022-04-15

Dependencies

  • Constrained ANTLR version to <4.10 (aee4851)

0.1.4 - 2022-04-02

Added

  • Added support for remote CWL definitions (47324bf)

Changed

  • Added Workflows Community info (9643692)

0.1.3 - 2022-02-23

Fixed

  • Solved Python >= 3.10 compatibility issue (252fa97)
  • Solved SSH compatibility issue with Python >= 3.9 (1c0a872)

0.1.2 - 2021-12-29

Fixed

  • Improved support for Windows platform (1a6b335)
  • Fixed Helm Connector, apsw and Windows execution (03c79d7)
  • Fixed Helm Connector checks (d4ff604)
  • Fixed docker GitHub Actions (4f7140b)

0.1.1 - 2021-12-25

Added

  • Added --streamflow-file option to cwl-runner (b9f4108)
  • Added Sphinx documentation (76f2ea7)

Changed

  • Changed contributing instructions in README.md (ba0be7e)
  • Better organised CWL badges (1ba5030)

Fixed

  • Fixed bug with SSH resources retrieving (dd0789b)
  • Fixed Release GitHub action (0470f8a)
  • Minor pre-release fixes (d3a2752)

0.1.0 - 2021-12-19

Changed

  • Switched to GitHub Actions for CI/CD (8f05da2)
  • Added CWL conformance badges (85e5209)
  • Minor fixes to dependencies and Docker (cab297a)
  • Replace Travis by GH Actions in the readme (#9)

0.0.34 - 2021-11-15

Added

  • Added CWL conformance tests (0276b3c)
  • Added step-to-step bindings to the StreamFlow file (a5565ed)
  • Added SQLite database and report class (e57e643)

Changed

  • Optimised dependencies in nested workflows (52232c9)
  • Improved performances of StreamFlow runtime (7421380)

Fixed

  • Fixed nested scatter patterns (a3c7f0b)
  • Fixed secondary files primary location (aea75a9)
  • Fixed Docker and Travis build (7d2a3a4)

0.0.33 - 2021-08-12

Fixed

  • Fixes to improve CWL compatibility (af4a0c2)

0.0.32 - 2021-07-05

Added

  • Added Local connector (67ca093)
  • Added Hardware requirements to jobs (c0dd8a6)
  • Added CWL format checking support (7602e52)

Changed

  • Updated SSH connector to handle multi-node models (9438dc3)
  • Replaced pipenv toolchain (e7474d1)

Fixed

  • Fixed CartesianProductInputCombinator behaviour (5a96a25)

0.0.31 - 2021-05-29

Fixed

  • Fixed remote-to-remote copy for QueueManagerConnector

0.0.30 - 2021-05-27

Fixed

  • Fixed WebSocket API client in Helm connectors

0.0.29 - 2021-05-20

Added

  • Added token tag to avoid out-of-order errors (dcddcb3)

Changed

  • Optimised Docker connector for bind mounts (9b7e295)

Fixed

  • Minor fixes to conditional behaviour (5f73cf3)

0.0.28 - 2021-05-04

Added

  • Added support for CWL conditional steps (deebe5a)
  • Added secondary files propagation (9e31d58)

Changed

  • Changed tar format to GNU (ff6277d)
  • Removed compression in tar transfers (6f4dd95)

0.0.27 - 2021-04-17

Changed

  • Improved data transfer logic (313ba9e)

0.0.26 - 2021-04-09

Fixed

  • Fixed SLURM job submission command
  • Fixed PBS job status checking

0.0.25 - 2021-04-05

Added

  • Added PBS and SLURM connectors (b75e6e2)
  • Added caching for available resources (2336e25)

Changed

  • Improved file transfer performances (cd54aec)
  • Control number of sessions per SSH connection (e160014)
  • Improved Singularity connector performances (18c1915)
  • Refactored connectors code (808af6d)

Fixed

  • Fixed pip search command (51ad998)

0.0.24 - 2021-03-22

Added

  • Added Singularity Connector (b4735a2)

Changed

  • Improved remote execution performances (e39b303)

0.0.23 - 2021-01-03

Added

  • Added Fault Tolerance to StreamFlow (0d1cbf8)
  • Added SSH and Slurm connectors (fbd2c80)
  • Added STMC support (c199ac8)
  • Added support for stream redirection in connectors (1e45a65)

Changed

  • Created custom workflow management system (6331926)
  • Split StreamFlow schema into multiple files (6388746)
  • Improved termination of workflow (60b7d92)

0.0.21 - 2020-06-17

Changed

  • Updated README.md to include PYTHONPATH inclusion (89e6958)

Fixed

  • Fixed wrong file creation in process.py (f975dfd)

0.0.20 - 2020-04-06

Changed

  • Updates to the Continuous Integration Pipeline (300ae29)

0.0.13 - 2020-03-19

Added

  • Added Kubernetes inCluster execution for Helm templates (6822935)
  • Added munipack workflow to StreamFlow examples (325e8ed)

Changed

  • Updated README.md to include Docker usage description (caf261b)

0.0.12 - 2020-03-13

Breaking Changes

  • Added support for Helm 3 environments (breaking Helm 2 compatibility) (922fde2)

Added

  • Initial StreamFlow implementation
  • Added Travis CI integration (f91d659)
  • Added support for in-place dependency reading (526ec7d)
  • Added support for remote files not present on local machine (21a1d2f)
  • Added support for directories generated by ExpressionTool (7430d03)
  • Added flatten_list function to avoid nested list outputs (2357b7c)

Fixed

  • Fixed remote_fs_access error for local jobs (1954d83)
  • Fixed bad behaviour with local job execution (729b2d9)
  • Fixed remote-to-remote data movement on Helm connector (0967668)
  • Patched WebSocket client update() function to handle binary format (ba48f19)