Add compatibility for TES 1.1.0

[stxue1]

py-tes on pypi currently has 2 prereleases tagged as 1.1.0. It seems like some of the requests that py-tes calls aren't working though. With py-tes==1.1.0rc2 and Funnel 0.11.0, attempting to run toil-wdl-runner with the TES batchsystem plugin results in TES seeing an unexpected argument:

Traceback (most recent call last):
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 115, in unmarshal
    output = o(**r)
             ^^^^^^
TypeError: Input.__init__() got an unexpected keyword argument 'streamable'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/heaucques/Documents/toil/venv3.12/bin/toil-wdl-runner", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/wdl/wdltoil.py", line 141, in decorated
    return decoratee(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/wdl/wdltoil.py", line 3156, in main
    output_bindings = toil.start(root_job)
                      ^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/common.py", line 930, in start
    return self._runMainLoop(rootJobDescription)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/src/toil/common.py", line 1414, in _runMainLoop
    jobCache=self._jobCache).run()
                             ^^^^^
  File "/home/heaucques/Documents/toil/src/toil/leader.py", line 257, in run
    self.innerLoop()
  File "/home/heaucques/Documents/toil/src/toil/leader.py", line 782, in innerLoop
    self.checkForDeadlocks()
  File "/home/heaucques/Documents/toil/src/toil/leader.py", line 819, in checkForDeadlocks
    totalRunningJobs = len(self.batchSystem.getRunningBatchJobIDs())
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil_batch_system_tes/src/toil_batch_system_tes/tes_batch_system.py", line 412, in getRunningBatchJobIDs
    task = self.tes.get_task(tes_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/client.py", line 190, in get_task
    return unmarshal(response.json(), Task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 111, in unmarshal
    field = _unmarshal(v, obj)
            ^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 100, in _unmarshal
    field.append(unmarshal(item, obj))
                 ^^^^^^^^^^^^^^^^^^^^
  File "/home/heaucques/Documents/toil/venv3.12/lib/python3.12/site-packages/tes/utils.py", line 120, in unmarshal
    raise UnmarshalError(msg)
tes.utils.UnmarshalError: {'description': '', 'name': '', 'path': '/root/.aws', 'streamable': False, 'type': 'DIRECTORY', 'url': 'file:///home/heaucques/.aws'} could not be unmarshalled to type: Input
TypeError: Input.__init__() got an unexpected keyword argument 'streamable'

A similar result happens with 1.1.0rc1, where the service_info() function tries to request at the wrong URL:

requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:8000/v1/tasks/service-info

[vsmalladi]

Agreed same thing is happening. When trying this out

[jmfernandez]

I have tried py-tes 1.1.0 with funnel 0.11.0 in order to test the output declaration capabilities, and it is failing in that test because path_prefix is not declared in tes.models.Output. The code is an evolution of your quickstart example:

import tes
import json

# Define task
task = tes.Task(
    inputs=[
        tes.Input(url="file:///tmp/wireplumber.log", path="/tmp/morcilla_de_burgos.txt"),
    ],
    outputs=[
        tes.Output(url="file:///tmp/wireplumber.log.md5sum", path="/tmp/panceta_de_avila.txt"),
    ],
    executors=[
        tes.Executor(
            image="alpine",
            command=["md5sum", "/tmp/morcilla_de_burgos.txt"],
            stdout="/tmp/panceta_de_avila.txt",
        )
    ]
)

# Create client
cli = tes.HTTPClient("http://localhost:8000", timeout=5)

# Create and run task
task_id = cli.create_task(task)
cli.wait(task_id, timeout=5)

# Fetch task info
task_info = cli.get_task(task_id, view="BASIC")
j = json.loads(task_info.as_json())

# Pretty print task info
print(json.dumps(j, indent=2))

Funnel runs it properly, but py-tes raises a tes.utils.UnmarshallError when it parses the final answer:

Traceback (most recent call last):
  File "/home/jmfernandez/projects/ESG/docker-shim/.full13/lib/python3.13/site-packages/tes/utils.py", line 115, in unmarshal
    output = o(**r)
TypeError: Output.__init__() got an unexpected keyword argument 'path_prefix'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jmfernandez/projects/ESG/docker-shim/quickstart_file_in_out.py", line 29, in <module>
    task_info = cli.get_task(task_id, view="BASIC")
  File "/home/jmfernandez/projects/ESG/docker-shim/.full13/lib/python3.13/site-packages/tes/client.py", line 190, in get_task
    return unmarshal(response.json(), Task)
  File "/home/jmfernandez/projects/ESG/docker-shim/.full13/lib/python3.13/site-packages/tes/utils.py", line 111, in unmarshal
    field = _unmarshal(v, obj)
  File "/home/jmfernandez/projects/ESG/docker-shim/.full13/lib/python3.13/site-packages/tes/utils.py", line 100, in _unmarshal
    field.append(unmarshal(item, obj))
                 ~~~~~~~~~^^^^^^^^^^^
  File "/home/jmfernandez/projects/ESG/docker-shim/.full13/lib/python3.13/site-packages/tes/utils.py", line 120, in unmarshal
    raise UnmarshalError(msg)
tes.utils.UnmarshalError: {'description': '', 'name': '', 'path': '/tmp/panceta_de_avila.txt', 'path_prefix': '', 'type': 'FILE', 'url': 'file:///tmp/wireplumber.log.md5sum'} could not be unmarshalled to type: Output
TypeError: Output.__init__() got an unexpected keyword argument 'path_prefix'

and this is because funnel 0.11.0 is compliant with GA4GH TES 1.1.0, where path_prefix does exist.