Add a new advanced tutorial on spooling/remote submit

[JoshKarpel]

Using the Submit object instead of Schedd.submit.

[simonepigazzini]

Hi,

this would be indeed very useful, I've been struggling to submit jobs using schedd.submit with spoll=True. I only need to spool output files, but jobs are held by the scheduler waiting for the inputs to be spooled, and I can’t find a way to release them. The cmd line condor_release does not release the jobs and I can't get schedd.spoll to work neither using the classad from SubmitResults nor those filled by schedd.submit into ad_results (which looking at the code seems to be bypassed when using itemdata.

I think there might be some issue in the python bindings, but a tutorial on this would be for sure helpful.

simone

PS: as a sidenote the same submit process works fine from the cmdline using condor_submit subfile -spool

[johnkn]

The jobs are being held in the schedd pending that spooling operation. You need to use the python schedd.spool method to spool the input files which will release the hold. The documentation for schedd.submit says this. param bool spool: If ``True``, jobs will be submitted in a spooling hold mode so that input files can be spooled to a remote *condor_schedd* daemon before starting the jobs. This parameter is necessary for jobs submitted to a remote *condor_schedd* that use HTCondor file transfer. You can get the collection of jobs that you need to pass to schedd.spool either by using the jobs method of the submit object. or by fetching the jobs that are on hold in the schedd using schedd.query.

…

-tj

________________________________ From: simonepigazzini ***@***.***> Sent: Friday, December 10, 2021 3:43 AM To: htcondor/htcondor-python-bindings-tutorials ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [htcondor/htcondor-python-bindings-tutorials] Add a new advanced tutorial on spooling/remote submit (#21) Hi, this would be indeed very useful, I've been struggling to submit jobs using schedd.submit with spoll=True. I only need to spool output files, but jobs are held by the scheduler waiting for the inputs to be spooled, and I can’t find a way to release them. The cmd line condor_release does not release the jobs and I can't get schedd.spoll to work neither using the classad from SubmitResults nor those filled by schedd.submit into ad_results (which looking at the code seems to be bypassed when using itemdata. I think there might be some issue in the python bindings, but a tutorial on this would be for sure helpful. simone PS: as a sidenote the same submit process works fine from the cmdline using condor_submit subfile -spool — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#21 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABDNHMSOCWFXEPSLFLRQRMLUQHDVVANCNFSM4OVTRZ6A>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[simonepigazzini]

Thank you @johnkn. Is there a way to spool only output files? At the moment I get an error from schedd.spoll [1], which most likely is misconfiguration on my side but given that I do not need to spool the inputs I would rather bypass the issue by just spooling the output.

PS: apologies if this is not the right place to discuss this, if this is the case where should I open an issue to further discuss this?

[1]:htcondor.HTCondorIOError: DCSchedd::spoolJobFiles:7002:File transfer failed for target job 9500411.0: TOOL at 137.138.62.88 failed to send file(s) to <137.138.76.70:9618>: error sending /var/lib/condor/spool/411/0/cluster9500411.proc0.subproc0/grid_proxy.x509; SCHEDD at 137.138.76.70 failed to receive file /var/lib/condor/spool/411/0/cluster9500411.proc0.subproc0.tmp/grid_proxy.x509

[HealthyPear]

Hello!

I arrive with almost 1 year of delay, but I have followd the same instuctions and I get the same error that @simonepigazzini found!

Any ideas? Meanwhile I will investigate on my own and if I end up being really desperate I might open a ticket on GGUS...

For reference, I am following the base tutorial.

Here is the code I am using to submit the test job,

from htcondor import Collector, Schedd, DaemonTypes, Submit

job_description = htcondor.Submit(
    {
        "executable": "/bin/hostname",
        "output": "hostname.out",
        "error": "hostname.err",
        "log": "hostname.log",
    }
)

collector = Collector()
scheduler_ad = collector.locate(
    daemon_type=DaemonTypes.Schedd, name="sn-02.cr.cnaf.infn.it"
)
scheduler = Schedd(scheduler_ad)

job = Submit(**job_description)
submit_result = scheduler.submit(job, spool=True)

Then I tried with both htcondor.Submit.jobs(),

In [21]: scheduler.spool([j for j in job.jobs()])
---------------------------------------------------------------------------
HTCondorIOError                           Traceback (most recent call last)
Cell In [21], line 1
----> 1 scheduler.spool([j for j in job.jobs()])

File ~/.conda/envs/swgo-htcondor-interface/lib/python3.11/site-packages/htcondor/_lock.py:70, in add_lock.<locals>.wrapper(*args, **kwargs)
     67 try:
     68     acquired = LOCK.acquire()
---> 70     rv = func(*args, **kwargs)
     72     # if the function returned a context manager,
     73     # create a LockedContext to manage the lock
     74     is_cm = is_context_manager(rv)

HTCondorIOError: DCSchedd::spoolJobFiles:7002:File transfer failed for target job 1.0: Failed to receive GoAhead message from 131.154.192.42.

and the query method from the scheduler object,

In [21]: scheduler.spool([j for j in job.jobs()])
---------------------------------------------------------------------------
HTCondorIOError                           Traceback (most recent call last)
Cell In [21], line 1
----> 1 scheduler.spool([j for j in job.jobs()])

File ~/.conda/envs/swgo-htcondor-interface/lib/python3.11/site-packages/htcondor/_lock.py:70, in add_lock.<locals>.wrapper(*args, **kwargs)
     67 try:
     68     acquired = LOCK.acquire()
---> 70     rv = func(*args, **kwargs)
     72     # if the function returned a context manager,
     73     # create a LockedContext to manage the lock
     74     is_cm = is_context_manager(rv)

HTCondorIOError: DCSchedd::spoolJobFiles:7002:File transfer failed for target job 1.0: Failed to receive GoAhead message from 131.154.192.42.

If you prefer I can open a new issue.

By the way - why is the repo for the python bindings archived? Shouldn't these kind of issues go there?