The job wrapper should report the status of the job to DiracX

[aldbr]

The new job wrapper could not report the status of the job if used within Dirac.
The methods to report the status of a given job area already part of DiracX and so should be preferred over the Dirac ones.

Tasks:

• Analyze the DIRAC JobWrapper and provide a description of the JobReport calls and their location
• Implement calls to the corresponding endpoints within diracx-api
• Call diracx-api from the new job wrapper (unless the submission is done locally)
• The job ID should also be added to the JobSubmissionModel (not clear if we need yet another child class or not - to be discussed)

Done criteria:

• Unit tests around the job wrapper
• Development in diracx is merged

[Loxeris]

DIRAC's Job Wrapper job reporting

JobWrapperTemplate ⇾ JobWrapperUtilities

Rescheduling

rescheduleFailedJob()

Reschedules happen when a job wrapper method returns an error :
Status: RESCHEDULED or FAILED if maxed reschedulings
MinorStatus depends on the failing function
ApplicationStatus: "Failed {minorStatus}"

createAndEnterWorkingDirectory()

if failed because directory already exists:
rescheduling with MinorStatus: "Working Directory already exists"

else other error:
rescheduling with MinorStatus: "Could Not Create Working Directory"

getJobWrapper()

if Exception in JobWrapper() or initialize():
rescheduling with MinorStatus: JOB_WRAPPER_INITIALIZATION = "Job Wrapper Initialization"

JobWrapper

initialize()

JobStatus: RUNNING
MinorStatus: JOB_INITIALIZATION = "Job Initialization"

transferInputSandbox()

JobStatus: (RUNNING expected)
MinorStatus: DOWNLOADING_INPUT_SANDBOX = "Downloading InputSandbox"

if fail:
JobStatus: (RUNNING expected)
MinorStatus: FAILED_DOWNLOADING_INPUT_SANDBOX = "Failed Downloading InputSandbox"

if some input sandboxes url are lfns:
JobStatus: (RUNNING expected)
MinorStatus: DOWNLOADING_INPUT_SANDBOX_LFN = "Downloading InputSandbox LFN(s)"

if lfn sandbox download failed:
JobStatus: (RUNNING expected)
MinorStatus: FAILED_DOWNLOADING_INPUT_SANDBOX_LFN = "Failed Downloading InputSandbox LFN(s)"

if failed, rescheduled with MinorStatus: DOWNLOADING_INPUT_SANDBOX = "Downloading InputSandbox"

resolveInputData()

JobStatus: (RUNNING expected)
MinorStatus: INPUT_DATA_RESOLUTION = "Input Data Resolution"

if failed, rescheduled with MinorStatus: INPUT_DATA_RESOLUTION = "Input Data Resolution"

execute()

Pre-Process: PrepareCommand()

if Missing Executable:
JobStatus: FAILED
MinorStatus: APP_NOT_FOUND = "Application not found"

Execution

MinorStatus: APPLICATION = "Application"

PostProcess()

if App Thread did not complete:
JobStatus: FAILED
MinorStatus: APP_THREAD_NOT_COMPLETE = "Application thread did not complete"

if App Thread error:
JobStatus: FAILED
MinorStatus: APP_THREAD_FAILED = "Application thread failed"

if watchdogError:
JobStatus: FAILED
MinorStatus: error returned by Watchdog

→ rescheduled with MinorStatus: EXCEPTION_DURING_EXEC = "Exception During Execution"

if no watchdog error but payloadStatus != 0:
JobStatus: COMPLETING
MinorStatus: APP_ERRORS = "Application Finished With Errors"

if payloadStatus in DErrno.EWMSRESC, DErrno.EWMSRESC & 255,
MinorStatus: GOING_RESCHEDULE
→ rescheduled with MinorStatus: JOB_WRAPPER_EXECUTION = "JobWrapper execution"

if no watchdog error and status == 0:
JobStatus: COMPLETING
MinorStatus: APP_SUCCESS = "Application Finished Successfully"

ProcessJobOutput()

if job execution did not fail:
JobStatus: COMPLETING
MinorStatus: UPLOADING_OUTPUT_SANDBOX = "Uploading Output Sandbox"

if not sandbox upload fail:
JobStatus: COMPLETING
MinorStatus: OUTPUT_SANDBOX_UPLOADED = "Output Sandbox Uploaded"

if failed, rescheduled with MinorStatus: UPLOADING_JOB_OUTPUTS = "Uploading Outputs"

transferOutputDataFiles()

MinorStatus: UPLOADING_OUTPUT_DATA = "Uploading Output Data"

if fail:
JobStatus: FAILED
MinorStatus: UPLOADING_OUTPUT_DATA = "Uploading Output Data"

else:
JobStatus: COMPLETING
MinorStatus: OUTPUT_DATA_UPLOADED = "Output Data Uploaded"

if failed, rescheduled with MinorStatus: UPLOADING_JOB_OUTPUTS = "Uploading Outputs"

finalize()

if Success:
JobStatus: DONE
MinorStatus: EXEC_COMPLETE = "Execution Complete"

if Success but file requests pending:
JobStatus: COMPLETE
MinorStatus: EXEC_COMPLETE = "Execution Complete"

if Fail:
JobStatus: FAILED

if Fail (failover request error):
JobStatus: FAILED
MinorStatus: FAILED_SENDING_REQUEST = "Failed sending requests"

if Fail (failover request pending):
JobStatus: FAILED
MinorStatus: PENDING_REQUEST = "Pending Requests"

Notes

source should be "JobWrapper" for all reports