docs: LHCb Workflows and Commands#125
Conversation
aldbr
left a comment
aldbr
left a comment
There was a problem hiding this comment.
Thanks! The approach sounds good. A few comments though:
-
I think it would be interesting to have, for each type of jobs:
- left: current XML workflow
- right: the CWL + pre/post process equivalent
-
Could you add a very brief description of each module please?
-
AnalyseXMLSummary cannot work in
ProcessingI think because it needs to set some file status. So in the new workflow, we would need to have it in a post process step. We could potentially have a check withinLbRunAppas we said, to fail the workflow before it starts executing further steps. -
BookkeepingReport and WorkflowAccounting would need modifications (there will process multiple app outputs at once, this needs to appear)
AcquaDiGiorgio
force-pushed
the
issue-112-doc-lhcb-workflows
branch
from
c2ac24f to
83bf433
Compare
| UploadOutputData0[UploadOutputData] | ||
| UploadLogFile0[UploadLogFile] | ||
| UploadMC0[UploadMC] | ||
| FailoverTransfer0[FailoverTransfer] |
There was a problem hiding this comment.
No?
| FailoverTransfer0[FailoverTransfer] | |
| FailoverRequest0[FailoverRequest] |
| AnalyseXmlSummary1[AnalyseXmlSummary] | ||
| UploadLogFile1[UploadLogFile] | ||
| UploadOutputData1[UploadOutputData] | ||
| FailoverTransfer1[FailoverTransfer] |
There was a problem hiding this comment.
Same here no?
| FailoverTransfer1[FailoverTransfer] | |
| FailoverRequest1[FailoverRequest] |
| RemoveInputData0[RemoveInputData] | ||
| UploadLogFile0[UploadLogFile] | ||
| UploadMC0[UploadMC] | ||
| FailoverTransfer0[FailoverTransfer] |
There was a problem hiding this comment.
And here again?
| FailoverTransfer0[FailoverTransfer] | |
| FailoverRequest0[FailoverRequest] |
| AnalyseXmlSummary1[AnalyseXmlSummary] | ||
| UploadLogFile1[UploadLogFile] | ||
| UploadOutputData1[UploadOutputData] | ||
| FailoverTransfer1[FailoverTransfer] |
There was a problem hiding this comment.
And here?
| FailoverTransfer1[FailoverTransfer] | |
| FailoverRequest1[FailoverRequest] |
|
|
||
| Registers every output generated to the corresponding SE and to the Master Catalog or to the FailoverSE in case of failure. | ||
|
|
||
| ### FailoverTransfer |
There was a problem hiding this comment.
And here too?
| ### FailoverTransfer | |
| ### FailoverRequest |
| UploadLogFile UploadLogFile_l5@===> transferAndRegisterFile | ||
| UploadLogFile UploadLogFile_l6@===> setJobParameter | ||
| UploadLogFile UploadLogFile_l7@===> setApplicationStatus | ||
| UploadLogFile UploadLogFile_l8@===> getFileTypes |
There was a problem hiding this comment.
I don't see that in the code
| UploadLogFile UploadLogFile_l8@===> getFileTypes |
| | BookkeepingReport | Outputs | bookkeeping.xml | StepID ApplicationName ApplicationVersion StartTime ProductionId StepNumber SiteName JobType | | ||
| | WorkflowAccounting | N/A | N/A | RunNumber ProdID EventType SiteName ProcessingStep CpuTime NormCpuTime InputsStats OutputStats InputEvents OutputEvents EventTime NProcs JobGroup FinalState | | ||
| | AnalyseFileAccess | XMLSummary.xml pool_xml_catalog.xml | N/A | N/A | | ||
| | UserJobFinalization | UserOutputData | bookkeeping.xml | JobId UserOutputSE SiteName UserOutputPath ReplicateUserOutData UserOutputLFNPrep | |
There was a problem hiding this comment.
Does it create the bookkeeping.xml file?
|
|
||
| FileUsage("FileUsage") | ||
|
|
||
| FileUsage FileUsage_l1@===> getValueGconf |
There was a problem hiding this comment.
What about dataUsageClient.sendDataUsageReport?
| subgraph Processing0[''Processing''] | ||
| direction TB | ||
|
|
||
| GaudiApplication0[GaudiApplication] |
There was a problem hiding this comment.
Don't you think we will need a preprocess command for GaudiApplication?
|
|
||
| Currently, the Workflow Modules execute in a predefined order. | ||
|
|
||
| For the new approach with CWL, the modules are called "commands" and can be executed in any order, because they don't depend on any other's outputs. However, an order has to be defined while defining the `JobType`, which can be the same as the current order. |
There was a problem hiding this comment.
Is that true?
BookkeepingReport creates the bookkeeping.xml file, then it is consumed by UploadOutputData for instance
2 participants
See #112
Apparantly, Git does not support ELK as a mermaid renderer...