Google Cloud Batch - logging & monitoring options

[puerta-amarilla]

Google Cloud Batch supports enableing logs and installing Ops Agent:

https://docs.cloud.google.com/batch/docs/create-run-job-ops-agent#ops-agent-requirements

1. For logging:

"logsPolicy": {
"destination": "CLOUD_LOGGING"
}

Nextflow seems to enable this by default. All messages written to stdout and stderr are captured. Even the central logging "nextflow.log" can be redirected. However, I can't find a way to redirect the tasks logs from each working directory to Cloud_Logging? The reason to want to use Cloud Logging instead of the default scratch space of cloud storage is two fold:

1. It's unpreticable when oogs in storage bucket will show up. It seems after the tasks are done, not during running.

2. Be able to have a single place to see all logs offers the convenience.
   set process.debug to true might be possible but it could print too many messages.

3. For monitoring:

Sometimes it's also useful to see Ops Agent based metrics
"allocationPolicy": {
"instances": [
{
"installOpsAgent": true
}
]
},

However, I can't find a way to enable it through Nextflow. Any suggestions?