-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Right now steps (at least 03.split) that write their output to $TMPDIR. As stated by @jelmervdl, this is probably because LUMI has not $SCRATCH set and files are written to a temporary dir in the node that is owned by the user and not the group. We should have something like this for LUMI that creates the temporary directory and changes ownerships.
Example of giashard doing it well but 03.split not:
$ ll output_wide15_filtered_sample3-shards/es/*/*
output_wide15_filtered_sample3-shards/es/0/1:
total 376M
-rw-r----- 1 zaragoza2 project_462000252 188M May 9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2 187M May 10 16:40 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252 12K May 9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 655K May 9 18:04 url.gz
output_wide15_filtered_sample3-shards/es/1/1:
total 322M
-rw-r----- 1 zaragoza2 project_462000252 162M May 9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2 160M May 10 16:39 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252 11K May 9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 661K May 9 18:04 url.gz
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working