Open
Description
Nistha reported that DOMINO failed on our departmental cluster when running in Singularity. I confirmed this when testing it with the SPRAS EGFR configuration:
Running Singularity containers
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Select jobs to execute...
Running slicer with arguments: slicer --network_file /spras/D263Q3O/network.txt --output_file /spras/X62MXDY/slices.txt
INFO: Using cached SIF image
Running DOMINO with arguments: domino --active_genes_files /spras/36SYPK7/active_genes.txt --network_file /spras/D263Q3O/network.txt --slices_file /spras/X62MXDY/slices.txt --output_folder /spras/DU7TQ24/tps-egfr-domino-params-V3X4RW7 --use_cache false --parallelization 1 --visualization true --slice_threshold 0.3 --module_threshold 0.05
{'message': ["start running DOMINO...\ngenerating graph from /spras/D263Q3O/network.txt\nnetwork' pkl is saved: /spras/D263Q3O/network.txt.pkl\ndone building network\ngenerating subgraphs...\nBefore slicing: n of cc:30, n of nodes: 15679, n of edges, 161886\n# of modules after extraction: 201\nAfter slicing: n of cc:203, n of nodes: 15615, n of edges, 47086\nsubgraphs' pkl is saved\nnumber of slices: 203\n# of slices after perturbation TH: 35/203\n33 relevant slices were retained with threshold 0.3\nn of putative modules: 33\nn of final modules: 26 (n=[60, 22, 18, 35, 30, 22, 20, 16, 11, 18, 81, 8, 9, 10, 16, 6, 28, 20, 20, 13, 22, 15, 15, 4, 18, 12])\n26 final modules are reported at /spras/DU7TQ24/tps-egfr-domino-params-V3X4RW7/active_genes/modules.out\nTrue\nvisualizing modules...\n", 'INFO: Using cached SIF image\nmultiprocessing.pool.RemoteTraceback: \n"""\nTraceback (most recent call last):\n File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker\n result = (True, func(*args, **kwds))\n File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar\n return list(map(*args))\n File "/usr/local/lib/python3.7/site-packages/src/utils/visualize_modules.py", line 199, in module_report\n generate_report_from_template(cy, output_base_dir, str(module_index))\n File "/usr/local/lib/python3.7/site-packages/src/utils/visualize_modules.py", line 169, in generate_report_from_template\n report_file_name=format_script(os.path.join(os.path.dirname(os.path.abspath(__file__)),\'../data\', "graph.html"), NUM_OF_GENES=len([x for x in cy if not "source" in x["data"] and len(x["data"]["modules"])>0]), HG_REPORT=[], MODULES_SUMMARY=[], DISEASE_GENES=[], DATA=json.dumps(cy))\n File "/usr/local/lib/python3.7/site-packages/src/utils/scripts.py", line 11, in format_script\n open(exec_file_name, "w+").write(formatted_script)\nOSError: [Errno 30] Read-only file system: \'/usr/local/lib/python3.7/site-packages/src/utils/../data/graph_0.9355720314087165.html\'\n"""\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/local/bin/domino", line 8, in <module>\n sys.exit(main_domino())\n File "/usr/local/lib/python3.7/site-packages/src/runner.py", line 53, in main_domino\n visualize_modules(os.path.splitext(cur_ag.split(\'/\')[-1])[0], G_final_modules, None, network_file, report_folder)\n File "/usr/local/lib/python3.7/site-packages/src/utils/visualize_modules.py", line 188, in visualize_modules\n p.map(module_report, params)\n File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 268, in map\n return self._map_async(func, iterable, mapstar, chunksize).get()\n File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 657, in get\n raise self._value\nOSError: [Errno 30] Read-only file system: \'/usr/local/lib/python3.7/site-packages/src/utils/../data/graph_0.9355720314087165.html\'\n'], 'return_code': 1}
I believe the problem is this line of the source code: https://github.com/Shamir-Lab/DOMINO/blob/85dad1515717b425b17f58f92b13a063ccccb85d/src/utils/visualize_modules.py#L169
DOMINO tries to write a temporary output file to the directory where it is installed, which is not a writeable location inside a Singularity container. Testing the container interactively with Singularity confirms this directory is owned by root:
$ singularity exec docker://reedcompbio/domino /bin/bash
Apptainer> ls -l /usr/local/lib/python3.7/site-packages/src/utils/
total 20
-rw-r--r--. 1 root root 0 Jul 23 11:06 __init__.py
drwxr-xr-x. 2 root root 206 Jul 23 11:06 __pycache__
-rw-r--r--. 1 root root 2451 Jul 23 11:06 ensembl2gene_symbol.py
-rw-r--r--. 1 root root 4068 Jul 23 11:06 graph_influence_linear_th.py
-rw-r--r--. 1 root root 452 Jul 23 11:06 scripts.py
-rw-r--r--. 1 root root 9327 Jul 23 11:06 visualize_modules.py
Edit: See https://github.com/Shamir-Lab/DOMINO/issues/13