Process `UTILS_MERGE_COHORT_STATS (joint_name: joint)` terminated with an error exit status (1)

[MohdKhairulNizam]

Good day,

Nizam here. I used the following script for analysis:

./nextflow run 'https://github.com/TORCH-Consortium/MAGMA' -r v1.1.1 -params-file my_parameters_1.yml -profile docker, low_memory --max_memory 10.0 GB -resume

and I came across this error:

[20/cefe9d] NOTE: Process UTILS_MERGE_COHORT_STATS (joint_name: joint) terminated with an error exit status (1) -- Execution is retried (1)
ERROR ~ Error executing process > 'UTILS_MERGE_COHORT_STATS (joint_name: joint)'

Caused by:
Process UTILS_MERGE_COHORT_STATS (joint_name: joint) terminated with an error exit status (1)

Command executed:

generate_merged_cohort_stats.py
--relabundance_approved_tsv approved_samples.relabundance.tsv
--relabundance_rejected_tsv rejected_samples.relabundance.tsv
--call_wf_cohort_stats_tsv joint.cohort_stats.tsv
--output_file joint.merged_cohort_stats.tsv

Command exit status:
1

Command output:
(empty)

Command error:
Traceback (most recent call last):
File "/home/khairulnizam/.nextflow/assets/TORCH-Consortium/MAGMA/bin/generate_merged_cohort_stats.py", line 55, in
df_final_cohort_stats['ALL_THRESHOLDS_MET'] = df_final_cohort_stats['MAPPED_NTM_FRACTION_16S_THRESHOLD_MET'].astype('bool') & df_final_cohort_stats['COVERAGE_THRESHOLD_MET'].astype('bool') & df_final_cohort_stats['BREADTH_OF_COVERAGE_THRESHOLD_MET'].astype('bool') & df_final_cohort_stats['RELABUNDANCE_THRESHOLD_MET'].astype('bool')
File "/opt/conda/lib/python3.9/site-packages/pandas/core/generic.py", line 6240, in astype
new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
File "/opt/conda/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 445, in astype
return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
File "/opt/conda/lib/python3.9/site-packages/pandas/core/internals/managers.py", line 347, in apply
applied = getattr(b, f)(**kwargs)
File "/opt/conda/lib/python3.9/site-packages/pandas/core/internals/blocks.py", line 526, in astype
new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
File "/opt/conda/lib/python3.9/site-packages/pandas/core/dtypes/astype.py", line 299, in astype_array_safe
new_values = astype_array(values, dtype, copy=copy)
File "/opt/conda/lib/python3.9/site-packages/pandas/core/dtypes/astype.py", line 227, in astype_array
values = values.astype(dtype, copy=copy)
File "/opt/conda/lib/python3.9/site-packages/pandas/core/arrays/masked.py", line 474, in astype
raise ValueError("cannot convert float NaN to bool")
ValueError: cannot convert float NaN to bool

Work dir:
/home/khairulnizam/work/b4/5a234241888263be2dbc9db3e4bdfc

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details

Can I kindly ask for your help to solve this? Thanks

Best Wishes,

Nizam

[abhi18av]

Hi @MohdKhairulNizam

It's a curious issue and from the python trace it seems like there are some missing values in one of the files used in the process.

approved_samples.relabundance.tsv
rejected_samples.relabundance.tsv
joint.cohort_stats.tsv

If you could please share these files I can take a further look, but otherwise it'd be tough to really try and address this issue.

Please feel free to share these files on my academic email abhinavsharma at sun dot ac dot za and I can take a further look.

Also, great job on getting the pipeline all setup and running with docker ✅

[MohdKhairulNizam]

Hi @abhi18av
I have emailed you the 3 files. Thanks :)
Best Wishes,
Nizam

[abhi18av]

Thanks for the email @MohdKhairulNizam ,

I’ve taken a look at the relevant files and something is amiss with the approved_samples.relabundance.tsv which doesn't contain any relevant sample IDs , instead it only has tbprofiler as a sample ID.

The UTILS_MERGE_COHORT_STATS step is designed to merge the statistics of the analysed samples with into a single file but maybe the case is your samples are being filtered out.

I suggest that you run the pipeline with tweaked parameters, I'm sharing the defaults used in v1.1.1

//The median coverage required to process the sample
median_coverage_cutoff : 10

//The breadth of coverage required to process the sample
breadth_of_coverage_cutoff : 0.90

//The relative abundunce of the majority strain required to process the sample
rel_abundance_cutoff : 0.80

//The maximum fraction of NTM DNA allowed to process the sample
ntm_fraction_cutoff : 0.20

// The minimum fraction of samples that need to have a call at a site before the site is considered in phylogeny
site_representation_cutoff : 0.95

Also, please remember that beginning from v2.0.0-alpha we have switched to WHO mutation calalog v2 which might be more appropriate for your needs?

====

Also, as this is not a pipeline bug, I'm moving this to a discussion and we can continue this discussion over there.