test is not working as expected. Fails on generate config

[chriswyatt1]

Description of the bug

Hi,

I am trying to run the test data to check it works on my HPC (sge). It keeps failing with the following command:
nextflow run main.nf -profile ucl_myriad,test -bg -resume. config is https://github.com/nf-core/configs/blob/master/conf/ucl_myriad.config

I checked on an sbatch cluster (at another uni) too and it has the same error:

bash

  generate_config.py \
      --fasta /myriadfs/home/ucbtcdr/Scratch/blobtoolkit/work/69/888cf78b264ab2b07281b7baaf11d1/GCA_922984935.2.subset.phiXspike.fasta \
      --taxon_query "Meles meles" \
      --lineage_tax_ids mapping_taxids-busco_dataset_name.2019-12-16.txt \
       \
      --accession GCA_922984935.2 \
      --nt nt_mMelMel3.1 \
      --read_id mMelMel3_hic --read_type hic --read_layout PAIRED --read_path /test-data/Meles_meles/analysis/mMelMel3.2_paternal_haplotype/read_mapping/hic/GCA_922984935.2.subset.unmasked.hic.mMelMel3.cram --read_id mMelMel2 --read_type illumina --read_layout PAIRED --read_path /test-data/Meles_meles/analysis/mMelMel3.2_paternal_haplotype/read_mapping/illumina/GCA_922984935.2.subset.unmasked.illumina.mMelMel2.cram --read_id mMelMel1 --read_type illumina --read_layout PAIRED --read_path /test-data/Meles_meles/analysis/mMelMel3.2_paternal_haplotype/read_mapping/illumina/GCA_922984935.2.subset.unmasked.illumina.mMelMel1.cram --read_id mMelMel3_ont --read_type ont --read_layout SINGLE --read_path /test-data/Meles_meles/analysis/mMelMel3.2_paternal_haplotype/read_mapping/ont/GCA_922984935.2.subset.unmasked.ont.mMelMel3.cram \
      --blastn https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/resources/nt_mMelMel3.1.tar.gz --blastx https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/resources/mMelMel3.1.buscoregions.dmnd.tar.gz --blastp https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/resources/mMelMel3.1.buscogenes.dmnd.tar.gz --taxdump https://tolit.cog.sanger.ac.uk/test-data/resources/new_taxdump.tar.gz \
      --output_prefix GCA_922984935.2
  
  cat <<-END_VERSIONS > versions.yml
  "SANGERTOL_BLOBTOOLKIT:BLOBTOOLKIT:INPUT_CHECK:GENERATE_CONFIG":
      generate_config.py: $(generate_config.py --version | cut -d' ' -f2)
  END_VERSIONS

Command exit status:
  1

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_TMP is set, but APPTAINERENV_TMP is preferred
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  Traceback (most recent call last):
    File "/myriadfs/home/ucbtcdr/Scratch/blobtoolkit/bin/generate_config.py", line 393, in <module>
      sys.exit(main())
    File "/myriadfs/home/ucbtcdr/Scratch/blobtoolkit/bin/generate_config.py", line 360, in main
      sequence_report = get_sequence_report(args.accession)
    File "/myriadfs/home/ucbtcdr/Scratch/blobtoolkit/bin/generate_config.py", line 216, in get_sequence_report
      if not response["reports"]:
  KeyError: 'reports'

Command used and terminal output

Relevant files

No response

System information

Nextflow version 25.04.6
sanger-tol/blobtoolkit 0.8.0
SGE, with singularity/apptainer

[muffato]

Hi Chris.
I suspect something went wrong the line before in the script when it did an API query to the NCBI. It's the script's fault for not checking the success of the API call.

First thing to check: do the compute nodes of your HPC have access to the internet ?

The other possible cause is that NCBI Datasets have faced some infrastructure difficulties recently, resulting in API calls failing. I think it's getting solved and maybe the pipeline will work if you try it again.

Otherwise I can work on adding some code to catch and handle API call failures, it's a good thing to have anyway.

Matthieu

[ssandovala]

I am also seeing the same error, and my cluster node has internet connection. Have you been able to figure out why the error is persisting?

[chriswyatt1]

Hey, Sorry for the delay. Running nextflow run main.nf -profile ucl_myriad,test -resume -bg works now. I did check and made sure the nodes had a connection, so likely it was just API issues, as you mentioned. Thanks