Missing results and FANNGO error

[huh688]

Describe the bug
I reran Sorghum bicolor genome peptides (v3.1.1, downloaded from Phytozome) with GOMAP, and found only 32,159 genes were successfully annotated (not 34,129 genes in your Cyverse dataset, DOI:10.25739/4ty0-ye98).

Then I compared outputs from b.raw_gaf folder, found number of records from TAIR, Argot, PANNZER and UniProt were exactly same as your Cyverse dataset. The differences from FANN-GO (331,081 vs 564,161) and InterProScan (179,129 vs 186,551).

The output/log from step 2 domain search didn't show any error, but step 3 FANN-GO had a lots errors below

Input File
Sbicolor_454_v3.1.1.protein_primaryTranscriptOnly.fa, .

Sobic.010G276200.1.p
MGGGGGGGGNTRELDQTPTWAVASVCGVIVLISILLEKGLHHVGEFFSHRKKKAMVEALEKVKAELMVLGFISLLLVFGQ
NYIIKICISKNAASTMLPCKLEAVEAAGAGHGKEPAEGKAAGGKDGHGKEGAAVPGKKKAAAEHLGGVLDWPPPYYAHNA
RMLMEASSKSKCPEGKVSLISINALHQLHIFIFFLAVFHVSYSAITMALGRAKIRAWKEWEKEAAGQDYEFSNDPTRFRF
THETSFVRNHMNVLNKFPASFYISNFFRQFFRSVRQADYSALRHSFVNVHLAPGSKFDFQKYIKRSLEDDFKVIVGISPT
LWASALIFLFLNVNGWHTMLWISIMPVVIILSVGTKLQGIICRMAIDITERHAVVQGIPLVQVSDSYFWFSKPTFVLFLI
HFTLFQNGFQIIYFLWILYEYGMDSCFNDSKEFVFARLCLGVVVQVLCSYVTLPLYALVSQMGSTMKQSIFDEQTSKALK
NWRAGAKKKHPTSSKHGGGGSPTAGGSPTKADGDA
Sobic.010G080700.2.p
MAVLAKGDNEGGKVVDTMPLPTYHSLAAPVRKPVDKFALLPAFLKVRGLVKEHIDSFNYFIDKGIKKIVEANNRIEARND
PSIYLRYLDVRVGMPSIYENCNVESITPHLCRLTDRTYSAPIEVDIQYTVGKQHVVKRRNKFIIGYMPIMLRSSACILNE
MDEDELAKYGECPLDPGGYFVVKGTEKVILIQEQLSKNRIIVDTDSKGRVIASVTSSTHAIKSKTVVLMDKGKIYLQLNQ
FTKPIPIIVVMKAMGMESDQEAVQMVGRDPRYGDLLFPSIQECASERIYTRQQALQYMDEKVMYPGATNNKEGRSNSILH
DVFIAHVPVKDGNLRPKCIYTAVMLRRMLDAILNSDTFDDKDYVGNKRFELSGQLISLLFEDLFKTMNSQAVERMNKNSD
KTRSSPLDFSQLIMQDTITFGLERAISTGNWDIKRFRMHRKGVSQVLSRLSYMATLGYMTRITPQFEKTRKTSGPRALQP
SQWGMLCPCDTPEGEACGLTKNLALLSHITTDQEEGPLIDLCHSLGVEDLLFLSSEEIHAPGSFLVMLNGLILGKHKDPQ
RLVIHLRNLRRSNVIGEFVSIFINEKQRCVHIASDGGRVCRPLIIADMGISRIDETHMTKLRDGTYDFDDFLSGGLIEYL
DVNEENNALIALYEHMDQDDVERSSITHIEIEPLSILGVVSGLIPYPHHNQSPRNTYQCAMGKQAMGNIAYNQLFRADSL
LYLLVYAQRPLLTTKTIELVGYDKLGAGQNATVAVMGYSGYDIEDAIVMNKSSLDRGFGRCIALKKYTVLKDNYVDGVSD
RIVKPQRDKDGVLMKQNMRALDDDGIVAPGQIIRNHDIYVNKQTPMNTSKGIGRPLTDSCYKDSPAIYKGVDGETTVVDR
VMLCSDTNDKLSIKCIIRHTRRPEVGDKFSSRHGQKGVCGTIIQQEDFPFSERGICPDLIMNPHGFPSRMTIGKMIELLG
GKAGVSCGRFHYGSAFGEPSGNADNVADISSTLIKHGFSYNGKDFLYSGILGQPLEAYIFMGPIYYQKLKHMVLDKMHAR
ASGPRVLLTRQPTEGRSRDGGLRLGEMERDCLIAYGASMLIFERLMLSSDPYQVQVCRKCGLLGYYNHKLKTSYCSMCKN
GENMAKMRLPYACKLLFQELHAMNVVPRLKLTEV

GOMAP step that crashed (if applicable)
FANN-GO and InterProScan

Attach the output files
Errors from 3-fanngo

Warning: [blastp] Query is Empty!
error: writing file '/home/hhu/.octave_hist': No space left on device
error: ignoring const octave::execution_exception& while preparing to exit
Warning: [blastp] Query is Empty!

No errors in step 2 domain search

System Details

• OS: RedHat
• Version: 4.8.5-16
• Memory Size: 96GB

[wkpalan]

Hi @huh688,

I am checking this out now. Give me some time as I am experiencing some technical issues with the cluster I am using

Thanks

Best
Kokul

[huh688]

Hi @wkpalan,

Do you have any update for this issue?

I also tried several other species (setaria, pearl millet, tef), and no one got 100% annotation coverage.

[wkpalan]

Hi @huh688 ,

I have not had time to finalize a solution, but I am looking into it.

Regret the delay

Thanks
Kokul

[wkpalan]

Hi @huh688,

Thank you for bringing this to my attention. This was a weird bug that was difficult to troubleshoot. The issue was with the
update of singularity version and how octave runs the FANN-GO code.

The shell script to run the container uses a -c argument to prevent any local folders being mounted that end up in some conflicts. This option has triggered an issue with FANN-GO that end up with some steps failing but the output is written with wrong scores. This results in missing annotations and lower number. This also was affecting InterProScan/domain results.

Steps to fix and reproduce the annotations

• Checkout the latest version
  • git checkout v1.3.5
• Delete any output in the following directories
  • Sbicolor.BTx623.v3.1.1/GOMAP-Sbicolor.v3.1.1/tmp/domain/
  • Sbicolor.BTx623.v3.1.1/GOMAP-Sbicolor.v3.1.1/tmp/mixed-meth/fanngo
• Run the domain, fanngo and aggregate steps

This should result in a complete dataset, but might produce a few more annotations (+/- 10) than the original dataset.

Thank you for your patience and please let me know if this fixes the issues from your side.

[huh688]

Hi @wkpalan,

I reran the FANNGO step, still got the same error:

Warning: [blastp] Query is Empty!
error: writing file '/home/hhu/.octave_hist': No space left on device
error: ignoring const octave::execution_exception& while preparing to exit

I redownloaded new 1.3.5, and git-checkout showed "HEAD is now at ad548de... updating the documentation".

But when I ran ./setup.sh initially, it showed "INFO: Downloading shub image", then the setup ended.

When ran it again, I got error messages in the middle: "WARNING: destination /home/hhu/gomap-test already in mount list: destination is already in the mount point list
ERROR: environment_properties::capture: missing environment file. should be at [/home/hhu/.irods/irods_environment.json]", which I didn't see in 1.3.4.

See below for the complete output:

[hhu@pete gomap-test]$ ./setup.sh
INFO: Downloading shub image
109.61 MiB / 109.61 MiB [======================================================================================] 100.00% 74.62 MiB/s 1s
[hhu@pete gomap-test]$ pwd
/home/hhu/gomap-test
[hhu@pete gomap-test]$ ./setup.sh
WARNING: destination /home/hhu/gomap-test already in mount list: destination is already in the mount point list
ERROR: environment_properties::capture: missing environment file. should be at [/home/hhu/.irods/irods_environment.json]
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
0/1 - 0.00% of files done 0.000/39254.793 MB - 0.00% of file sizes done
Processing GOMAP.sif - 39254.793 MB 2021-05-17.15:14:06
From server: NumThreads=16, addr:r03c06u29-ds18.cyverse.org, port:20167, cookie=1281065451
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
NOTICE: created irodsHome=/iplant/home/anonymous
NOTICE: created irodsCwd=/iplant/home/anonymous
GOMAP.sif - 120.000/39254.793 MB - 0.31% done 2021-05-17.15:14:11

[wkpalan]

Let me take a look tonight and send you an update. Thanks Kokul

…

On Mon, May 17, 2021 at 11:20 AM Hao Hu ***@***.***> wrote: Hi @wkpalan <https://github.com/wkpalan>, I reran the FANNGO step, still got the same error: Warning: [blastp] Query is Empty! error: writing file '/home/hhu/.octave_hist': No space left on device error: ignoring const octave::execution_exception& while preparing to exit I redownloaded new 1.3.5, and git-checkout showed "HEAD is now at ad548de <ad548de>... updating the documentation". But when I ran ./setup.sh initially, it showed "INFO: Downloading shub image", then the setup ended. When ran it again, I got error messages in the middle: "WARNING: destination /home/hhu/gomap-test already in mount list: destination is already in the mount point list ERROR: environment_properties::capture: missing environment file. should be at [/home/hhu/.irods/irods_environment.json]", which I didn't see in 1.3.4. See below for the complete output: ***@***.*** gomap-test]$ ./setup.sh INFO: Downloading shub image 109.61 MiB / 109.61 MiB [======================================================================================] 100.00% 74.62 MiB/s 1s ***@***.*** gomap-test]$ pwd /home/hhu/gomap-test ***@***.*** gomap-test]$ ./setup.sh WARNING: destination /home/hhu/gomap-test already in mount list: destination is already in the mount point list ERROR: environment_properties::capture: missing environment file. should be at [/home/hhu/.irods/irods_environment.json] NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous 0/1 - 0.00% of files done 0.000/39254.793 MB - 0.00% of file sizes done Processing GOMAP.sif - 39254.793 MB 2021-05-17.15:14:06 From server: NumThreads=16, addr:r03c06u29-ds18.cyverse.org, port:20167, cookie=1281065451 NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous NOTICE: created irodsHome=/iplant/home/anonymous NOTICE: created irodsCwd=/iplant/home/anonymous GOMAP.sif - 120.000/39254.793 MB - 0.31% done 2021-05-17.15:14:11 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#21 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQPFNUIRBCDXOEEHBD2MXTTOEX2BANCNFSM4ZJWXPUQ> .

[wkpalan]

Hey @huh688,

I forgot to mention that the existing containers need to be deleted. Can you please do the following and try again..

1. Delete the icommands.sif and the GOMAP.sif files and then run setup.sh
2. git fetch -p && git checkout v1.3.5
3. re-run the above steps after the new image has been downloaded

Thanks

Best
Kokul

[wkpalan]

@huh688 can this be closed?

[huh688]

@wkpalan, sorry for the delayed reply. Our HPC is experiencing technical issues, the scheduler is heavily backlogged, and I'm still waiting to start FANNGO and aggregating steps. I'll let you know asap when I have results.

[huh688]

Hi @wkpalan,

I just got the results by rerunning the pipeline with Sorghum bicolor genome, but I got a pretty similar results as before: 177862 records from InterProScan, and 331079 from FANNGO.

The step 3 FANNGO didn't show any error (an empty .err output, no error in .out)

I followed your direction, installed new 1.3.5 version. But how can I confirm that I'm running 1.3.5? The README and RELEASE NOTE are still showing 1.3.4 info.

[wkpalan]

Hey @huh688.

Can you please get the latest version which is 1.3.7 and check of the output improves. There were some bugs related to this I had to fix.

The README and CHANGELOG have been updated.

[huh688]

Hi @wkpalan,

I reran the Sorghum dataset with v1.3.8 build, and got the 100% coverage of all 34,129 proteins.

I also reran several different species that I got lower coverage before, and found 100% sequences were successfully annotated with v1.3.8.

I think the problem is fixed now!