Create new dataset using Fish-AIR (1/5)

[johnbradley]

The current dataset used by the workflow is from the legacy predecessor of Fish-AIR.
The new URL for Fish-AIR is https://fishair.org/api/docs#

[johnbradley]

@megbalk has created a new dataset using the above URL and deposited it into datacommons:
https://datacommons.tdai.osu.edu/dataset.xhtml?persistentId=doi:10.5072/FK2/GYHRV8&version=DRAFT
The dataset isn't currently published. We have run limited tests with the workflow and everything seemed correct.
All files from the Fish-AIR zip file (including metadata) are included in the datacommons dataset.

The two remaining items for this issue are:

• Run the full workflow ensuring everything works as expected with the new dataset
• Make the dataset public

[meghalithic]

@johnbradley I ran the full dataset, but received the following error (slurm-24142472.out):

Files/Previous Fish Measurements - Burress et al. 2016.csv, Library
output: Results/presence.absence.matrix.csv, Results/sampling.species.in.Burress.csv, Results/sampling.minnows.95.blob.3.segments.csv, Results/sampling.df.seg.csv, Results/presence.absence.sample.dist.png, Results/heatmap.avg.blob.png, Results/heatmap.sd.blob.png
shell:
R_LIBS_USER=Library Rscript /users/PAS2136/balkm/minnowTraits/workflow/../Scripts/Presence_Absence_Analysis.R
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
cluster_jobid: Submitted batch job 24146094
Error executing rule presence_absence_analysis on cluster (jobid: 1, external: Submitted batch job 24146094, jobscript: /users/PAS2136/balkm/minnowTraits/.snakemake/tmp.utkryvfp/snakejob.presence_absence_analysis.1.sh). For error details see the cluster log and the log files of the involved rule(s).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-04-05T113025.704681.snakemake.log

[meghalithic]

also @johnbradley should we get help / advisement on adding metadata before making the dataset public?

[johnbradley]

@johnbradley I ran the full dataset, but received the following error (slurm-24142472.out):

@megbalk A little further up in the output you should be able to see the log filename for the presence_absence_analysis rule.

[johnbradley]

@megbalk I ran the latest code and also had the presence_absence_analysis rule fail.
Here is part of the error message from the logs/presence_absence_analysis/presence_absence_analysis... log:

5 Cyprinella spiloptera (2) sd.trunk     0
6   Notropis volucellus (1) sd.trunk    NA
Error in drop_na(., value) : could not find function "drop_na"
Calls: %>%
Execution halted

I think we need to add tidyr:: before drop_na here:

Minnow_Segmented_Traits/Scripts/Presence_Absence_Analysis.R

Line 281 in 91649c1

melt_stats_sd.na <- melt_stats_sd %>% drop_na(value)