Consider removing combine hail table step

[rileyhgrant]

Currently the four datasets are combined into a single hail table with one row for every gene, a struct for gene info containing info per dataset, and a struct for variants containing each dataset and its respective list of variants.

Then, when writing the tables, this entire table is written to a temp .tsv file, then the .tsv file is written to individual gene results, and variant results files per gene, one for each dataset.

Unless I am missing something, this combination into a single combined table does very little, given we validate the outputs from each of the individual pipelines. We could just generate the results json files from each individual dataset.

---

I was working on getting Transcript Consequences on a variant level for the IBD dataset, and after it working in development with a smaller subset of genes, it choked in production in the write results files steps, leading to looking a bit closer at whats happening and filing this issue.

[rileyhgrant]

From browser meeting: this may allow ensuring sharing of certain data, particularly at the gene level.

To my eyes (and without properly digging into it), all the combined variant information gets split right back up as part of the next step.

Changing this would also be a big change, and would require a good deal of effort. The juice is likely not worth the squeeze -- I had only been looking into this when trying to get Transcript Consequences as a variant level field, and at that point scale became an issue (the variants field of the combined table became quite large, and memory constraints started to become an issue during the export to tsv step).