Filter MAT before chronumental

[Haddox]

It looks like errors in estimated dates are leading to errors in mutation counts over time. This appears to be skewing some fitness trajectories.

A few things that could be leading to errors in estimated dates include:

• leaves that only have an estimated year (e.g., 2020)
• leaves that have many mutations on the branch leading to them (e.g., from chronic infection)

What if we just remove the above set of leaves from the MAT before estimating dates with chronumental? My impression is that this would only remove a small fraction of sequences. The filtering criteria would be to remove leaves that:

• only have an estimated year (e.g., 2020) and no month or day information
• have more than four nucleotide mutations on the branch leading to them

[Haddox]

Suggestions for implementation using BTE:

• The get_leaves() function could be used to get a list of all leaves in the input tree
• We could iterate over each leaf and filter based on its reported date (from metadata file) and number of mutations associated with the leaf's node object
• The remove_node() function could be used to remove the leaf from the tree

We'd then use the filtered tree as input to chronumental

[willdumm]

Not sure if Jared has already tried this, but matutils extract has a prune option, which removes a list of samples:

-p [ --prune ] Remove the selected samples instead of
keeping them in the output files.

The advantage of doing this instead of using the bte tree methods is that we don't have to delete edges without mutations, etc, since matutils should take care of this for us. Also, it may be a good idea to run some matoptimize iterations on the resulting tree, since optimal placements of samples could change once we remove the ones we're filtering?

So, it seems we can load the original MAT with bte, create a text file containing a list of samples to be filtered, run matutils extract to get our new input MAT, and pass that to the existing pipeline, starting at the very beginning?

@jgallowa07 you were having trouble with something like this, where you found the (internal?) node names didn't match between chronumental output and the new MAT. Is that right?

[jgallowa07]

Thanks, @willdumm. So yes I agree matUtils makes it easy to extract and subset the MAT tree. The pipeline currently has a step where we already use the extract command to get the newick for chronumental, and I've exposed a string parameter in the config file where you might pass any relevant kwargs (such as -p).

So, it seems we can load the original MAT with bte, create a text file containing a list of samples to be filtered, run matutils extract to get our new input MAT, and pass that to the existing pipeline, starting at the very beginning?

This seems like the correct approach, but remember we only start with the tree (protobuf MAT), and metadata (tsv). It's easy to see how we might identify the low-resolution sampling date strains, but I'm failing to see how we might identify the branches with many mutations for pruning before traversing the tree ...

you were having trouble with something like this, where you found the (internal?) node names didn't match between chronumental output and the new MAT. Is that right?

Correct, the extract step described above also output's the protobuf equivalent of the newick. When I tried any form of subsetting in those kwargs (-c, -p etc) it seems the internal node names were preserved in the newick passed to chronumental, but re-assigned for the MAT. In other words, when I pass the subsetted protobuf, along with the output times from chronumental to MATWrapper, it throws an error when looking for "node_2" in the times dictionary.

Note that I passed the full metadata file to chronumental, so I assume it inferred the earliest time point in that and set it as "node_1"?

I'm not great at looking at these files, but here

/fh/fast/matsen_e/shared/SARS2-mut-fitness-trajectory/jared/s2trajectory/Attic/results_attic/results_public_2023-06-21_try_delta_subset_crash_at_will/mat/

you'll find extracted_tree.nwk, and extracted_tree.pb as the two outputs from the matUtils extract -c B.1.617.2 command.

[willdumm]

Sorry I missed this mention again!

It's easy to see how we might identify the low-resolution sampling date strains, but I'm failing to see how we might identify the branches with many mutations for pruning before traversing the tree ...

matUtils --max-parsimony 5 (for example) I think prunes the tree to samples which have fewer than 5 mutations on their parent edge. I'm not totally certain this does what we want, and if it doesn't we'll just have to have a separate python script that loads the tree with bte and writes a list of samples to prune based on parent branch length, then input that list to matoptimize extract's -s argument.

Correct, the extract step described above also output's the protobuf equivalent of the newick. When I tried any form of subsetting in those kwargs (-c, -p etc) it seems the internal node names were preserved in the newick passed to chronumental, but re-assigned for the MAT. In other words, when I pass the subsetted protobuf, along with the output times from chronumental to MATWrapper, it throws an error when looking for "node_2" in the times dictionary.

From what I can see, this is probably intended (but undocumented) behavior. The MAT protobuf requires internal node IDs to be consecutive integers, so they're reassigned when writing to protobuf. You can get the behavior you want by first running matUtils extract to get the subsetted protobuf file, then running matUtils extract -i subsetted.pb -t subsetted.nwk a second time to get the newick with node IDs that match the protobuf.

Note that I passed the full metadata file to chronumental, so I assume it inferred the earliest time point in that and set it as "node_1"?

I'm not sure how Chronumental works, but I imagine it's inferring the time for node_1 to be some small amount of time earlier than the earliest sample observed in the tree. I would hope that metadata for samples that don't appear in the tree are automatically ignored by Chronumental.