Param tuning: ensembling (version 2 but all the same code as version 1) #212
Conversation
|
Reminder: There was one unresolved comment that we can continue discussing here #193 (comment). |
| @@ -105,8 +105,9 @@ def make_final_input(wildcards): | |||
| final_input.extend(expand('{out_dir}{sep}{dataset}-ml{sep}{algorithm}-ensemble-pathway.txt',out_dir=out_dir,sep=SEP,dataset=dataset_labels,algorithm=algorithms)) | |||
|
|
|||
| if _config.config.analysis_include_evaluation: | |||
| final_input.extend(expand('{out_dir}{sep}{dataset_gold_standard_pair}-evaluation.txt',out_dir=out_dir,sep=SEP,dataset_gold_standard_pair=dataset_gold_standard_pairs,algorithm_params=algorithms_with_params)) | |||
|
|
|||
| final_input.extend(expand('{out_dir}{sep}{dataset_gold_standard_pair}-eval{sep}precision-recall-curve-ensemble-nodes.png',out_dir=out_dir,sep=SEP,dataset_gold_standard_pair=dataset_gold_standard_pairs,algorithm_params=algorithms_with_params)) | |||
There was a problem hiding this comment.
The algorithm_params=algorithms_with_params part isn't being used and can be deleted
| final_input.extend(expand('{out_dir}{sep}{dataset_gold_standard_pair}-eval{sep}precision-recall-curve-ensemble-nodes.png',out_dir=out_dir,sep=SEP,dataset_gold_standard_pair=dataset_gold_standard_pairs,algorithm_params=algorithms_with_params)) | ||
| if _config.config.analysis_include_evaluation_aggregate_algo: | ||
| final_input.extend(expand('{out_dir}{sep}{dataset_gold_standard_pair}-eval{sep}{algorithm}-precision-recall-curve-ensemble-nodes.png',out_dir=out_dir,sep=SEP,dataset_gold_standard_pair=dataset_gold_standard_pairs,algorithm=algorithms)) |
There was a problem hiding this comment.
Can we call both of these pr-curve instead of precision-recall-curve to shorten the long filenames?
|
|
||
| # Run evaluation for each ensemble.txt for a dataset against its paired gold standard |
There was a problem hiding this comment.
| # Run evaluation for each ensemble.txt for a dataset against its paired gold standard | |
| # Run evaluation for each ensemble pathway for a dataset against its paired gold standard |
| dataset_label = get_dataset_label(wildcards) | ||
| return f"{out_dir}{SEP}{dataset_label}-ml{SEP}{wildcards.algorithm}-ensemble-pathway.txt" | ||
|
|
||
| # Run evaluation per algortihm for each ensemble.txt for a dataset against its paired gold standard |
There was a problem hiding this comment.
| # Run evaluation per algortihm for each ensemble.txt for a dataset against its paired gold standard | |
| # Run evaluation per algorithm for each ensemble pathway for a dataset against its paired gold standard |
| rule evaluation: | ||
| input: | ||
| gold_standard_file = get_gold_standard_pickle_file, | ||
| pathways = expand('{out_dir}{sep}{dataset_label}-{algorithm_params}{sep}pathway.txt', out_dir=out_dir, sep=SEP, algorithm_params=algorithms_with_params, dataset_label=get_dataset_label), | ||
| output: eval_file = SEP.join([out_dir, "{dataset_gold_standard_pairs}-evaluation.txt"]) | ||
| ensemble_file=lambda wildcards: f"{out_dir}{SEP}{get_dataset_label(wildcards)}-ml{SEP}ensemble-pathway.txt", |
There was a problem hiding this comment.
We use single quotes in most of the rest of the Snakefile. Also space before and after the =.
| else: | ||
| plt.figure() | ||
| plt.plot([], []) | ||
| plt.title("Empty Ensemble File") |
There was a problem hiding this comment.
Should the title be the same? Should we print a warning message instead?
There was a problem hiding this comment.
If this is the gold standard, let's indicate that in the filename.
| INPUT_DIR = 'test/evaluate/input/' | ||
| OUT_DIR = 'test/evaluate/output/' | ||
| EXPECT_DIR = 'test/evaluate/expected/' | ||
| NODE_TABLE = pd.read_csv(INPUT_DIR + "node_table.csv", header=0) |
There was a problem hiding this comment.
NODE_TABLE is pretty generic, let's have it refer to gold standard nodes
| plt.plot(recall, precision, marker='o', label='Precision-Recall curve') | ||
| plt.axhline(y=auc_precision_recall, color='r', linestyle='--', label=f'Avg Precision: {auc_precision_recall:.4f}') |
There was a problem hiding this comment.
These lines aren't what I thought they were at first. I thought the red horizontal line was the baseline PR curve, which is (number positives)/(number instances). I don't think we should plot the average precision as a horizontal line. Typically we would plot the baseline instead and put the average precision in the legend by the blue line.
|
|
||
| if not node_ensemble.empty: | ||
| y_true = [1 if node in gold_standard_nodes else 0 for node in node_ensemble['Node']] | ||
| y_scores = node_ensemble['max_freq'].tolist() |
There was a problem hiding this comment.
If the node ensemble does not include all of the gold standard nodes, this will not achieve full recall. That will be biased toward some methods that make more conservative predictions, right?
@agitter my param-tuning-ensembling branch #207 was out of sync with the changes I had locally. I needed to redo the branch to be up to date
@agitter Do this PR second (then follow up with the pull requests #208, #209 after this one is merged)
Will need to merge with updated master after #193 is merged. (hopefully this will remove the repeated files through out the PRs)
Included in this PR:
update to evaluation.py that will deal with making node ensemble frequencies then create a node PR curve
a new test suite evaluate for only ensembling idea
updates to Snakemake file that will run evaluation per dataset and per algortihm-dataset pair