Describe the bug
compute_ctrl_recall_posthoc() documents that samples with no control variables in the gold label are skipped because recall is undefined.
That happens when solver output parses successfully, because compute_recall(..., set()) returns NaN. But if the same sample has malformed solver output, preds becomes None and the function instead sets:
So a sample that should always be excluded from the average is counted as a zero only when parsing fails, which can incorrectly lower the reported post-hoc control recall.
Expected behavior
Samples whose gold hypothesis is invalid, or whose valid gold hypothesis contains no control variables, should be skipped consistently regardless of solver-output parse success.
Proposed fix
Check the gold-label applicability before handling predictions, return NaN for no-control samples, and add regression coverage combining an undefined malformed sample with a normal valid recall sample.
Describe the bug
compute_ctrl_recall_posthoc()documents that samples with no control variables in the gold label are skipped because recall is undefined.That happens when solver output parses successfully, because
compute_recall(..., set())returnsNaN. But if the same sample has malformed solver output,predsbecomesNoneand the function instead sets:So a sample that should always be excluded from the average is counted as a zero only when parsing fails, which can incorrectly lower the reported post-hoc control recall.
Expected behavior
Samples whose gold hypothesis is invalid, or whose valid gold hypothesis contains no control variables, should be skipped consistently regardless of solver-output parse success.
Proposed fix
Check the gold-label applicability before handling predictions, return
NaNfor no-control samples, and add regression coverage combining an undefined malformed sample with a normal valid recall sample.