Is your feature request related to a problem? Please describe.
In the TMLR publication by Lin et al., 2024, the authors propose using three NLI-based confidence scores. Given a matrix of NLI-estimated entailment scores over sampled responses to the same prompt, they compute the following:
- Sum of Eigenvalues of the Graph Laplacian
- Trace of the Degree matrix
- Eccentricity
Describe the solution you'd like
Add these scoring methods to BlackBoxUQ. To enable this, they should implemented via uqlm.black_box.ConsistencyScorer. Note that computing the full matrix of NLI scores over sampled responses requires more NLI comparisons than our existing NLI-based scorers (semantic_negentropy, semantic_sets_confidence, entailment, noncontradiction). With this in mind, we should be careful to avoid redundant NLI computations if multiple NLI-based scorers are requested.
Additional context
Please ensure you have carefully read and fully understand the paper before volunteering to work on this issue.
Is your feature request related to a problem? Please describe.
In the TMLR publication by Lin et al., 2024, the authors propose using three NLI-based confidence scores. Given a matrix of NLI-estimated entailment scores over sampled responses to the same prompt, they compute the following:
Describe the solution you'd like
Add these scoring methods to
BlackBoxUQ. To enable this, they should implemented viauqlm.black_box.ConsistencyScorer. Note that computing the full matrix of NLI scores over sampled responses requires more NLI comparisons than our existing NLI-based scorers (semantic_negentropy,semantic_sets_confidence,entailment,noncontradiction). With this in mind, we should be careful to avoid redundant NLI computations if multiple NLI-based scorers are requested.Additional context
Please ensure you have carefully read and fully understand the paper before volunteering to work on this issue.