Is your feature request related to a problem? Please describe.
We would like to add Kernel Language Entropy Nikitan et al., 2024 to our available scorers. This method defines positive semidefinite unit trace kernels to encode the semantic similarities of LLM outputs and quantifies uncertainty using the von Neumann entropy. It considers pairwise semantic dependencies between answers (or semantic clusters), providing more fine-grained uncertainty estimates than previous methods based on hard clustering of answers.
Describe the solution you'd like
This should be a new black-box scorer, exposed via BlackBoxUQ. A new class called KernelLanguageEntropy should be added to uqlm.black_box.
Additional context
Please ensure you have read and fully understand the paper before requesting to work on this issue.
Is your feature request related to a problem? Please describe.
We would like to add Kernel Language Entropy Nikitan et al., 2024 to our available scorers. This method defines positive semidefinite unit trace kernels to encode the semantic similarities of LLM outputs and quantifies uncertainty using the von Neumann entropy. It considers pairwise semantic dependencies between answers (or semantic clusters), providing more fine-grained uncertainty estimates than previous methods based on hard clustering of answers.
Describe the solution you'd like
This should be a new black-box scorer, exposed via
BlackBoxUQ. A new class calledKernelLanguageEntropyshould be added touqlm.black_box.Additional context
Please ensure you have read and fully understand the paper before requesting to work on this issue.