Is your feature request related to a problem? Please describe.
Currently, our SemanticEntropy class only supports discrete semantic entropy, where cluster probabilities are computed based on cluster frequencies. We would like to enable continuous semantic entropy, where cluster probabilities are based on token probabilities instead. This new approach includes aspects of both black-box UQ and white-box UQ.
Describe the solution you'd like
An boolean parameter discrete that specifies whether to include discrete or continuous semantic entropy.
Describe alternatives you've considered
Only offering discrete semantic entropy.
Additional context
If discrete=True, we should check that the provided LLM supports token probability access, as we do for the WhiteBoxUQ class.
Note: Semantic Entropy methods are based on the work by Farquhar et al., 2024. Please only ask to work on this issue if you have read and fully understand the paper.
Is your feature request related to a problem? Please describe.
Currently, our
SemanticEntropyclass only supports discrete semantic entropy, where cluster probabilities are computed based on cluster frequencies. We would like to enable continuous semantic entropy, where cluster probabilities are based on token probabilities instead. This new approach includes aspects of both black-box UQ and white-box UQ.Describe the solution you'd like
An boolean parameter
discretethat specifies whether to include discrete or continuous semantic entropy.Describe alternatives you've considered
Only offering discrete semantic entropy.
Additional context
If
discrete=True, we should check that the provided LLM supports token probability access, as we do for theWhiteBoxUQclass.Note: Semantic Entropy methods are based on the work by Farquhar et al., 2024. Please only ask to work on this issue if you have read and fully understand the paper.