Skip to content

Latest commit

 

History

History
335 lines (282 loc) · 7.47 KB

Evaluator.md

File metadata and controls

335 lines (282 loc) · 7.47 KB

TFSimilarity.callbacks.Evaluator

Evaluates search index performance and calibrates it.

Inherits From: ABC

Index evaluators are derived from this abstract class to allow users to override the evaluation to use additional data or interface with existing evaluation system. For example allowing to fetch data from a remote database.

Methods

calibrate

View source

calibrate(
    target_labels: Sequence[int],
    lookups: Sequence[Sequence[Lookup]],
    thresholds_targets: MutableMapping[str, float],
    calibration_metric: <a href="../../TFSimilarity/callbacks/ClassificationMetric.md">TFSimilarity.callbacks.ClassificationMetric```
</a>,
    matcher: Union[str, <a href="../../TFSimilarity/callbacks/ClassificationMatch.md">TFSimilarity.callbacks.ClassificationMatch```
</a>],
    extra_metrics: Sequence[<a href="../../TFSimilarity/callbacks/ClassificationMetric.md">TFSimilarity.callbacks.ClassificationMetric```
</a>] = [],
    distance_rounding: int = 8,
    metric_rounding: int = 6,
    verbose: int = 1
) -> <a href="../../TFSimilarity/indexer/CalibrationResults.md">TFSimilarity.indexer.CalibrationResults```
</a>

Computes the distances thresholds that the classification must match to meet a fixed target.

Args
target_labels Sequence of expected labels for the lookups.
lookup Sequence of lookup results as produced by the Index.batch_lookup() method.
thresholds_targets classification metrics thresholds that are targeted. The function will find the closed distance value.
calibration_metric Classification metric used for calibration.
matcher 'match_nearest', 'match_majority_vote' or ClassificationMatch object. Defines the classification matching, e.g., match_nearest will count a True Positive if the query_label is equal to the label of the nearest neighbor and the distance is less than or equal to the distance threshold.
extra_metrics Additional metrics that should be computed and reported as part of the classification. Defaults to [].
distance_rounding How many digit to consider to decide if the distance changed. Defaults to 8.
metric_rounding How many digit to consider to decide if the metric changed. Defaults to 6.
verbose Be verbose. Defaults to 1.
Returns
CalibrationResults containing the thresholds and cutpoints Dicts.

evaluate_classification

View source

evaluate_classification(
    query_labels: <a href="../../TFSimilarity/callbacks/IntTensor.md">TFSimilarity.callbacks.IntTensor```
</a>,
    lookup_labels: <a href="../../TFSimilarity/callbacks/IntTensor.md">TFSimilarity.callbacks.IntTensor```
</a>,
    lookup_distances: <a href="../../TFSimilarity/callbacks/FloatTensor.md">TFSimilarity.callbacks.FloatTensor```
</a>,
    distance_thresholds: <a href="../../TFSimilarity/callbacks/FloatTensor.md">TFSimilarity.callbacks.FloatTensor```
</a>,
    metrics: Sequence[<a href="../../TFSimilarity/callbacks/ClassificationMetric.md">TFSimilarity.callbacks.ClassificationMetric```
</a>],
    matcher: Union[str, <a href="../../TFSimilarity/callbacks/ClassificationMatch.md">TFSimilarity.callbacks.ClassificationMatch```
</a>],
    distance_rounding: int = 8,
    verbose: int = 1
) -> Dict[str, np.ndarray]

Evaluate the classification performance.

Compute the classification metrics given a set of queries, lookups, and distance thresholds.

Args
query_labels Sequence of expected labels for the lookups.
lookup_labels A 2D tensor where the jth row is the labels associated with the set of k neighbors for the jth query.
lookup_distances A 2D tensor where the jth row is the distances between the jth query and the set of k neighbors.
distance_thresholds A 1D tensor denoting the distances points at which we compute the metrics.
metrics The set of classification metrics.
matcher 'match_nearest', 'match_majority_vote' or ClassificationMatch object. Defines the classification matching, e.g., match_nearest will count a True Positive if the query_label is equal to the label of the nearest neighbor and the distance is less than or equal to the distance threshold.
distance_rounding How many digit to consider to decide if the distance changed. Defaults to 8.
verbose Be verbose. Defaults to 1.
Returns
A Mapping from metric name to the list of values computed for each distance threshold.

evaluate_retrieval

View source

evaluate_retrieval(
    target_labels: Sequence[int],
    lookups: Sequence[Sequence[Lookup]],
    retrieval_metrics: Sequence[<a href="../../TFSimilarity/indexer/RetrievalMetric.md">TFSimilarity.indexer.RetrievalMetric```
</a>],
    distance_rounding: int = 8
) -> Dict[str, np.ndarray]

Evaluates lookup performances against a supplied set of metrics

Args
target_labels Sequence of the expected labels to match.
lookups Sequence of lookup results as produced by the Index().batch_lookup() method.
retrieval_metrics Sequence of RetrievalMetric() to evaluate lookup matches against.
distance_rounding How many digit to consider to decide if the distance changed. Defaults to 8.
Returns
Dictionary of metric results where keys are the metric names and values are the metrics values.