Skip to content

[FEATURE] Support custom/external retrievers in benchmarking harness #346

Description

@acarbonetto

Package

lexical-graph

Problem statement

Allow users to benchmark their own custom-designed retrievers alongside the built-in ones (traversal, agentic, semantic_guided, etc.) by providing either:

  1. A Python module path to a script/class that implements the retriever interface, or
  2. A formal interface/protocol that custom retrievers must implement

Related to PR #345 (see :#345 (comment)) by @aghassel.

Proposed solution

  1. Define a CustomRetriever protocol/interface — A minimal contract requiring a query(question: str) -> Response method (matching what run_benchmark_query expects), plus an optional get_config() -> dict for recording hyperparameters in metrics_summary.json.
  2. Accept a custom:<module_path> retriever ID — e.g. BENCHMARK_RETRIEVER=custom:my_module.MyRetriever. The factory would dynamically import the class and instantiate it with graph_store and vector_store.
  3. Record custom retriever config — get_retriever_config() should handle the custom case by calling the retriever's get_config() if available, or recording the module path and any constructor kwargs.

Alternatives considered

Acceptance Criteria

  • A documented protocol/ABC that custom retrievers must satisfy
  • retriever_factory.py supports a custom:<module_path> (or similar) retriever ID
  • Custom retriever hyperparameters are recorded in metrics_summary.json
  • Example showing how to write and benchmark a custom retriever
  • Existing built-in retrievers continue to work unchanged

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions