Package
lexical-graph
Problem statement
Allow users to benchmark their own custom-designed retrievers alongside the built-in ones (traversal, agentic, semantic_guided, etc.) by providing either:
- A Python module path to a script/class that implements the retriever interface, or
- A formal interface/protocol that custom retrievers must implement
Related to PR #345 (see :#345 (comment)) by @aghassel.
Proposed solution
- Define a CustomRetriever protocol/interface — A minimal contract requiring a query(question: str) -> Response method (matching what run_benchmark_query expects), plus an optional get_config() -> dict for recording hyperparameters in metrics_summary.json.
- Accept a custom:<module_path> retriever ID — e.g. BENCHMARK_RETRIEVER=custom:my_module.MyRetriever. The factory would dynamically import the class and instantiate it with graph_store and vector_store.
- Record custom retriever config — get_retriever_config() should handle the custom case by calling the retriever's get_config() if available, or recording the module path and any constructor kwargs.
Alternatives considered
Acceptance Criteria
Package
lexical-graph
Problem statement
Allow users to benchmark their own custom-designed retrievers alongside the built-in ones (traversal, agentic, semantic_guided, etc.) by providing either:
Related to PR #345 (see :#345 (comment)) by @aghassel.
Proposed solution
Alternatives considered
Acceptance Criteria