Skip to content

refactor: remove retrieval module from document-search package #444

Open
@micpst

Description

@micpst

Description

Remove rephrasers module from ragbits-document-search package and move rerankers to the ragbits-core.

Motivation

Currently retrieval module contains two things - rephrasers and rerankers. Most of the time we can't use provided rephrasers because the api is too limitted to do an effective rephrasing and this step ends up outside the document search pipeline. Rerankers are fine, but they are tied to the output of the document search and I can't decide what to put inside. Also the fact that this api is tied to the document search is kinda misleading, because I can rerank many things, not just documents. That's why moving it to the core seems most reasonable to me.

This idea also works well with the ragbits-piplines/ragbit-flow package, where we would put all the stuff inside the ragbits DAG, and it seems to be the best way to structure this retrieval workflow.

Metadata

Metadata

Assignees

Labels

needs discussionTask is under discussion or need more clarificationrefactorCode change that neither fixes a bug nor adds a feature

Type

No type

Projects

Status

Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions