Skip to content

Allow user to provide output/qrel file names with dataset retrieval#339

Merged
lilyjge merged 2 commits intocastorini:mainfrom
lilyjge:retrieval-dataset-enable-files
Feb 8, 2026
Merged

Allow user to provide output/qrel file names with dataset retrieval#339
lilyjge merged 2 commits intocastorini:mainfrom
lilyjge:retrieval-dataset-enable-files

Conversation

@lilyjge
Copy link
Member

@lilyjge lilyjge commented Feb 8, 2026

Pull Request Checklist

Following a discussion with @sahel-sh :

There are two modes for retrieval 1- dataset name+ retrieval method 2- requests.jsonl file which bypasses retrieval.

in mode 1, the proper qrels file is used based on the combo of the dataset name. and the other three output files are not needed, since the names for them are calculated in https://github.com/castorini/rank_llm/blob/main/src/rank_llm/rerank/reranker.py#L96C9-L96C29.
In absence of retreival method and other info in mode 2, we need to get these names from users.

This PR allows the user to pass in names for output or qrels files in mode 1, and if they're provided, they'll be used. Otherwise, fallback to the current, default values for these files (for output files, constructed file names; and for qrels, the ones mapped to the dataset).

Reference Issue

Please provide the reference to issue this PR is addressing (# followed by the issue number). If there is no associated issue, write "N/A".

ref: N/A

Checklist Items

Before submitting your pull request, please review these items:

  • Have you followed the contributing guidelines?
  • Have you verified that there are no existing Pull Requests for the same update/change?
  • Have you updated any relevant documentation or added new tests where needed?

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no API changes)
  • Documentation content changes
  • Reproduction logs
  • Other...
    • Description:

@lilyjge lilyjge requested a review from sahel-sh February 8, 2026 03:29
Copy link
Member

@sahel-sh sahel-sh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks

@lilyjge lilyjge merged commit 44929f2 into castorini:main Feb 8, 2026
5 checks passed
@lilyjge lilyjge deleted the retrieval-dataset-enable-files branch February 8, 2026 03:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants