Reorder

[XKTZ]

This is a superset of issue Top Down. This PR reorganized the various reordering methods including sliding window, top down, as well as ListT5's tournament sort methodology. Now it is allowed to use command like --reorder_policy="top_down:{\"top_k\": 10, \"pivot\": ${PIVOT}, \"shuffle\": true, \"r\": 1}" to specify a reorderer.

What changes

Incompatibility

• In RankListwiseOSLLM, the create prompt now returns a JSON string, then in run llm batched, it would parse it into a passage with type { prompt: string, num_passages: int }. This is for backward compatibility with the old data types of prompt (str | list[dict[str,str]]) while we need num passages in a single trial.

Reorder Update

• Now, the reorder and reranker are splitted for listwise_rankllm. That means, sliding_window(_batched) and permutation_pipeline shall be deprecated. Instead, we can define a class called ReorderPolicy to perform the reranking. So this is a small (simplified) definition on what an reorder policy is

class ReorderPolicy where
    reorder :: ([Requests], LLMReranker) -> [Result]

For detailed definition, please check src/rank_llm/rerank/listwise/reorder package

Now, we have three reorder policies, Sliding Window (sliding_window), tournament sort (tournament_sort) and top down (top_down). To add a reorder policy in future, create such a class extends Reorder Policy with a name, and put your class into listwise_rankllm.py's SUPPORT_REORDER_POLICIES.

To pass a parameter into reorder policy, append a JSON, separated by : when passing into the command line arguments reorder_policy, an example is following:

--reorder_policy="top_down:{\"pivot\": 11, \"top_k\": 10}"

Notice that to convenience, the JSON string behind : will be processed with json-repair library. So the string might not neet to be very JSON. (A javascript version should work, so you don't need to add a lot of \", using single quote ' should also work, so you don't need to put \" instead only need \')

So in above case, you actually only need to pass

--reorder_policy="top_down:{pivot: 11, top_k: 10}"

The JSON will be parsed as **json_dict to pass into the policy's class

Pointwise LLM

Adjust Pointwise LLM to compatible with the type usage right now (Not MonoT5, MonoT5's type def is still not corresponding with base class')

Experiments

Transformers

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmpr21z6o83', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/rank_zephyr_7b_v1_full_4096_100_rank_GPT_dl20_slidingwindow_stp10_2025-01-03T22:33:59.620645_window_20.txt']
Results:
ndcg_cut_10           	all	0.8211

VLLM

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --vllm_batched
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmpx7i37yy_', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/rank_zephyr_7b_v1_full_4096_100_rank_GPT_dl20_slidingwindow_stp10_2025-01-03T21:59:23.872518_window_20_vllm.txt']
Results:
ndcg_cut_10             all     0.8065

SGLang

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --sglang_batched
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmpacfdy6ub', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/rank_zephyr_7b_v1_full_4096_100_rank_GPT_dl20_slidingwindow_stp10_2025-01-03T21:54:34.718335_window_20_sglang.txt']
Results:
ndcg_cut_10             all     0.8085

TensorRT

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --tensorrt_batched
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmpf0t2uee6', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/rank_zephyr_7b_v1_full_4096_100_rank_GPT_dl20_slidingwindow_stp10_2025-01-04T20:28:13.770028_window_20_tensorrt.txt']
Results:
ndcg_cut_10             all     0.8113

VLLM, with top down policy

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --vllm_batched --reorder_policy="top_down:{pivot: 11, top_k: 10}"
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmp48wzevjb', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/rank_zephyr_7b_v1_full_4096_100_rank_GPT_dl20_topdown_tpk10_pvt11_2025-01-04T00:38:58.718882_window_20_vllm.txt']
Results:
ndcg_cut_10             all     0.7968

VLLM, with tournament sort policy

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/rank_zephyr_7b_v1_full --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --vllm_batched --reorder_policy="tournament_sort:{top_k: 10, r: 1}"
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmpcznevsch', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/rank_zephyr_7b_v1_full_4096_100_rank_GPT_dl20_tournamentsort_tpk10_r1_2025-01-04T00:52:37.214645_window_20_vllm.txt']
Results:
ndcg_cut_10             all     0.8012

VLLM, with ** FIRST ** logit's method, and sliding window

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/first_mistral --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --use_logits --use_alpha --vllm_batched --num_gpus 1
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmp6tzqimq3', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/first_mistral_4096_100_rank_GPT_dl20_slidingwindow_stp10_2025-01-04T01:02:51.749029_window_20_vllm.txt']
Results:
ndcg_cut_10             all     0.7885

VLLM, with ** FIRST ** logit's, and top down

python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/first_mistral --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_mode=rank_GPT  --context_size=4096 --variable_passages --use_logits --use_alpha --vllm_batched --num_gpus 1 --reorder_policy="top_down:{top_k: 10, pivot: 11}"
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmpttqpc9ew', 'rerank_results/SPLADE_P_P_ENSEMBLE_DISTIL/first_mistral_4096_100_rank_GPT_dl20_topdown_tpk10_pvt11_2025-01-04T01:35:06.172132_window_20_vllm.txt']
Results:
ndcg_cut_10             all     0.7856

MonoT5

python src/rank_llm/scripts/run_rank_llm.py --model_path=castorini/monot5-3b-msmarco-10k --top_k_candidates=100 --dataset=dl19   --retrieval_method=bm25 --prompt_mode=monot5 --context_size=512 --batch_size=16
Running command: ['java', '-jar', '/home/yidic/.cache/pyserini/eval/jtreceval-0.0.5-jar-with-dependencies.jar', '-m', 'ndcg_cut.10', '/tmp/tmpxcjujitq', 'rerank_results/BM25/monot5-3b-msmarco-10k_512_100_monot5_dl19_2025-01-04T21:09:53.071527_window_20.txt']
Results:
ndcg_cut_10             all     0.7174

[ronakice]

Remove

[ronakice]

step and window are different?

[ronakice]

@XKTZ what is this?

[ronakice]

I don't think this is the first instance of tournament sort, something has probably been done before, maybe we drop it for now