GPT Cache Paper Replicability

[daniyal-rana]

Hello,

I'm working on replicating the benchmark results from the GPT Cache paper.

I noticed that the default cache size within get_data_manager is 1000 and the default eviction strategy is LRU.

However, within examples/benchmark/benchmark_sqlite_faiss_onnx.py, the default cache size is overridden to 100,000.

Which configurations correspond to the tabular results in the paper?

Thank you for your time.

[SimFG]

[daniyal-rana]

Thank you for your response.

However, it is still ambiguous on what the cache size is (when eviction happens and whether you used LRU vs FIFO)?

When I attempted to replicate this experiment with caching 30,000 keys and randomly selecting 1000 samples, I had the following results:

• cache_hint_positive: 177
• hit_cache_negative: 814
• fail_count: 9

This seems to significantly deviate from the results shown in the table: Any suggestions on what we might need to change?

Configurations:

• Similarity threshold: 0.7
• Cache size: 100,000 (also tried 10,000)
• Eviction strategy: LRU
• similiar_qqp.json