Analyze PPTSS sampling behavior

[wrigleyDan]

The PPTSS sampler stops when an already sampled query is sampled repeatedly.

This happens especially in small scale scenarios (less than 100 unique queries in ubi_queries index).

We want to understand if this behavior is correct and if there are ways to handle such situations better, for example by falling back to the "all" sampling method.

Use ubi_queries data in a Python notebook example to see if this behavior can be reproduced outside of the eval app.

Example configuration for creating a query set.

{
  "sampler": "pptss",
  "name": "a new pptss sampled set",
  "description": "",
  "sampling": "",
  "querySetSize": 10
}

[wrigleyDan]

Analysis findings

Loading the data into a Python notebook showed different behavior than the Java app: In the notebook-based implementation it happens constantly that less than the specified amount of queries is sampled. While this is similar to the Java-based implementation the notebook constantly samples more queries than in the Java-based application.

The one missing piece in the PPTSS Java implementation of the Search Eval App seems to be calculating the cumulative weights after calculating the normalized weights. This results in some queries being picked very frequently, others not at all, so the Java app terminates early more often and earlier than expected.

[wrigleyDan]

Here's a table showing the individual steps for three queries:

Query	Frequency	Total Number of Queries	Normalized Frequency(=Frequency/Total)	Normalized Frequency(Result)	Cumulative Weight (Sum Explained)	Cumulative Weight (Result)
laptop	10	10+3+2=15	10/15	0.67	0.67	0.67
smartphone	3	10+3+2=15	3/15	0.20	0.67+0.20	0.87
wireless earbuds	2	10+3+2=15	2/15	0.13	0.67+0.20+0.13	1.00

The cumulative weights are the ones missing in the Java app is my claim.