Incorrectly combining data for chrome/chromium for certain subtests

[jandem]

For example:
https://arewefastyet.com/win10/benchmarks/raptor-desktop-jetstream2?numDays=60
https://arewefastyet.com/win10/benchmarks/raptor-desktop-speedometer?numDays=60

The first graph (total benchmark score) is okay and has separate data points for chrome and chromium. For the sub tests, we get a single data point for both. It looks like the data is incorrectly combined somewhere.

[davehunt]

@beatrice-acasandrei is going to look into this when she's back from PTO next week.

[jandem]

Ping? It would be great to have this fixed :-)

[davehunt]

Thanks for the ping. @beatrice-acasandrei any further progress getting to the bottom of this issue?

[beatrice-acasandrei]

The results from chrome and chromium are sharing the same signature hash. Since the logic is relying on the signature hash it is not able to differentiate between chrome and chromium results. Same issue is with the generated link for the Perfherder Graph View.
One workaround can be to go to Perfherder's Graph and add data manually for chromium from the 'Add test data' button.
I have upcoming PTO and we might take a while before we'll be able to fix this issue.

[davehunt]

It sounds like there may be an underlying bug in Perfherder that's not distinguishing the signatures for subtests for chrome vs chromium. It's also affecting the suite level value, although only when clicking the Perfherder link (both chrome/chromium are displayed but with a shared series). This seems to only affect the benchmark detail view (the overview appears to be fine). This suggests we might be able to resolve this in AWFY but I suspect we need to get to the root cause of the shared signature issue. With PTO affecting the team over the next couple of weeks it's unlikely we're going to have a fix soon. The recommended workaround is to use Perfherder graphs view and add the tests you're interested in.

[davehunt]

@jandem I've created a redash query that takes a platform and suite as parameters and builds a Perfherder graph link for all applications: https://sql.telemetry.mozilla.org/queries/89799. It's not perfect, as it includes Firefox for both autoland and mozilla-central, but it may be useful as a workaround for this issue?

[jandem]

@davehunt thanks for doing that! I'll give it a try :)

[davehunt]

I've improved the query to only show autoland for Firefox and only mozilla-central for non-Firefox.

[beatrice-acasandrei]

@jandem Hi! Does the query provided by Dave fulfill your needs for now? This issue is more complex than we initially anticipated and might take a while until it's fixed.

[jandem]

@beatrice-acasandrei I think that works for now. Maybe we can put a banner on the relevant pages to warn people about this bug, because the data is kind of misleading atm?