WARNING: No Threshold found!

[tpluscode]

I used bench-node to create Bencher JSON with latency and throughput measures. The latter has only value if that's relevant.

In Bencher report I see that there is no threshold for throughput... but there is. At least for master branch. I since uploaded a report from branch ts and a third threshold appeared - only for the latency...

I also notice no model set for the throughput threshold. I edited that manually and I think the warning disappears when I open a report from master online. Adding a threshold manually for the ts branch does not seems to have changed anything.

Feels like I'm doing something wrong here

[epompeii]

@tpluscode you are only creating a singe Threshold (latency) for the baseline on main:

          --threshold-measure latency \
          --threshold-test t_test \
          --threshold-max-sample-size 64 \
          --threshold-upper-boundary 0.99 \

https://github.com/zazuko/rdf-validate-shacl/blob/5ed9c4154d262e0657839093b4db6ca7341571fd/.github/workflows/benchmarks-baseline.yml#L28-L31

In order to create an additional Threshold (throughput), you need to add something like this directly below it:

          --threshold-measure throughput \
          --threshold-test t_test \
          --threshold-max-sample-size 64 \
          --threshold-upper-boundary 0.99 \

See the multiple thresholds section of the docs.

Note, the old Reports will still warn that no Threshold is found, even if one now exists as that warning is generated based on the data within the Report itself.

[tpluscode]

Ah, that's the penalty for blind copy-pasta without comprehension. You suggestion worked, of course.

That still leave a weird impression that bencher suggests adding thresholds in the UI but it's actually necessary (?) to set them in the CLI call. Or is it?

[epompeii]

That still leave a weird impression that bencher suggests adding thresholds in the UI but it's actually necessary (?) to set them in the CLI call. Or is it?

Historically, the UI and API were the only ways to create a Threshold. It wasn't until relatively recently that it was possible to create a Threshold using bencher run. The pointer to the docs is there for an explanation on the CLI flags route.