Create e2e integration tests for Adaptive Sampling

[yurishkuro]

We never had these tests properly setup. Now that Jaeger SDKs are deprecated we don't even know if there is a working example of adaptive sampling using OTEL SDKs. We need to set up integration tests to validate that the functionality is still working.

There are two possible ways we could validate adaptive sampling:

1. Start generating a load with a certain known rate and wait until adaptive sampling probability is adjusted from the default to a certain expected level. This is a bit tricky since the performance of the trace generator is unpredictable in GH Action runners, so we don't quite know how much traffic it would generate.
2. Instead of waiting to achieve an expected level of sampling, we could instead run two different workloads, e.g. 1x volume and then 2x volume. Even if we don't know the exact expected level of sampling probability, we can expect that it should become about 2x smaller once the load increases.

And even simpler alternative might be to just detect that the pre-configured default probability (which would be returned by the backend in the absence of any traffic for a given service) would change to a different calculated probability. For example, we could configure default probability as 100%, configure adaptive sampling target to be 1 trace per second, and run load generator with 10 traces per second. We would expect the calculated rate to go down to about 10% (that's the strategy 1 above), but even a change to 50% would already be an indication that adaptive sampling is working end to end. We don't need the e2e test to validate the exact calculation, since we do that in the unit tests.

[mahadzaryab1]

@yurishkuro question regarding your comment We would expect the calculated rate to go down to about 10%. How can we get what the calculated rate is ?

[yurishkuro]

@mahadzaryab1 we can query /sampling endpoint for the current sampling strategy (which is what the SDK does).

[mahadzaryab1]

@yurishkuro got it - thank you! i will give this issue a shot. are there any other tasks for this issue? do we want a docker-compose setup, readme, etc?

[yurishkuro]

You can see the discussion on the closed PR. I suggest writing up your plan first before jumping into implementation.

[mahadzaryab1]

@yurishkuro Here's what I'm thinking (all of the following steps being in a shell script):

1. Start jaeger backend with the following configs

  remote_sampling:
    adaptive:
      sampling_store: some_store
      initial_sampling_probability: 1.0
      target_samples_per_second: 1
    http:
    grpc:

2. Generate load using tracegen to send 10 traces per second (-pause 100ms duration=60m )
3. Query /sampling endpoint (with a max tries or timeout of some sort) until the sampling rate drops below a certain threshold, say 50%. If threshold drops, test passes. Otherwise, it fails.

[mahadzaryab1]

I was also wondering, what was the purpose of #5718? If the Jaeger backend is responsible for doing the sampling, why does tracegen need to know about the sampling endpoint?

[yurishkuro]

Jaeger backend is not doing sampling, it only calculates the desired sampling rate based on the observed traffic. The SDK in the application polls for that sampling rate.

[mahadzaryab1]

@yurishkuro Oh I see - so it's the client's responsibility to change the rate that it emits traces at? What's the reason for that instead of having Jaeger do the sampling?

[yurishkuro]

The SDK performs the sampling. If you instead delegate it to the backend it means you would have to collect traces for 100% of requests, which could be a huge overhead on the application's performance.

[mahadzaryab1]

@yurishkuro Ah okay I see! Is this the Jaeger SDK that you're talking about? Are these still being used for v2? I was under the impression that client side libraries were retired in favour of OTEL.

[mahadzaryab1]

Do you have any feedback on the plan I mentioned above?