Filter http server spans that have 4xx http status in Node

[AbhiPrasad]

Problem Statement

Drop all 4xx root spans

Solution Brainstorm

Tasks

Preview Give feedback

1. Next.js (routes that do not exist are not instrumented)
2. Express
3. feat(node): Add shouldHandleError option to fastify error handler #13198
   Integration: fastify +1
   
4. Koa
5. Hapi
6. Nestjs integration reports 404 errors to sentry #12523
   Integration: nest.js Package: node Type: Bug +3
   
7. Remix
8. SvelteKit

[mydea]

So to add some details on this:

We cannot do this in the sampler or in any hook for the instrumentation, because at this point we don't know the status yet.
So really the only thing we can realistically do is detect a http span with a 404 status code in the span exporter and drop the transaction event with all children there. But this means that during the transaction we'll still propagate the "incorrect" sampling decision (because we do not know yet that we'll drop this eventually). Which is not really great, but I can't think of anything else to do this.

Is this an OK solution for us? Or would it possibly make more sense to do this in relay then, and just drop this there (or have a new inbound filter for this, for example)?

It would also be interesting to see how others handle this, e.g. is it normal to filter these out in other otel products or not so much cc @smeubank

[cleptric]

I would assume that in the majority of cases where an app emits a 404, very little application code is actually invoked. This leaves us with a much smaller subset of requests where the developer, somewhere in their application, emits a 404 on purpose. I think this is an OK trade-off to make.
We've been not sending any 404 transactions on Laravel for the last nine months, and so far, I haven't noticed a case where a trace was broken nor did we see any reports from users.

[MattJustMatt]

As a user I'd also like note that some amount of sampling of 4xx is useful for us. For example we throw 4xx for invalid auth headers and spikes here could alert us to upstream issues as well as providing telemetry on sign up issues (e.g. missing password validation requirements spiking might be a UX concern).

That said, I would love to be able to sample these independently.

[s1gr1d]

Thanks for sharing your use case! I think it would make sense to filter the 4xx codes by default and give you the possibility to configure which codes you still want to have enabled.

[xr0master]

As a user I'd also like note that some amount of sampling of 4xx is useful for us. For example we throw 4xx for invalid auth headers and spikes here could alert us to upstream issues as well as providing telemetry on sign up issues (e.g. missing password validation requirements spiking might be a UX concern).

You should use a log service instead of an issue service for this purpose. IMO