Span status mapping and long durations received by backend

[vitomedved]

Hey, first of all - thanks for the open sourced SDK. We have encountered 2 challenges on which hopefully you could give us some clarity.

I understand these are generic questions since you don't see the actual implementation, I'm trying to pinpoint the issue. If you need more implementation details feel free to ask.

1. Span status

Please correct me if I'm wrong, but as far as I can see on one side on Android we are unable to set the EmbraceSpan status explicitly to Ok so all our spans on backend are received as either Error or Unset. On the other side, on iOS SDK spans have status code set to Ok automatically and are received on backend side with either status code Ok or Error.

If this is correct observation, is there any specific reason why there is missalignment in the two platform implementations? Can we somehow override status on Android side to have consistend mapping on the backend?

Will Android SDK ever send status Ok for some spans?

2. Span duration

• We are trying to measure a certain screen load time
• We start the span in the activity's onCreate method
• The span is started with autoTerminationMode = AutoTerminationMode.ON_BACKGROUND (to be sure it's closed automatically)
• We stop the span when either data is rendered on the screen, error is shown, or the activity's onPause is called (in case auto termination fails for some reason)

With this implementation, on the backend we see some outliers with spans (5-10 spans per week) lasting from couple of hours up to 8 hours (which is weird because users would have to be in the application doing nothing for couple of hours).

• Is there any known issue/limitation related to very long span durations reported by the Embrace SDK?
• Will auto termination ever fail?
• Does Embrace SDK somehow cache spans that are not sent in a session?

Any other additional info on the problem with long spans is appreciated.

[bidetofevil]

Regarding span status; the recommendation OTel is to for instrumentation to not explicitly set the status to OK upon completion, but that fits more for the distributed tracing work than for mobile apps. The misalignment was an oversight - Android had changed the default to align with OTel but iOS never got to it. Will will resolve this in the future, likely allowing whether the status is set upon successful completion to be configurable.

CC @ArielDemarco

[bidetofevil]

For span duration:

• No, there's no problem with spans being very long. Things get trickier when it's over 24 hours, but that's more an issue on the server side and how it's presented and processed. The SDK should not care - as long as a process is active, it's OK
• Auto termination will fail if the app process is terminated before the app backgrounds. But a Span Snapshot will be created and you will be able to see that in the crashed session
• Spans that last through a session will be sent in the session they actually finish it. So if you have a session, start a span, background, start another session, then end the span, the span will be sent in the payload of the second session.

Hopefully that clears things up.

[vitomedved]

Appreciate the detailed answers. That does clear some things up, however what I was aiming at is following example where we are trying to measure screen load time with spans:

• We start a span when Android fragment is created in onCreate
• We stop the span when either:
  • Error happened while fetching data
  • Data is rendered on screen successfully
  • Fragment's onPause is called (failsafe for when user navigates away from fragment when data was never loaded)
• We additionally set the Embrace SDK's AutoTermination mode to ON_BACKGROUND to make sure that span gets stopped (additional failsafe measure)

With above implementation, we are seeing some screen load spans lasting for hours which does not sound right since user would need to be on the same fragment for hours without navigating away from it. Therefore we are trying to pinpoint where do these long-lasting spans come from.

What I'm still uncertain about is:

• Does stopping a span mark it's end time immediately (so it does not matter when span is sent, it's end time was marked when we called stop method)?
• What happens to spans that were never stopped manually? When is their end time?
• If the app crashes, what is span's end time? Is it time of the crash?
• Auto termination will fail if the app process is terminated before the app backgrounds - what happens with end time of the span?

[bidetofevil]

Sorry for the late reply, @vitomedved. To answer you questions:

• Yeah, stopping a span ends it at that time - or whatever timestamp you pass in. When it's delivered is irrelevant
• Spans that are never stopped explicitly will end implicitly when the app process terminates with a failed status.
• When an app crashes due to uncaught JVM exception or error, any on-going spans will be stopped explicitly, again with a failed status. With native crashes, it behaves similarly, the mechanism is closer to process termination as we don't have a chance to explicitly stop it.
• If a span is set to auto terminate, but the process terminates before backgrounding happens, we use the last timestamp of the session as the end time, much like what happens when there's a native crash.

Basically, it means that if a span is set to auto terminate on background, but it lasts for hours, either the backgrounding never happened, or there is a bug in the auto termination logic.

In the instances where the screen load seemingly lasts for hours, are there other telemetry around that time or in that session that could give you a clue as to what might have happened? Could that fragment's load been interrupted somehow so that onPause never fires?