Improving Request Explanation & Debugging in Microcks

[Apoorva64]

Currently, when working with Microcks, it can be challenging to understand why a request was matched (or not matched) and to debug issues during dispatching.

This leads to several pain points:

• Difficulty understanding the reasoning behind request/response matching.
• No visibility into which request has been executed by Microcks.
• Dispatcher logs and errors are hidden in backend logs instead of being exposed in the UI.
• Debugging custom dispatchers or scripts requires external log inspection.
• No straightforward way to identify requests that did not match any mock.

Related issues:

• [Issue #895](Add explicit error message in mock response #895) about returning matching feedback to the user.
• [PR #1237](Log request order #1237) introducing custom per-request logging.

Example Use Case

Imagine a service that calls multiple mocked dependencies through Microcks. If one of those dependencies is not mocked in Microcks, today there is no simple way to identify this missing mock. From the UI, it’s not obvious which requests were received, which were matched, and which were ignored or failed. Developers are forced to inspect backend logs to track down the problem.

Another example would be if we are using SCRIPTS. In the Dispatcher scripts we do have ways to log, but these are not yet surfaced in a user-friendly way through the Microcks UI.

Proposal

Introduce a Request Explanation / Debug Panel within Microcks, supported by a lightweight per-request tracing system, that would:

• Expose dispatcher logs and script errors directly in the UI.
• Clearly show which request was executed by Microcks.
• Allow developers to follow the dispatcher’s decision-making process step by step.
• Provide clear feedback when a request fails to match or when no mock exists.

Analogy: In SQL, the EXPLAIN command allows developers to understand how a query is executed by showing the query plan. Similarly, Microcks could provide an EXPLAIN for requests, showing how the engine tried to match the request, which rules or dispatchers were applied, and why a particular response (or no response) was selected.

Possible Implementation Direction

One option would be to leverage OpenTelemetry for this per-request tracing:

• Each incoming request could get a correlation ID / trace ID.
• Dispatcher decisions, logs, and errors could be attached as spans or events.
• Traces could be viewed directly in Microcks UI (for quick debugging) or exported to an external tracing system like Jaeger/Tempo/Zipkin for deeper analysis.

I have already done an exploratory POC using OpenTelemetry tracing, which you can see in this short demo video

https://www.youtube.com/watch?v=BmjMDK4pUIQ

The code for this exploration is available here: StartUpNationLabs/microcks – tracing branch. But this is for demonstration purposes only, the code is not pretty xD

[Apoorva64]

@lbroudoux, @yada what are your thoughts on this idea?

[lbroudoux]

Hey @Apoorva64

Thanks again for putting all of this together. I would have liked to address a lot of topics and great stuff!

First, on the pain points: I totally agree with the bullet points - finding why things are not working as expected is not easy these days, especially since we expanded the number of features and dispatchers this last year... So it's a shared situation and a bug +1 from my point of view.

I like the EXPLAIN analogy, and I think it's the correct one. An early prototype that I never finished was, in fact, looking for an x-microcks-explain header in the incoming request to collect the Log4j logs and return them as response headers to provide the caller with an explanation of the returned response. But it was before our OpenTelemetry implementation, and I think this approach might be better now.

Your POC and demo are awesome regarding the feature implementation, and I have some usability comments. However, my first question here is regarding the OpenTelemetry collection: Would this collection be an alternative to another collector, or could they work together? In a nutshell, would it be possible to use both the "microcks internal collector" AND another external one like the LGTM stack or Jaeger or whatever? Is it an OR or an XOR?

Now, regarding the usability comments:

• Don't you think we could embed the "Live traces" panel into the Service properties page (where you edit your dispatching rules)? It would allow for "live testing" of dispatcher updates and check the SCRIPT and our logic?
• I'm afraid adding "Live traces" in the service details page bloats the UI a bit... Have you think about having a similar (but simpler) page dedicated to that purpose? It could be a dedicated menu entry (where you access the graph for example) and could embed both service unit view and graph view on different tabs for example?
• Regarding the "Graph" display: I was expecting to provide a correlation identifier or span ID or client IP as the first step before having the graph displayed... How do you things behave when you have multiple users connected to the Microcks instance? Are all users represented by the "External User" node in the UI?

Thanks for priving insights on those points! I'm looking forward to learning more on this!

PS: I'll probably find answers by looking at the code but I don't have time at the moment. Will try to do that pretty soon!

[Apoorva64]

1. Would this collection be an alternative to another collector, or could they work together?
   The implementation hooks into the Opentelemetry System to collect the 'explain' traces and prcesses them with 'CustomExplainTraceProcessor'. The span are not modified and so the spans continue their journey in the otel lifecycle to an exporter is configurable to export to whatever we want (LGTM, Jaeger,etc...).

In short the implementation acts on part of otel that is before exporting traces so LGTM, Jaeger are not affected by the changes. But we need to replace the otel agent with an implementation directly in the java app or find a way to inject the spanProcessor into the agent to be able to collect the traces and store them in memory

3. Are all users represented by the "External User" node in the UI?
   Yes currently all users are represented with "External User". But we can easily implement grouping using the 'Server Span' Which contains the client IP

[lbroudoux]

Thanks for the explanation. I've got a follow-up question but putting it into a #1728 comment 😉

[SebastienDegodez]

I'm afraid adding "Live traces" in the service details page bloats the UI a bit... Have you think about having a similar (but simpler) page dedicated to that purpose? It could be a dedicated menu entry (where you access the graph for example) and could embed both service unit view and graph view on different tabs for example?

I totally understand your about "details page", it's a valid point, but personally, I feel that moving "Live traces" to a separate page might degrade the overall user experience, as it introduces more navigation steps and breaks the flow of service exploration.

What do you think about keeping it within the service details page, but using a collapsible accordion or a tab to display the trace data? That way, when "Live trace" is selected, we could show spans and traces while temporarily hiding other sections like "Dispatching properties" and "Mock".

I believe this feature adds value, but you need to be careful not to reimplement full observability tooling inside Microcks.

When external observability integration is enabled, we could consider redirecting users to the UI of the observability tool instead of displaying traces directly in Microcks.
This would keep the Microcks interface clean and focused, while still offering access to detailed trace data through the appropriate platform.

I think it's really valuable that Microcks show a trace ID (W3C compliant).
We should make it clickable (to copy it easily or to redirect users to the corresponding distributed trace URL)
1: "Copy"
2: "Observability" URL template is configured in Microcks
-> No= Click "open" accordeon or / tab.
-> Yes= Click "open" to show in another view

[lbroudoux]

I totally understand your about "details page", it's a valid point, but personally, I feel that moving "Live traces" to a separate page might degrade the overall user experience, as it introduces more navigation steps and breaks the flow of service exploration.

Got your point. I think it should be present both on "Service details" page and also on "Edit operation properties" one for rapid diagnostic/validation of dispatcher rules changes.

What do you think about keeping it within the service details page, but using a collapsible accordion or a tab to display the trace data? That way, when "Live trace" is selected, we could show spans and traces while temporarily hiding other sections like "Dispatching properties" and "Mock".

👍 +1 here!

When external observability integration is enabled, we could consider redirecting users to the UI of the observability tool instead of displaying traces directly in Microcks.
This would keep the Microcks interface clean and focused, while still offering access to detailed trace data through the appropriate platform.

Good point! It could be also a global enhancement to have a link to the observability tool/dashboard when the feature is enabled.

I think it's really valuable that Microcks show a trace ID (W3C compliant).
We should make it clickable (to copy it easily or to redirect users to the corresponding distributed trace URL)
1: "Copy"
2: "Observability" URL template is configured in Microcks
-> No= Click "open" accordeon or / tab.
-> Yes= Click "open" to show in another view

👍 +1 here!

Hey @Apoorva64! Do you think we'd be able to make progress on this this week? I'd like to include some of these features so that I can brag about them at my incoming Devoxx BE talk 😉

[Apoorva64]

yep xD, i'll make a quick task list of what must be done to integrate the POC into the app

[Apoorva64]

Overview Architecture

sequenceDiagram
  autonumber
  participant C as Client (HTTP)
  participant RC as RestController
  participant RIP as RestInvocationProcessor
  participant CDC as computeDispatchCriteria (child span)
  participant OTel as OpenTelemetry SDK
  participant P as CustomExplainTraceProcessor
  participant S as SpanStorageService
  participant E as Spring Event Bus
  participant SSE as TracingSseManager/Controller
  participant SC as Client (SSE)

  C->>RC: REST request /rest/{service}/{version}/...
  RC->>RC: Tag current span (explain-trace=true,<br/>service/operation attrs)
  RC->>RIP: processInvocation()
  RIP->>CDC: create INTERNAL span (explain-trace=true)
  CDC-->>RIP: dispatch criteria + events
  RIP-->>RC: ResponseResult (mock/proxy)
  RC-->>C: HTTP response

  note over RC,RIP: Spans finish after logic completes
  RC-->>OTel: End server span
  RIP-->>OTel: End internal spans
  CDC-->>OTel: End child span

  OTel-->>P: onEnd(span)
  alt span has attribute explain-trace
    P->>S: storeSpan(span)
    opt span is root (no parent)
      S->>E: publish SpanStoredEvent(traceId)
      E->>SSE: handle event (async)
      SSE->>S: getSpansForTrace(traceId)
      SSE-->>SC: SSE event "trace" with array<SpanData>
    end
  end

Loading

Responsibilities

Component	Role
RestController	Tags top-level span, resolves service/operation, invokes processing, exposes REST retrieval endpoints
RestInvocationProcessor	Creates child/internal spans, adds detailed events & attributes, dispatch logic
OpenTelemetry SDK	Lifecycle & context for spans, invokes registered SpanProcessor(s) on end
CustomExplainTraceProcessor	Filters finished spans for presence of explain-trace attribute and forwards them for storage
SpanStorageService	In-memory retention (1000 traces / 100 spans per trace), publishes event on root span arrival
Spring Event Bus	Asynchronous decoupling of storage from streaming layer via SpanStoredEvent
TracingSseManager	Manages subscriptions (serviceName+operationName keys), pushes full trace snapshot upon root span completion, heartbeat maintenance
SSE Client	Receives trace events (array of SpanData), may also query REST endpoints for historical retrieval

Data / Control Flow Highlights

1. REST request produces server span; components tag it (explain-trace=true).
2. Internal logic creates additional spans (also tagged) with rich events.
3. On span end, processor inspects attributes; qualifying spans added to per-trace buffer.
4. First root span end for a trace (or finalization) triggers SpanStoredEvent → streaming notification.
5. SSE manager fetches entire trace snapshot (current spans list) and emits one trace event (atomic view).
6. Capacity enforcement silently evicts oldest traces/spans; clients wanting durability must export early

Task List

• Move opentelemetry from Agent mode to included directly into the Spring ecosystem to be able to customize it
• Add custom spans and events to explain the inner workings of Microcks
• Add the Custom Span Processor and Storage to store and query 'explain' spans in memory
• Implement Event based 'notifications' on changes in the SpanStorageService
• Add TracingSseManager and expose the subscriptions and traces to the client
• Create an Otel spans parser for the frontend to map to the otel types
• Create a 'Live trace' Component and use them to add trace visualisation to the 'Service' page and 'Edit Properties'
• Create a new page to show the graph and a per service view

[Apoorva64]

@lbroudoux the first task may be a breaking change 'Move opentelemetry from Agent mode to included directly into the Spring ecosystem to be able to customize it'

[Apoorva64]

Should we create an EPIC to track all of this? -> #1728