query: expose partial response header

[func25]

Is your feature request related to a problem? Please describe

VictoriaLogs supports partial responses. These partial responses can be misleading, as they may contain critical logs needed for troubleshooting. We need a way to expose this information and help the frontend detect when a response is partial.

Describe the solution you'd like

Expose VL-Partial-Response header

Describe alternatives you've considered

Alternative options:

• Use an HTTP trailer, but @valyala rejected it due to poor support
• Perform a pre-check before querying
• Not supported in streaming, only available in /hits, /field_names, etc
• Add a sentinel log at the end

[makasim]

Have you considered the approach used by VictoriaMetrics, where every JSON response includes an "isPartial": bool property at the root level to indicate whether the response is partial?

It might be worth considering this option, as it could help unify mental models across logs and metrics, reducing the need to remember separate conventions.

[func25]

It's okay for a non-streaming endpoint, but ndjson streaming does not fit the case (one json object per line as logs arrive, there’s no root object to host "isPartial"). The header I propose is ideal, but it's also not reliable in streaming since the header is sent once the first byte of the body is transmitted.

[valyala]

VictoriaLogs provides two types of querying APIs:

1. APIs, which prepare the full response in memory before sending it to the client. These APIs may reliable detect whether the response is full or partial before sending the response to the client, since they already performed requests to all the backends listed in the -storageNode command-line flag before sending the response to the client. So they can set the VL-Partial-Response: true or VL-Partial-Response: false in the response header before sending the response to the client.
2. APIs, which start sending the response to the client in streaming mode as soon as they receive the first portion of data from any backend listed in the -storageNode command-line flag. This is what /select/logsql/query API does if the limit query arg isn't passed in the request to this API. In this case it is impossible to detect reliably whether the response is partial or not before sending the first byte of data to the client, since requests to some -storageNode backends may not be started yet. This also may lead to issues similar to chunk hex-length char not a hex digit: 0xa #738 when vlselect detects that some of the backends is unavailable after the first response byte is already sent to the client. Let's send VL-Partial-Response: unknown response header in this case.

The main potential users of the VL-Partial-Response header, which will benefit from this header the most, are built-in web UI for VictoriaLogs and VictoriaLogs plugin for Grafana. They could show a warning to users when VictoriaLogs returns partial responses. Good news are that both these users do not use the second type of APIs mentioned above - they always pass the limit query arg to /select/logsql/query endpoint - and this disables response streaming.

Implementation directions: the most straightforward way would be to add IsPartial field into the QueryStats struct. This struct is available after query execution at querying API handlers, so they can reliably and uniformly use the QueryStats.IsPartial field for setting the VL-Partial-Response header.

[manute]

Hey , just created this initial PR #884 for that, and I have some q:

• All the fields in QueryStats are uint64 , is ok to add there the IsPartial bool?
• Added the field as atomic.Bool, is that needed?
• and wdyt about the swap here https://github.com/VictoriaMetrics/VictoriaLogs/pull/884/files#diff-732a61e789967536259323b5a8a5313da0d923161294e6590e9c236894d0e380R83

Thanks!