lib/logstorage: add the ability to set a timestamp offset via options (#414)

Co-authored-by: Aliaksandr Valialkin <[email protected]>

Author: vadimalekseev

docs/victorialogs/CHANGELOG.md

@@ -18,6 +18,7 @@ according to [these docs](https://docs.victoriametrics.com/victorialogs/quicksta
 
 ## tip
 
+* FEATURE: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): add support for applying a global time offset to the `<q>` [query](https://docs.victoriametrics.com/victorialogs/logsql/#query-syntax) via `options(time_offset=<duration>) <q>` syntax. This is useful for comparing query results on some time range to query results on the same time range with the given offset (similar to the [`offset` modifier in PromQL](https://prometheus.io/docs/prometheus/latest/querying/basics/#offset-modifier)). See [these docs](https://docs.victoriametrics.com/victorialogs/logsql/#query-options) and [#78](https://github.com/VictoriaMetrics/VictoriaLogs/issues/78) for details.
 * FEATURE: [LogsQL](https://docs.victoriametrics.com/victorialogs/logsql/): speed up queries, which select big number of logs and end with [`| sort by (_time) desc offset M limit N` pipe](https://docs.victoriametrics.com/victorialogs/logsql/#sort-pipe). E.g. these queries allow building simple pagination over the selected logs. See [#96](https://github.com/VictoriaMetrics/VictoriaLogs/issues/96).
 
 ## [v1.27.0](https://github.com/VictoriaMetrics/VictoriaLogs/releases/tag/v1.27.0)

docs/victorialogs/LogsQL.md

@@ -338,6 +338,8 @@ It is possible to specify generic offset for the selected time range by appendin
 - `_time:2023-07Z offset 5h30m` matches logs on July, 2023 by UTC with offset 5h30m.
 - `_time:[2023-02-01Z, 2023-03-01Z) offset 1w` matches logs the week before the time range `[2023-02-01Z, 2023-03-01Z)` by UTC.
 
+See also [`time_offset` option](#query-options), which allows applying the given offset to all the filters on `_time` field without the need to modify the query.
+
 Performance tips:
 
 - It is recommended specifying the smallest possible time range during the search, since it reduces the amounts of log entries, which need to be scanned during the query.
@@ -4184,6 +4186,16 @@ VictoriaLogs supports the following options, which can be passed in the beginnin
   options(concurrency=2) _time:1d | count_uniq(user_id)
   ```
 
+- `time_offset` – subtracts the given offset from all the [time filters](#time-filter) in the query,
+  and then adds the given offset to the selected [`_time` field](https://docs.victoriametrics.com/victorialogs/keyconcepts/#time-field) values
+  before passing them to query [pipes](#pipes). Allows comparing query results for the same duration at different offsets.
+  Accepts any [duration value](#duration-values) like `12h`, `1d`, `1y`. For example, the following query returns the number of logs with `error` [words](#word)
+  over the last hour 7 days ago.
+
+  ```logsql
+  options(time_offset=7d) _time:1h error | stats count() as 'errors_7d_ago'
+  ```
+
 - `ignore_global_time_filter` - allows ignoring time filter from `start` and `end` args of [HTTP querying API](https://docs.victoriametrics.com/victorialogs/querying/#http-api)
   for the given (sub)query. For example, the following query returns the number of logs with `user_id` values seen in logs during December 2024, on the `[start...end]`
   time range passed to [`/api/v1/query`](https://docs.victoriametrics.com/victorialogs/querying/#querying-logs):

lib/logstorage/block_header.go

@@ -986,6 +986,13 @@ func (th *timestampsHeader) copyFrom(src *timestampsHeader) {
 	th.marshalType = src.marshalType
 }
 
+func (th *timestampsHeader) subTimeOffset(timeOffset int64) {
+	if timeOffset != 0 {
+		th.minTimestamp = subNoOverflowInt64(th.minTimestamp, timeOffset)
+		th.maxTimestamp = subNoOverflowInt64(th.maxTimestamp, timeOffset)
+	}
+}
+
 // marshal appends marshaled th to dst and returns the result.
 func (th *timestampsHeader) marshal(dst []byte) []byte {
 	dst = encoding.MarshalUint64(dst, th.blockOffset)

lib/logstorage/block_search.go

@@ -212,7 +212,6 @@ func (bs *blockSearch) search(ss *searchStats, bsw *blockSearchWork, bm *bitmap)
 	bs.reset()
 
 	bs.ss = ss
-
 	bs.bsw = bsw
 
 	// search rows matching the given filter
@@ -488,6 +487,19 @@ func (bs *blockSearch) getValuesForColumn(ch *columnHeader) []string {
 	return values.a
 }
 
+func (bs *blockSearch) subTimeOffsetToTimestamps(timeOffset int64) {
+	bs.bsw.bh.timestampsHeader.subTimeOffset(timeOffset)
+	if bs.timestampsCache != nil {
+		subTimeOffset(bs.timestampsCache.A, timeOffset)
+	}
+}
+
+func subTimeOffset(timestamps []int64, timeOffset int64) {
+	for i := range timestamps {
+		timestamps[i] = subNoOverflowInt64(timestamps[i], timeOffset)
+	}
+}
+
 // getTimestamps returns timestamps for the given bs.
 //
 // The returned timestamps belong to bs, so they become invalid after bs reset.
@@ -518,6 +530,7 @@ func (bs *blockSearch) getTimestamps() []int64 {
 	if err != nil {
 		logger.Panicf("FATAL: %s: cannot unmarshal timestamps: %s", bs.partPath(), err)
 	}
+
 	bs.timestampsCache = timestamps
 	return timestamps.A
 }