app/vlagent/remotewrite: introduce VictoriaLogs transformations language

[vadimalekseev]

Documentation and syntax: https://github.com/VictoriaMetrics/VictoriaLogs/blob/vlt/docs/victorialogs/vlagent/transformations.md

Closes #858

[cubic-dev-ai]

3 issues found across 33 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found and verified against the latest diff

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[vadimalekseev]

These changes are not related to log transformations, but without them, my tests fail.

[vadimalekseev]

This code was written in the hope that it would be replaced with supporting relative time filters in VictoriaLogs transformations.

[cuongleqq]

Inside initTimestamps, if brSrc is not nil, it takes the timestamps from brSrc. So if our blockResult has an overwritten _time column, that column is ignored:

VictoriaLogs/lib/logstorage/block_result.go

Lines 632 to 636 in 2e51424

	func (br *blockResult) initTimestamps() {
	if br.brSrc != nil {
	srcTimestamps := br.brSrc.getTimestamps()
	br.initTimestampsInternal(srcTimestamps)
	return

In an if branch, when the filter matches only part of the block, a new blockResult is created with the brSrc set:

VictoriaLogs/lib/logstorage/transforms.go

Lines 558 to 559 in 2e51424

	brMatched.initFromFilterAllColumns(br, bmMatched)
	brUnmatched.initFromFilterAllColumns(br, bmUnmatched)

VictoriaLogs/lib/logstorage/block_result.go

Lines 141 to 149 in 1796949

	func (br *blockResult) initFromFilterAllColumns(brSrc *blockResult, bm *bitmap) {
	br.reset()
	br.rowsLen = bm.onesCount()
	if br.rowsLen == 0 {
	return
	}
	br.brSrc = brSrc

Combining both, with a filter like:

if (service:=payment) {
  time_add 1h;
}

on this data:

{"_time":"2026-07-01T10:00:00Z","service":"products","_msg":"a"}
{"_time":"2026-07-01T10:05:00Z","service":"products","_msg":"b"}
{"_time":"2026-07-01T10:00:00Z","service":"payment","_msg":"c"}
{"_time":"2026-07-01T10:05:00Z","service":"payment","_msg":"d"}

The payment rows keep their original _time instead of the one time_add produced

[vadimalekseev]

Thank you! Fixed.
br.getTimestamps now initializes timestamps if timestampBuf is empty (original behavior). Additionally, it parses the _time column if timestampBuf is not empty (initialized) but contains outdated values (isTime == false).

It looks like either I am misusing getTimestamps(), or the issue is also present in the original code. Could you please validate the new behavior?

[cuongleqq]

I checked the other callers of getTimestamps() - they all check c.isTime first before calling it. So getTimestamps() is only called when _time still holds the real timestamps. That is why this problem does not happen elsewhere.

About the new fix: time_add works now, but format / copy / rename ... as _time still do not. time_add fills timestampsBuf first to read time, so it's not empty. But other pipes only create new _time columns without reading _time first, so timestampsBuf is empty in these cases.

One solution to fix this could be to change initFromBlockResult. Maybe we only call br.getTimestamps() if the column _time is still isTime, or else we parse it? I am not very sure this is best in term of performance, but it will be correct since it just copies from how other pipes do it before calling getTimestamps().

[cuongleqq]

Minor, not a change request: this name reads like a generic LogRows→blockResult converter, but it actually contains transform-specific logic (vl_account_id / vl_project_id, and it's only called from the transformer). This might cause a little confusion or misuse later. Same for initFromBlockResult. Feel free to resolve, just flagging it.

[cuongleqq]

treats them as ordinary fields

The docs might be wrong here. From the code in initFromBlockResult, these fields always stripped from the result.

[vadimalekseev]

The documentation is correct, as mustInitFromLogRows overrides user-defined vl_account_id and vl_project_id fields:

1. We call addResultColumn here:

   VictoriaLogs/lib/logstorage/block_result.go

   Line 377 in d940917

   br.addResultColumn(resultColumn{name: "vl_project_id", values: projectIDs})

2. addResultColumn replaces existing column:

   VictoriaLogs/lib/logstorage/block_result.go

   Line 2279 in d940917

   br.csAddOrReplace(&csBuf[len(csBuf)-1])

This behavior is intentional: if a user enabled log transformations and passed vl_account_id in the request, but later disabled transformations, the logs would be written to the default tenant (because we won't call the initFromBlockResult function). The current approach is more explicit: if you want to set the tenant dynamically, you must provide it and set up an explicit transformation.