flux short circuit boolean evaluation not short circuiting!

[markddavidoff]

Steps to reproduce:
List the minimal actions needed to reproduce the behavior.

1. filter with a compound filter errors out with cannot compile @ 21:19-22:54: unsupported binary expression string >= int:

|> filter(fn: (r) =>
		(r["_field"] == "offline_for" and r["_value"] >= 0))

Expected behavior:
all offline_for fields are floats, so i'd expect the left hand filter to only return float rows and the exit before passing non-float rows to the right hand side.

Actual behavior:
errors out with cannot compile @ 21:19-22:54: unsupported binary expression string >= int

Workaround

|> filter(fn: (r) => (r["_field"] == "offline_for"))
|> filter( fn: (r) => r["_value"] >= 0)

splitting up the filter into 2 rows works as expected

Environment info:
influx db cloud 2.0

Other Context
oddly, this seems to only manifest if I add another query before this compound query, trying it by itself works fine. you can shoot me an email or have support contact me if you need the full query

[nathanielc]

@markddavidoff Thanks again. I think I might know what is going on here but it would help to see the full query.

In short something is making Flux think that the type of r["_value"] is a string and that is the core of the issue. Flux determines the type of values based on how they are used, perhaps part of the query uses the r["_value"] as a string?

Short circuit should work once we get past the type error. Can you share the full query here? If not I can get it out of band like you mentioned.

[markddavidoff]

errors or doesn't work as expected:

import "influxdata/influxdb/monitor"
option task = {name: "OfflineTimeMonitor", every: 5m}

bucket = "<>"
data = from(bucketID: bucket)
|> range(start: -1h, stop: now())
|> filter(fn: (r) =>
(r["_measurement"] == "pae"))
|> map(fn: (r) =>
({r with keep_row: if exists r.incident_ignore and r.incident_ignore == true then false else true}))
|> filter(fn: (r) =>
(r.keep_row))
|> filter(fn: (r) =>(r["_field"] == "offline_for")) and r["_value"] >= 0))
|> group(columns: ["_time", "acid"], mode: "by")
|> sum()
|> fill(column: "_value", value: 0.0)
|> map(fn: (r) =>
({
_field: "offlineTime",
_value: r._value,
acid: r.acid,
_time: r._time,
_measurement: "offlineTime-Test",
}))

but this works fine:

import "influxdata/influxdb/monitor"

option task = {name: "OfflineTimeMonitor", every: 5m}

bucket = "<>"
data = from(bucketID: bucket)
|> range(start: -1h, stop: now())
|> filter(fn: (r) =>
(r["_measurement"] == "pae"))
|> map(fn: (r) =>
({r with keep_row: if exists r.incident_ignore and r.incident_ignore == true then false else true}))
|> filter(fn: (r) =>
(r.keep_row))
|> filter(fn: (r) =>
(r["_field"] == "offline_for"))
|> filter(fn: (r) =>
(r["_value"] >= 0))
|> group(columns: ["_time", "acid"], mode: "by")
|> sum()
|> fill(column: "_value", value: 0.0)
|> map(fn: (r) =>
({
_field: "offlineTime",
_value: r._value,
acid: r.acid,
_time: r._time,
_measurement: "offlineTime-Test",
}))

[markddavidoff]

r["_value"] can be a string or a float depending on the _field but for the field offline_for it is always a float/double

[jpacik]

@markddavidoff Flux transformations like map and filter will check the type of each incoming table against what they expect or what they have inferred the types should be. In the case of

|> filter(fn: (r) =>(r["_field"] == "offline_for")) and r["_value"] >= 0))

each incoming table will be checked (before the anonymous function is evaluated) to make sure that _value is an integer column. When the filter function receives tables where _field != "offline_for" and _value is a string column, it doesn't know those tables will actually be filtered out without executing the function. All it sees is that it expected an integer column but got a string column instead. So from the perspective of the filter function it's not safe to evaluate fn over that table.

When you filter the on _field first in a separate filter call, the following filter

|> filter(fn: (r) => r._value >= 0)

receives only tables that have an integer _value column and so filter knows it's safe to evaluate fn over each one of those tables.

I hope I've helped to explain some of the underlying reasoning behind that error.

[markddavidoff]

The odd thing is that in most cases the combined filter works fine, and I only see the type error if i add another preceeding filter.

Regardless, this behavior is inconsistent with my expectations of how short circuiting should work, so it seems from my side as something which should be remedied via a fix or additional documentation caveats