YQ's data model?

[philomory]

I think there's something about the "data model" of yq that I'm not understanding, and maybe someone can shed some light on it?

I'm trying to understand why these two pipelines give different results:

#example.yaml
- app_version: 0.1.0 
  chart: foo-1.2.3
  name: foo
- app_version: 0.2.0 
  chart: bar-4.5.6
  name: bar
- app_version: 0.3.0 
  chart: baz-7.8.9
  name: baz

$ yq '.[] | select(.name=="foo").app_version // 0' example.yaml # attempt 1
0.1.0
0
0
$ yq '.[] | select(.name=="foo") | .app_version // 0' example.yaml # attempt 2
0.1.0

I expected the same results in both cases, but it seems like select doesn't actually remove the things it's filtering away until it gets to the pipe operator? Like select makes them temporarily null, and then the | actually removes them from the array entirely? Except that's clearly not right, because an array of literal nulls survives the pipe operator without issue.

Can someone explain why these work differently?

(Also .[] | select(.name=="foo").app_version | . // 0, which gives the same results as attempt 2, begs an explanation)

[mikefarah]

Basically it's operator precedence rules, same as you have when evaluating a maths expression like 3 + 4 x 2 = 3 + ( 4 x 2). You multiply before adding.

The first expression is evaluated like this:
'.[] | ( select(.name=="foo").app_version // 0 )'

• First it splats all the results .[],
• then for each one, it's going to try to find the one with name "foo" and get the app_version. If it can't, it returns 0 for that result.

The second expression is evaluated like this:

• First it splats the results, selects the one with name "foo", gets the "app_version". This will be a single value.
• then, if that all returns something falsy, it will return 0 instead

Hope that clears things up - TLDR - use brackets to make sure you get want you want in complex expressions. I have tried to make the precedence rules as intuitive as possible, and mimic what jq does. The numbers aren't in the user docs yet, but if you are interested you can see them here: https://github.com/mikefarah/yq/blob/master/pkg/yqlib/lib.go

[philomory]

Hi, Mike, thanks for answering!

So, I think my confusion only partially comes from operator precedence, but more fundamentally comes from my brain refusing to accept that select doesn't operate on collections. Even when typing .[] | select(...) my brain writes off .[] | as a weird syntactical wart in front of select, a function which (in my brain) takes an collection as input and returns a smaller collection as output. But actually, it takes an arbitrary value, applies a test to it, and then either returns it or doesn't. And .[] essentially takes an array as input, and returns an iterator over that array as output; any function applied to the iterator is applied to each element in the array, an the results of that are wrapped back into a new iterator so that subsequent function calls operate, again, on the individual elements rather than the iterator itself. And then if you actually want an array again you have to wrap everything in an outer [...], that starts right before the .[] and ends at the point you want to have an array again? Is that about right?

The other thing that really confused me was the distinction between null and "undefined", and the fact that // works on both of them, but most other parts of the language treat them differently. The key point here being that when the select condition is false, it returns "undefined", which is seen by // but, which is thrown away by a lot of other operators. For example, this surprised me:

$ yq -n '{"a": 1} | .a, .b'
1
null
$ yq -n '{"a": 1} | [.a, .b]'
- 1
$ yq -n '{"a": null} | .a, .b'
null
null
$ yq -n '{"a": null} | [.a, .b]'
- null

That's not to say that none of the operator precedence rules surprised me, though:

$ yq -n '{"a": null} | .a, .b | . // 0'
null
0
$ yq -n '({"a": null}) | (.a, .b) | (. // 0)
0
0
$ yq -n '{} | .a, .b | . // 0'
null
0
$ yq -n '{} | (.a, .b) | . // 0'
0
0

The fact that the , operator (or any operators other than paired delimiters like [...] or (...)) binds less tightly than | is really unexpected to me. jq doesn't seem to work that way, either (apologies if this is a bug that has been fixed, I've been using yq 4.25.1).

I guess it might help to include some documentation that actually points out that you can have a "context" which consists of multiple "results", but which contains neither an array/list, nor a set of multiple YAML documents (and in fact doesn't correspond to anything that exists in YAML whatsoever). Looking at even a simple pipeline like [1, 2] | .[], it's wasn't really clear (as someone who doesn't actually use jq) what "data type" I was supposed to interpret the result as being, since 1\n2 isn't actually a valid YAML document, or document stream.

[philomory]

Actually, reading back over what I wrote, I'm still not convinced I actually understand why the original example works the way it does; do the empty "results" from select not get culled until the | operator, for some reason? It works this way in jq too, even though the jq docs say that .a.b.c is equivalent to .a | .b | .c; so I don't understand why select(...).b // c isn't equivalent to select(...) | .b | . // c, unless that documentation was just stating a specific fact about the . operator rather than a general fact about pipelines (but, it's documentation for the pipeline operator).

[mikefarah]

I've got a little bit of documentation on how it works under the hood here: https://mikefarah.gitbook.io/yq/how-it-works - though the documentation isn't as good as I want it to be yet, need time to add some diagrams and whatnot.

.a.b.c is not quite equivalent to .a | .b | .c because the pipe operator | has a low order of precedence. It would be more correct to say it's equivalent to (.a | .b | .c)

The 'select(bool_exp)' operator runs against the context of nodes and evaluates bool_exp for each one. If bool_exp is true, the node is kept, if it's false, it's not included. You need to 'splat' an array into its components before piping to select, otherwise the select will run against the array itself (which can be useful, for instance checking array length).

[philomory]

The 'select(bool_exp)' operator runs against the context of nodes and evaluates bool_exp for each one. If bool_exp is true, the node is kept, if it's false, it's not included. You need to 'splat' an array into its components before piping to select, otherwise the select will run against the array itself (which can be useful, for instance checking array length).

I'd take issue with the assertion that select(bool_exp) doesn't include in its output any nodes for which bool_exp is false (it seems instead to include them, but with a flag that they should be removed at the next | operator; otherwise, the // wouldn't really work). But Overall I more-or-less get the idea, I think, at this point.

What about this, though? This seems to be a genuine bug, as far as I can tell?

$ yq -n '{} | .a, .b | . // 0'
null
0
$ jq -n '{} | .a, .b | . // 0'
0
0
$ yq -n '{} | (.a, .b) | . // 0'
0
0
$ yq -n '{} | .a, (.b | . // 0)'
null
0

The expression {} | .a, .b | . // 0 seems to be being parsed as {} | .a, (.b | . // 0); it seems like it should be parsed as {} | (.a, .b) | . // 0?