break syntax

[jlambatl]

Hello,

I notice that the VRL documentation states that the break command/syntax is reserved for future use. Is there anything blocking that from being implemented?

There's context; we have several large payloads that we are processing with VRL in vector. The current implementation requires us to go through the entire array or object, even when there are no further matches. We have tried various approaches with filter and map_values with little difference in overall throughput. Maybe there's another way that we are missing; otherwise, we would be happy to submit a PR to introduce break.

Happy to discuss this further if there are other options beyond break, or potentially other avenues we should explore.

[pront]

Hi @jlambatl, I cannot think anything apart from development effort that is blocking break support. I would go even further and implement for (since we already support for_each and even fully recursive functions).

[jlambatl]

@pront I've now taken the time to look more closely at what's involved in adding the required features. I wanted to loop back around now that I've a better understanding of how VRL's internals work.

It appears that VRL currently lacks ControlFlow support. To add break, continue, and other more advanced features, ControlFlow support needs to be added. I've found other approaches in other projects that use LALRPOP and their own parsers; however, this seems to be the most idiomatic way in Rust. Assuming that this has been discussed internally, is this aligned?

On the surface, this seems like it could be made a non-breaking change because users of functions like for_each would remain stable while new lower-level primitives, like for, would be added. It seems that an additional try_for_each would be added because the default iter() doesn't support break, although it does support return, afaik.

It seems that if this approach is followed, it could be non-breaking; however, this is the first time I've delved into compilers and parsers of this type in some time. I'm keen to understand how your team sees the problem and solution so that I can make sure the approach is agreeable before I start working on it.

[pront]

Hi @jlambatl, thank you for this write up!

I researched this a bit a more. Here are a couple of links on this topic:

• regarding for loops: https://github.com/vectordotdev/vector/blob/master/rfcs/2021-08-29-8381-vrl-iteration-support.md?utm_source=chatgpt.com#for-loop
  • I am not really against introducing for loops, but there are some valid concerns in historical RFCs that we would need to address. An RFC would be best before any implementation.
• return is supported and it can terminate loops early: https://github.com/vectordotdev/vrl/pull/712/files

It seems that if this approach is followed, it could be non-breaking; however, this is the first time I've delved into compilers and parsers of this type in some time. I'm keen to understand how your team sees the problem and solution so that I can make sure the approach is agreeable before I start working on it.

Please take some time going through the links above since introducing for loops would be a major change but I agree, non-breaking for users since it can co-exist with for_each.

I've found other approaches in other projects that use LALRPOP and their own parsers

Tangential, but I always wanted to experiment with https://docs.rs/nom/latest/nom/ as an alternative to what we have now. Obviously, this is also not a trivial project. Just sharing it here with the community in case anyone is interested.