Implement repl filter and split up main file

[01mf02]

This PR implements a REPL akin to fq's REPL, namely as a filter called repl.
REPL usage can be nested, as demonstrated below:

$ jaq -n repl
> .
null
> .+1
1
> .+1 | repl
  > .
1
  > ^D
> .
null
> ^D
$

Unlike fq, jaq does not require a --repl flag, and the repl filter can be used anywhere, not just "last in pipeline".

Due to depending on rustyline for implementing the REPL, this requires bumping MSRV to 1.66.

Limitations

• Nested REPLs cannot access variables from parent REPLs (like in fq).
• The indentation does not increase for nested REPLs. This has been implemented in 1e35912.
• Only terms can be input; that is, you cannot write def f: 1;, hit ENTER, then use f in another term. However, you can write def f: 1; f + f. (This could be overcome by a simple heuristic that checks whether the line ends with ';' and treats this as intermediate command.) fq also has this limitation.
• The REPL cannot be used with inputs read from stdin. This is due to an issue with the underlying rustyline library. This has been resolved in 9cb4384.

Ideas for how to overcome these limitations are welcome.

Acknowledgements

Thanks to @wader for having implemented repl in fq, which has served as primary inspiration for me.

[wader]

Nice to see work on this! have you felt the urge yet to implement most of it in jq? 😬 it was very satisfying to more or less be able to do repeat(read as $expr | eval($expr) | print) 😄

[01mf02]

Nice to see work on this! have you felt the urge yet to implement most of it in jq? 😬 it was very satisfying to more or less be able to do repeat(read as $expr | eval($expr) | print) 😄

It sounds tempting ... I already wondered whether there is an eval in fq, but thanks to you, now I know.

There is a pretty nasty restriction in jaq that makes implementing repl in jq tough: I do not know how to implement eval/1 lazily because of a lifetime issue. In particular, I currently make the assumption that the main filter lives as long as the Inputs object (that is used for the input / inputs filters). That means that I cannot create some kind of child filter as part of eval and use its outputs lazily for the main filter, because that child filter would need to live as long as the Inputs of the main filter, but given that the child filter is only temporary, this is not satisfied. I have already tried to go around this restriction by saving the filter data (the look-up table, i.e. Lut) without lifetimes (that is, inside an Rc instead of as '&a Lut), but I was not able to make this compile. I also did not push this very hard once I found out that fq's repl does never return anything, so I was able to weasel around this restriction.

TL;DR: I currently do not know how to implement a lazy eval in jaq. That means that I currently cannot implement repl by definition in jaq, unfortunately.

[01mf02]

P.S.: Even if I was able to store the Lut as Rc, it would imply some performance overhead for each context clone, which can happen quite frequently, even if eval is never used. That's also why I am hesitating that this is a good move.
But on the other hand, having a functional eval is still quite tempting, because it would allow some kind of metaprogramming. As you can see, I'm a bit split up about this. 🤔

[01mf02]

In any case, having eval work is most likely not possible before jaq 3.0, because that would imply an API break. I'll adopt my current solution for jaq 2.3 and see if I can make eval work for jaq 3.0.

[01mf02]

I have already tried to go around this restriction by saving the filter data (the look-up table, i.e. Lut) without lifetimes (that is, inside an Rc instead of as '&a Lut), but I was not able to make this compile.

Yesterday, I was able to create a little prototype after all that uses this approach, see #304.

[wader]

Nice work, i'm trying to follow along about the lifetime reasoning, think i see what the problem but my internal borrow checker is not that good :) would be great to later discuss how a more competent eval function could work, binding, isolation, provide input etc

PS there is an old jq PR jqlang/jq#1843 that adds eval as part of adding more IO and co-routines support. Might be an interesting read.

PS2 realised that a basicrepl can be even more satisfying repeat(eval(read)) 🧘‍♂️ ...but in practice if will probably get uglier once you want to handle errors, printing, completion and all kinds of bells and wissels

[01mf02]

Nice work, i'm trying to follow along about the lifetime reasoning, think i see what the problem but my internal borrow checker is not that good :) would be great to later discuss how a more competent eval function could work, binding, isolation, provide input etc

Yeah I know, this stuff is pretty hard to understand, and it took me several days/weeks to figure out why my pointer approach is not working. But, in a nutshell, in jaq, the output of a filter has the type BoxIter<'a, ...>, where 'a is the lifetime of the executed filter. That means that the filter must be kept alive as long as the filter may yield outputs. Sounds logical.
Now, if we have a filter f that may call eval(g), then the outputs of f must live as long as (the compiled version of) f. However, because f may return outputs from eval(g), that means that eval(g) must also live as long as (the compiled version of) f. And we can only achieve this if we a) leak memory (that is, make the compiled version of g live forever, which is bad for RAM ^^) or b) use some kind of garbage collection / reference counting, e.g. Rc. There, reference counting frees the memory for g once it is not referenced anymore; that is, it has finished execution. That approach is what I took in my PR, but as I mentioned there, it unfortunately comes with performance degradation, because we have to check at runtime a lot whether g is still referenced.

PS there is an old jq PR jqlang/jq#1843 that adds eval as part of adding more IO and co-routines support. Might be an interesting read.

This is quite impressive! Even if I'm somewhat skeptical about the I/O features, because I like to think of jq as a (mostly) pure functional programming language after all. ;)

PS2 realised that a basicrepl can be even more satisfying repeat(eval(read)) 🧘‍♂️ ...but in practice if will probably get uglier once you want to handle errors, printing, completion and all kinds of bells and wissels

🤯 (about the video)

I also thought about the repeat(read | eval(print)) approach, but this makes recording the REPL history much more cumbersome. I think I'll go with my current approach for now, given that it has taken me enough time already. Anyway, thanks a lot for your input, and have a nice day!