for a jq code beautifier

[tst2005]

Hello wader,

I hope your are fine.
I discovered your jqjq project late.
I also made my own POSIX Shell wrapper with jq for jq but for another goal with another design (mainly 1 jq to process the code 1 jq to evaluate it).
I've been dreaming of a jq lexer/parser for a while, you did it!

Now I think about a side : a jq code beautifier !

With your lex and parse functions, I get the AST.
I need a function to convert back the AST to string.

• This "AST to string" does not exist in jqjq, isn't it?

I plan to write this "AST_to_string" function, but I'd rather ask you if you haven't already done it?

Regards,

[wader]

Hey! all good here, thanks for asking

Yes that would be very nice to have, and i've had thoughts about doing a jqlint/jqfmt tool but not much progress yet. I've asked itchny the gojq author about it some time ago itchyny/gojq#123 itchyny/gojq#62, he had some ideas that might be useful.

Off the top of my head there are some tricky things:

• An AST that can preserve comments and whitespace (new lines etc). Ideally i would like it to support to automatically know if a bunch of sequential pipe/commas should be all on one line or automatically have them all on separate lines. Thinks like that and i hope there is some heuristics based on line break you can use? I guess you have seen the jq code style i use i jqjq, i think that one works very well for me in big jq code bases.
• String interpolation expressions. Just reformat without line breaks?
• The jqjq lexer/parser is not very good at handling errors, that might be something such a tool need to do? maybe some other parsing technique would be more fitting. Thinking something like https://github.com/tree-sitter/tree-sitter
• Probably a lot more...

BTW there is an old WIP jqjq branch to support serializing back to compact string https://github.com/wader/jqjq/tree/tostring, also a WIP branch to produce graphviz! :) https://github.com/wader/jqjq/tree/dot
BTW2 looked at https://github.com/wader/jq-lsp? it can do some linting stuff

[tst2005]

Thanks @wader for this reply.

This weekend I worked on your jqjq's tostring branch : It's much more fun than doing everything from scratch!
I found some bug and started to implement some missing part (but not all !). I will publish my contrib soon.

For now I'm not looking to make a perfect beautifier, just a simple way to parse and get the code back.
It would be nice to have a powerful and highly configurable jq code beautifier but that's not my goal at the moment.
My real goal is to be able to remove unused function definitions: parse, detect calls, remove unused function definitions and recover the jq code.

Note: I know that the missing support for "String interpolation expressions" in your jq parser will probably be the first major problem in detecting function calls but never mind!

Regards,

[wader]

Thanks @wader for this reply.

This weekend I worked on your jqjq's tostring branch : It's much more fun than doing everything from scratch! I found some bug and started to implement some missing part (but not all !). I will publish my contrib soon.

Nice, hope it was in ok shape?

For now I'm not looking to make a perfect beautifier, just a simple way to parse and get the code back. It would be nice to have a powerful and highly configurable jq code beautifier but that's not my goal at the moment. My real goal is to be able to remove unused function definitions: parse, detect calls, remove unused function definitions and recover the jq code.

Good strategy i think, its usually what i do also, work on something that feel motivating and fun and while you do that you learn more and get ideas how to do the rest :)

Note: I know that the missing support for "String interpolation expressions" in your jq parser will probably be the first major problem in detecting function calls but never mind!

Yeap ignore parsing of string interpolation for now, it will probably involve modifying the lexer a bit... a bit tricky. If you want to write "tostring" code for string interpolation anyway you can always produce an AST manually, or use:

$ fq -n '`"\(1+2)"` | _query_fromstring'
{
  "term": {
    "str": {
      "queries": [
        {
          "term": {
            "query": {
              "left": {
                "term": {
                  "number": "1",
                  "type": "TermTypeNumber"
                }
              },
              "op": "+",
              "right": {
                "term": {
                  "number": "2",
                  "type": "TermTypeNumber"
                }
              }
            },
            "type": "TermTypeQuery"
          }
        }
      ]
    },
    "type": "TermTypeString"
  }
}

to get gojq's AST as JSON. I've tried to reuse that gojq's structure as much as possible in jqjq. Note also that fq has support for "raw" string literals using `...` which is quite handy in these cases.

[tst2005]

TL;DR

• if you want to take a look on my modified ast_tostring : see here
• if you want to see which cases I support for now : see here

---

My way to run is ...

• I made my jq functions collection : jq-mods
• I use my own wrapper (shell+jq) to
  1. run my jq stuff (with dynamic loading of the needed functions). Usefull for me to experiment thing, for dev.
     but also
  2. run my jq stuff over a generated static script (a jq call with a big code that include all functions defs needed). Usefull for me to produce a more "production ready" script (no dependency except jq it-self).

You may see the both way (bin-dev VS bin) in jsondiff.

About the "jq beautifier" idea (and more)...

I took your lex, parse and ast_tostring functions.
Note: I renamed ast_tostring to ast_to_string (mainly because I also have an data convertion project where everything is named format1_to_format2).

I put them in lex.jq, parse.jq, ast_to_string.jq to be able to easily use them onver my wrapper (jq_stack4).

My tests was written in a separated repository "jq-mods-demo" (a sort of poor test suite :-$ ).
I suppose my stuff is not so easy to understand/use (except for me fo course !).

For now it is easier for me to show you my working space directly, but I can put my ast_to[_]string in a jqjq fork (change indentation from tab to 2space) and make you a Pull Request.

Regards,

[wader]

TL;DR

• if you want to take a look on my modified ast_tostring : see here
• if you want to see which cases I support for now : see here

My way to run is ...

• I made my jq functions collection : jq-mods

• I use my own wrapper (shell+jq) to

  1. run my jq stuff (with dynamic loading of the needed functions). Usefull for me to experiment thing, for dev.
     but also
  2. run my jq stuff over a generated static script (a jq call with a big code that include all functions defs needed). Usefull for me to produce a more "production ready" script (no dependency except jq it-self).

You may see the both way (bin-dev VS bin) in jsondiff.

👍 I haven't looked closer how the jq shell/mods stuff work yet. I usually run a fq -i when i want a "jq shell", something similar?

About the "jq beautifier" idea (and more)...

I took your lex, parse and ast_tostring functions. Note: I renamed ast_tostring to ast_to_string (mainly because I also have an data convertion project where everything is named format1_to_format2).

It's a bit annoying that jq's standard library is inconsistent with this, i can never make up my mind which one i prefer. I went with without underscore of most of fq but it do feel messy at times so maybe should change it.

I put them in lex.jq, parse.jq, ast_to_string.jq to be able to easily use them onver my wrapper (jq_stack4).

My tests was written in a separated repository "jq-mods-demo" (a sort of poor test suite :-$ ). I suppose my stuff is not so easy to understand/use (except for me fo course !).

Looks good i think! would be nice if there was a more consistent way ppl write jq, so many different styles it seems :) after lots of written jq i've ended up with 2 space indent which aligns nice with pertnesses with multiple lines

What happens here https://github.com/tst2005sh/jq-mods/blob/master/lib/jq/ast_to_string.jq#L43 splitting on " "?

Have you looked at how jq implements tests? quiet neat. There is implementation of it in jq for jqjq, also one in fq that might be interesting https://github.com/wader/fq/blob/master/pkg/interp/jqtest.jq, but it require some kind of eval.

For now it is easier for me to show you my working space directly, but I can put my ast_to[_]string in a jqjq fork (change indentation from tab to 2space) and make you a Pull Request.

Great! i'm happy as long as it's consistent inside one project, so just make it look similar to the rest for now, maybe i will change to more underscores and whatnot later :)

One thing, i try to keep jqjq compatible with jq 1.6 (the one used on jqplay.org) which most of times usually just means avoid using optional else for if-statements.

I wonder if we can figure out some way nice way to have tests for various ast things. Currently jqjq.test won't work as is i think unless we add plumbing so they are available in the jqjq environment, maybe as _jqjq_<func> somehow etc?