[FR] - CLI improvements for formatting and consistency

[disassembler]

Internal/External
Internal if an IOHK staff member.

Area

Other Any other topic (Delegation, Ranking, ...).

Describe the feature you'd like
Currently, we abuse a lot of the UI output of CLI in internal and community tools. We also trick the CLI into giving us json for some commands but not others by setting an --out-file /dev/stdout. Ideally, we'd have a unified interface that could be used for scripts for all CLI commands that provide output to the terminal. My thoughts are add a --format flag that is required for any command in CLI that gives output and initially only support plain and json gives a machine readable format that can be used for scripts.

Some examples:
for submit, something like:

{ "status": "submitted", "txid": TXID }

OR for failure

{ "status": "error", "txid": TXID, "message": "ERROR MESSAGE" }

For things like query utxo, --out-file would give the default format for the command and be overridable with the --format flag.

I think every query command absolutely needs a --format json flag added to it.

For commands that are already json, they should support a --format plain option (for example with tip.

transaction view should have the yaml format parsed as json with --format json flag.

Also, all query commands should support --out-file to put output in the specified format output to a file instead of stdout.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context / screenshots
Add any other context or screenshots about the feature request here.

[AndrewWestberg]

I can't upvote this issue enough.

[gitmachtl]

The query utxo normal output should stay as the normal output, adding a --json flag to get the output in json format makes sense. But having the normal output gives some advantages in post processing also with the number overflow limitations of some jq versions. So please keep the normal output too, or invent a --non-json flag

[robinboening]

I was thinking of creating such issue myself, but now I found this. This would be so helpful, thank you!

I am currently writing a cardano-cli wrapper in Ruby and its a pain not knowing what output format to expect unless you know exactly what command returns what.

One example: Splitting the query utxo output for a machine readable data is so laborious!

Than you for considering this!

[gitmachtl]

This post above is a bit old. When you set the output flag it will output json.

Processing the plain text output - which will stay and is important as alternative - is not difficult. I generate my own JSON version of an UTXO with stringified numbers like: https://github.com/gitmachtl/scripts/blob/7b59fa3a41aad421d840a89fe21f14e25c36b9c3/cardano/mainnet/00_common.sh#L416-L479

The nice think about the plaintext output is to manipulate it superfast with the normal tools, filter out utxos, etc. 😄

[robinboening]

This post above is a bit old. When you set the output flag it will output json.

Could you show an example for the utxo query? I don't see it documented, nor can I get it working with the latest cardano-cli version: ​cardano-cli 1.33.0 returns Invalid option --output or Invalid option --json

Processing the plain text output - which will stay and is important as alternative - is not difficult. I generate my own JSON version of an UTXO with stringified numbers like: https://github.com/gitmachtl/scripts/blob/7b59fa3a41aad421d840a89fe21f14e25c36b9c3/cardano/mainnet/00_common.sh#L416-L479

Parsing isn't that difficult, but there is a good reason there are standardised formats like JSON. I don't think you're trying to say everyone should write their own parser as we all do at the moment, do you? I've done that myself as well, and I hate it.

Only a tiny change in the current format and your hardly readable 60 lines of code will break. I mean, c'mon you've added comments everywhere to explain why you're doing what you're doing. This is completely superfluous when choosing a standard that can be parsed by existing parsers.

The nice think about the plaintext output is to manipulate it superfast with the normal tools, filter out utxos, etc. 😄

I don't agree. There is no need to manipulate the pretty plaintext output if there is a JSON version as you can easily build any output you want, superfast, with normal tools. It's still a CLI and not a GUI tool.

[robinboening]

your hardly readable 60 lines of code will break. I mean, c'mon you've added comments everywhere to explain why you're doing what you're doing.

After re-reading my comment I thought I should come back and say I didn't mean to offend you. My comment might read harsh, but what I really mean is that any parser for this output will not be very easy to read and it should simply not be needed at all.

[gitmachtl]

The CLI will output a JSON structure if there is an --out-file parameter. So a simple --out-file /dev/stdout will give you a JSON format output.

The reason why i am using the plain text output and converting it to JSON by myself to the exact same output format as the above method will give you is, that the jq tool has a problem with large numbers. The numbers of the assets are numbers and not strings, for very large ones - and we have those in cardano - the jq tool (which is the most common used on the cli) is failing. Thats the reason i redo it and i put the numbers into strings. Because the jq -r command is not having a problem with that. There is also a new one, that is much quicker than the old one. 😄
https://github.com/gitmachtl/scripts/blob/7b59fa3a41aad421d840a89fe21f14e25c36b9c3/cardano/testnet/00_common.sh#L481-L571

You can directly use the JSON format output of the query, but be careful with the calculations and do a double check on large numbers.

[robinboening]

The CLI will output a JSON structure if there is an --out-file parameter. So a simple --out-file /dev/stdout will give you a JSON format output.

Thank you!

the jq tool has a problem with large numbers

Right, that's actually a known issue for quite some years. The PR that supposedly fixes that issue got merged in 2019, but jq hasn't been released since 2018.

Maybe one day, when package managers are equipped with a jq version including that fix, we can get rid of the table representation :)

Anyways, makes sense now. Thanks for clarifying!

[cblp]

I'm working on this in #3548. Please comment here or there which commands need JSON output and which ones have wrong JSON behaviour and must be fixed.