Practical happy CLI arg parsing with Scryer

[jjtolton]

Got very annoyed writing bash scripts like this because I can never remember the syntax. I stared writing some code like below only to realize getopts only handles short args like -u and -p and not long params like --host... then I'd have to write some stupid shift 2 parser and look up even more syntax. Which is fine I guess BUT thank God I remembered Prolog exists!

#Note: this code does not fully work because getopts does not handle long params
function parse-check-user-exists() {
  local username=""
  local password=""
  local host="localhost"
  local port=5309
  local dbusername="admin"
  local dbpassword=""

  while getopts "u:p:h" opt; do
    case "$opt" in
      u)
        username="$OPTARG"
        ;;
      host)
        host="$OPTARG"
        ;; 
      port)
        port="$OPTARG"
        ;;
      dbusername)
        dbusername="$OPTARG"
        ;;
      dbpassword)
        dbpassword="$OPTARG"
        ;;
      help)
        parse-check-user-exists-usage 
        exit 0  # Important: Exit after displaying usage
        ;;
      \?)
        echo "Invalid option: -$OPTARG" >&2
        parse-check-user-exists-usage
        exit 1  # Exit after displaying usage
        ;;
      :)
        echo "Option -$OPTARG requires an argument." >&2
        parse-check-user-exists-usage
        exit 1  # Exit after displaying usage
        ;;
    esac
  done

  
  # Check if username and password are provided
  if [ -z "$username" ]; then
    echo "Please supply username" >&2
    parse-check-user-exists-usage
    exit 1
  fi

  shift $((OPTIND - 1)) 
  echo username=$username
  echo host=$host
  echo port=$port
  echo dbusername=$dbusername
  echo dbpassword=$dbpassword
  return 0
}

Then I remembered -- Scryer Prolog!!

:- use_module(library(dcgs)).
:- initialization(main).

/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   Intended to be used with the following style of bash script:

   local parsed=$(scryer-prolog -f ./path/to/this.pl -- $@)
   declare -A params
   while IFS='=' read -r key value; do
      params[$key]=$value;
   done <<< $parsed
   
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

args(As) :-
        '$toplevel':argv(As).


param(username(Username)) -->
        ( ["--username"] | ["-u"] ),
        [Username].

param(port(Port)) -->
        (   ["--port"] | ["-p"] ),
        [Port].

param(host(Host)) -->
        (   ["--host"] | ["-h"] ),
        [Host].

inputs_params([]) --> [].
inputs_params([Param|Params]) -->
        param(Param),
        inputs_params(Params).

inputs_params(Inputs, Params) :-
        phrase(inputs_params(Params), Inputs).

functors_pairs([], []).
functors_pairs([Functor|Functors], [Pair|Pairs]) :-
        Functor =.. [Head,Arg],
        atom_chars(Head, Hs),
        Pair=Hs-Arg,
        functors_pairs(Functors,Pairs).

printkvs([]).
printkvs([K-V|Pairs]) :-
        format("~s=~s~n", [K,V]),
        printkvs(Pairs).
        
main :-
        args(As),
        inputs_params(As, Params),
        functors_pairs(Params, Pairs),
        printkvs(Pairs),
        halt(0).

Voila!

$ RESULTS=$(scryer-prolog -f ./scripts/admin/parse-check-user-params.pl -- --username bob -p 1234)
$ echo $RESULTS
username=bob port=1234
$ while IFS='=' read -r key value; do echo "$key: $value"; done <<< $RESULTS
username: bob
port: 1234

Earth shattering? No. But great example of a place where it's legitimately faster and more maintainable to write Scryer code than some other production code for a practical industry task.

(comments improving the script welcome!)

[triska]

Thank you a lot for sharing this! I find such examples extremely helpful for learning Prolog.

The code looks very elegant. Personally, I would do 2 things a bit differently:

Instead of using initialization(main), I would invoke Scryer with -g main in the script. In this way, we can consult the file and interactively test all predicates in isolation without invoking main/0. Also, argv/1 is part of the interface of library(os).

Second, I would most likely write functors_pairs/2 slightly differently, by pulling the unification into the clause head. When I'm writing:

functors_pairs([Functor|Functors], [Pair

then I already know that Pair is a pair, and if it's a pair I know that will be inspected or constructed (as oppossed to being only passed through by the predicate), then I would tend to write the key and value already at that place, i.e.:

functors_pairs([Functor|Functors], [Hs-Arg

And then I would try to use an analogous name for the tail of the list, a name that is somehow suggestive of and connected with Hs-Arg. In this case, I would use for example HAs for the tail:

functors_pairs([Functor|Functors], [Hs-Arg|HAs]) :-

And now could be a good time to say: One moment, what about first describing this for a single functor and pair?

functor_pair(Functor, Hs-Arg) :-
        Functor =.. [Head,Arg],
        atom_chars(Head, Hs).

In an ideal implementation, atom_chars/2 is an O(1) operation. Even so, maybe this conversion is not even necessary.

In fact, the first argument is not even a functor? So, a better name would also be appropriate.

And then we can write functors_pairs/2 as:

functors_pairs(Functors, Pairs) :-
        maplist(functor_pair, Functors, Pairs).

or alternatively use maplist/3 in the only place where that predicate is currently used. Analogous for printkvs/1, which can be expressed with maplist/2 and a lambda expression, or alternatively with a DCG together with phrase_to_stream/2.

[triska]

Personally, I have come to like ~n in format strings more than \n in DCGs, so the only small change I would make here is to use format_//2:

list_newlinephrase([X|Xs]) -->
        format_("~s~n", [X]),
        list_newlinephrase(Xs).

[jjtolton]

Personally, I have come to like ~n in format strings more than \n in DCGs, so the only small change I would make here is to use format_//2:

list_newlinephrase([X|Xs]) -->
format_("sn", [X]),
list_newlinephrase(Xs).

This is exquisite. Really elegant use of DCGs. Thanks for sharing this!

[bakaq]

Is there any reason for using ~n instead of \n apart from aesthetics?

[triska]

Personally, I find the intent of the "line" a bit easier to read with a format string such as "~s~n", instead of, say seq(Cs), "\n". Semantically it's the same.

There is a reason to prefer seq(Cs) or format_("~s", [Cs]) over just Cs, because the former enforces the (string) type check on Cs, whereas just Cs would for example lead to execution of a goal if invoked with Cs = { Goal }.

[jjtolton]

Messing around, in this short example... probably not. But in other examples, ~n can be factored into the layout calculations while \n does not (seem to get factored into the layout calculations).

?- phrase(format_("~tx~n~t~s~tx~n~t~12|~t~s~t\n~t.~10|", ["a", "b"]), P), format("~s", [P]).
%@ x
%@ ax
%@             b
%@ .   P = "x\nax\n            b\n ...".


?- phrase(format_("~tx~n~t~s~tx~n~t~12|~t~s~t~n~t.~10|", ["a", "b"]), P), format("~s", [P]).
%@ x
%@ ax
%@             b
%@          .   P = "x\nax\n            b\n ...".

There is a very subtle difference in the placement of the periods in the first and second example. I don't have a non-contrived example of when this may be important.

[triska]

Regarding param//1: We can use term_expansion/2 to generate the code from simpler descriptions. For instance, with:

term_expansion(option(Name,Long,Short),
               ( param(Option) -->
                         ( [[-,-|Long]] | [[-|Short]] ),
                         [Value] )) :-
        Option =.. [Name,Value].


option(username, "username", "u").
option(port, "port", "p").
option(host, "host", "h").

We get:

?- phrase(param(P), ["--username","test"]).
   P = username("test")
;  false.

[jjtolton]

This is great -- I was wondering about something like this. It seemed like there was some redundancy in the param descriptions.

Regarding param//1: We can use term_expansion/2 to generate the code from simpler descriptions. For instance, with:

term_expansion(option(Name,Long,Short),
( param(Option) -->
( [[-,-|Long]] | [[-|Short]] ),
[Value] )) :-
Option =.. [Name,Value].

option(username, "username", "u").
option(port, "port", "p").
option(host, "host", "h").
We get:

?- phrase(param(P), ["--username","test"]).
P = username("test")
; false.

[jjtolton]

Version 2 of the script:

:- use_module(library(dcgs)).
:- use_module(library(pio)).
:- use_module(library(lists)).
:- use_module(library(os)).

/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   Intended to be used with the following style of bash script:

   local parsed=$(scryer-prolog -f ./path/to/this.pl -g main -- $@)
   declare -A params
   while IFS='=' read -r key value; do
      params[$key]=$value;
   done <<< $parsed
   
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

%% v2: per https://github.com/mthom/scryer-prolog/discussions/2862#discussioncomment-12663273
term_expansion(option(Name,Long,Short),
               (   param(Option) -->
                   (   [[-,-|Long]] | [[-|Short]] ),
                   [Value] )) :-
        Option =.. [Name,Value].

option(username, "username", "u").
option(port, "port", "p").
option(host, "host", "h").
option(dbusername, "dbusername", "du").
option(dbpassword, "dbpassword", "dp").

%% v2: add param "spellcheck"
param(_) -->
        [Param],
        [Value],
        !,
        { error(unknown_values, [param(Param), value(Value)])}.

inputs_params([]) --> [].
inputs_params([Param|Params]) -->
        param(Param),
        inputs_params(Params).

inputs_params(Inputs, Params) :-
        phrase(inputs_params(Params), Inputs).

% v2: per https://github.com/mthom/scryer-prolog/discussions/2862#discussioncomment-12660682
functor_pair(Functor, Hs-Arg) :-
        Functor =.. [Head,Arg],
        atom_chars(Head, Hs).

functors_pairs(Functors, Pairs) :-
        maplist(functor_pair, Functors, Pairs).

% v2: per https://github.com/mthom/scryer-prolog/discussions/2862#discussioncomment-12668080
list_newlinephrase([]) --> [].
list_newlinephrase([X|Xs]) -->
        X,
        "\n",
        list_newlinephrase(Xs).

printkvs(KVs) :-
        phrase_to_stream(list_newlinephrase(KVs), user_output).

main :-
        argv(As),
        inputs_params(As, Params),
        functors_pairs(Params, Pairs),
        printkvs(Pairs),
        halt(0).

[jjtolton]

Version 3 of the script:

% CLI parsing example, version 3

:- use_module(library(dcgs)).
:- use_module(library(debug)).
:- use_module(library(pio)).
:- use_module(library(lists)).
:- use_module(library(os)).

/* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   Intended to be used with the following style of bash script:

   local parsed=$(scryer-prolog -f ./path/to/this.pl -g main -- $@)
   declare -A params
   while IFS='=' read -r key value; do
      params[$key]=$value;
   done <<< $parsed
   
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

%% v2: per https://github.com/mthom/scryer-prolog/discussions/2862#discussioncomment-12663273
term_expansion(option(Name,Long,Short),
               (   param(Option) -->
                   (   [[-,-|Long]] | [[-|Short]] ),
                   [Value] )) :-
        Option =.. [Name,Value].

option(username, "username", "u").
option(port, "port", "p").
option(host, "host", "h").
option(dbusername, "dbusername", "du").
option(dbpassword, "dbpassword", "dp").

% v2: add param "spellcheck"
param(_) -->
        [Param],
        [Value],
        !,
        { error(unknown_values, [param(Param), value(Value)])}.

inputs_params([]) --> [].
inputs_params([Param|Params]) -->
        param(Param),
        inputs_params(Params).

inputs_params(Inputs, Params) :-
        phrase(inputs_params(Params), Inputs).

% v2: per https://github.com/mthom/scryer-prolog/discussions/2862#discussioncomment-12660682
functor_pair(Functor, Hs-Arg) :-
        Functor =.. [Head,Arg],
        atom_chars(Head, Hs).

functors_pairs(Functors, Pairs) :-
        maplist(functor_pair, Functors, Pairs).

% v2: per https://github.com/mthom/scryer-prolog/discussions/2862#discussioncomment-12668080
% v3: per https://github.com/mthom/scryer-prolog/discussions/2862#discussioncomment-12669324
list_newlinephrase([]) --> [].
list_newlinephrase([K-V|Xs]) -->
        format_("~s=~s~n", [K,V]),
        list_newlinephrase(Xs).

printkvs(KVs) :-
        phrase_to_stream(list_newlinephrase(KVs), user_output).

main :-
        argv(As),
        inputs_params(As, Params),
        functors_pairs(Params, Pairs),
        printkvs(Pairs),
        halt(0).

[jjtolton]

So I realized that I'll need to reuse the script multiple times with different options. Rather than copy/paste the script, I thought I would try to dynamically load the options. Honestly, I'm not crazy about this approach, but it works. If anyone has any more elegant thoughts, please let me know:

main :-
        getenv("PARSE_OPTS_FILE", ParseOptsFile),
        open(ParseOptsFile, read, Stream),
        load(Stream),
        argv(As),
        inputs_params(As, Params),
        functors_pairs(Params, Pairs),
        printkvs(Pairs),
        halt(0).

Idea being to invoke with something like:

$ PARSE_OPTS_FILE=./scripts/admin/create_user_options.pl scryer-prolog -f ./scripts/admin/create_user.pl -g main -- --username username -du admin -dp 4567 -h localhost

where create_user_options.pl is something like:

option(username, "username", "u").
option(port, "port", "p").
option(host, "host", "h").
option(dbusername, "dbusername", "du").
option(dbpassword, "dbpassword", "dp").

I would definetly prefer a metainterpreter based approach, but my creativity is failing me here.

[triska]

I think it would suffice to specify both files on the command line?

scryer-prolog -f -g main ./scripts/admin/create_user.pl $PARSE_OPTS_FILE -- --username username -du admin -dp 4567 -h localhost

Let Scryer Prolog load the file for you!

Note that Scryer-specific options should be placed before the files. After the files come application-specific options.

[jjtolton]

scryer-prolog -f -g main ./scripts/admin/create_user.pl $PARSE_OPTS_FILE -- --username username -du admin -dp 4567 -h localhost
Let Scryer Prolog load the file for you!

Brilliant!

[hurufu]

Imho interesting idea would be to parse CLI arguments (or any other sort of configuration), populate environment variables from those values and then exec() into another program, possibly written in another language.