printf format string with mix'n'match % and %x$ format, should we fix or withdraw.

[phidebian]

Hi All,

Before fixing #324 I'd like to poll you opinion about it.

printf builtin allow sequential access to argv[] values with % format and allow indexed access to argv[x] with %x$ format, not only %2$s but as for width/precision as well.

Mix'n'match of this two % and %x$ is very prone to misinterpretation, and result in a lot of troubles.

C printf(3) don't allow mix'n'match, it want all be % or all be %x$, yet this is a warning, and ignoring it, it behave like if any non indexed acces got internally indexed by its occurence in the format string.

Something like "%s %s %5$s %4$s %s" is treated like "%1$s %2$s %5$s %4$s %5$s" with the fact that 2 occurences of a given index (here 5) assume the argv[4] is of an appropriate type for the corresponding format, i.e printf("%1$d %1$x\n",32);is ok and printf("%1$s %1$x\n",32); ` is not ok.

Should we insist of allowing indexed access knowing that current implementation is plain wrong, then probably nobody use it, or do we consider there are still some canonical case that some may use and do work as expected.

Bash refrain to do any x$ addressing
Zsh do it, still mostly wrong, yet it tend to simply provide wrong result, compared to ksh that provide, wrong result, goes in almost infinite loops, or even segfault.

I think I could make it work, but would wait the poll answer, if nobody need that, no need to work on it, just add some security check, and reject x$ with a message as bash do.

I provide some test case here that shows how bad it is on ksh.

TC$ f='%s %s\n'     ; printf "$f" 1 2 3 4
1 2 <-- OK canonical
3 4

TC$ f='%s %1$s\n'   ; printf "$f" 1 2 3 4
1 2 <-- WRONG zsh give 1 1 as well as C99 (albeit with warning) I'd like 1 1 here
3 4

TC$ f='%1$s %s\n'   ; printf "$f" 1 2 3 4
1 2 <-- WRONG same as above.
3 4

TC$ f='%1$s %1$s\n' ; printf "$f" 1 2  
1 1 <-- lucky one
2 2
 
TC$ f='%1$d %1$s\n'  ; printf "$f" 1 2
Segmentation fault  <-- zsh gives 1 1 \n 2 2 meaning zsh convert argv[] for each occurence [1], something C99 prohibit.

f='%1$s %*1$s\n' ; printf "$f" 3 4     
loop, not infinite, just n loop very big. zsh convert each argv[1] 
%1$s get argv[1] as a string "3"
%*1$s get argv[1] as an int width "3" for next %s that is "4" 
 
TC$ f='%1$s %1$d\n'  ; printf "$f" 1 2 
1 1487268033 <-- WRONG 
2 1487268065

There are many more case of failure just providing a few.

So fix it or not ?

Thanx for your help.

[hyenias]

Thanks for all the research and examples. In my opinion, I believe mix'n'match should not be allowed unless it is defined in some way in POSIX. The Wikipedia printf Parameter field section for n$ (which is referenced above as %x$) mentions, "This is a POSIX extension and not in C99." I wonder if one of our POSIX group community members could hunt this specification down and enlightened us to its contents.

Wikipedia continues within that section and states:

If any single placeholder specifies a parameter, all the rest of the placeholders MUST also specify a parameter.

[hyenias]

POSIX printf reference.

[ormaaj]

Maybe I missed it but In case it hasn't been said - numbered conversion specifiers are a required feature for printf(1) in issue 8. POSIX does explicitly leave the behavior for "mixed" argument types unspecified.

The format can contain either numbered argument conversion specifications (that is, ones
beginning with "%n$"), or unnumbered argument conversion specifications, but not
both. The only exception to this is that "%%" can be mixed with the "%n$" form. The
results of mixing numbered and unnumbered argument specifications that consume an
argument are unspecified.

Personally I find disallowing mixed conversion arguments pretty useless and think being able to mix them would be intuitive and useful. I don't foresee how this would have any impactful side-effects, seems to me it would be a win. The correct mechanics for this are pretty obvious - just keep incrementing the ordinal position by one for each specifier that doesn't have it given.

[phidebian]

That's what the proposed patch is doing, non-indexed args are as if numbered sequentially so

%s %s %s is equivalent to %1$s %2$s %3$s

and

%*.*s is equivalent to %3$*1$.2$s that may look non intuitive but match the definition found in printf(3)

Excerp from my docco (man 3 printf)

...
 One can also specify explic-
       itly  which  argument  is taken, at each place where an argument is re-
       quired, by writing "%m$" instead of '%' and "*m$" instead of '*', where
       the  decimal integer m denotes the position in the argument list of the
       desired argument, indexed starting from 1.  Thus,

           printf("%*d", width, num);

       and

           printf("%2$*1$d", width, num);
...

EDIT: Forgot to precise that arg index is one part of the problem, the other is as you said the 'type' of the args, as it is well easy to create type mismatch, the approach of the patch is to do late dynamic type conversion. So as it is now, it is possible to write ugly things like %1$f %1$d %1$s without core dumping, this approach is partially the one used with zsh, but then they still have problems that we don't have (until bugs are founds indeed), see at the end of this discussion.

[phidebian]

Thanx @hyenias , it is a start of reflexion :-) .

I started my work from printf(3) docco that sez (on my linux distro)

The C99 standard does not include the style using '$',
       which comes from the Single UNIX Specification.  If the style using '$'
       is used, it must be used throughout for all conversions taking an argu-
       ment and all width and precision arguments, but it may  be  mixed  with
       "%%"  formats,  which do not consume an argument. 

So doesn't mention posix anywhere, when you talk about posix, did you mean posix for C or posix for shell's ? I am totally ignorant about posix thing, so may be posix itself define both, and if so I guess they have 2 sections for printf, one for printf(1) and one for printf(3) ? and indeed they differ a lot, and probably in incompatible ways (consumed argv[], mix'n'match)

So far we got 3 options.

• withdraw any x$ (bash)
• no mix'n'match, (A bit like C libc from the above docco)
• Working mix'n'match This is doable, I got a patch almost working, but if the majority goes with one of the 2 above, there is no need to get to this complexity.

Now regarding NO mix'n'match, even this one is non trivial, and ksh as it is is not completly ready for it.

All my examples are written f=fmt-string ; printf .... because that's the format used in test suite and I simply cut/paste here....

With a shell (in the sh family) is is perfectly possible to write f='%1$d %1$s\n' ; printf "$f" 1 2 (as demonstrated in prev post), this can be qualified as full %x$ (not mixed with %s) so non mix'n'match and yet illegal in C, and prone to buggy implementation in a shell.

zsh try to be smarter than ksh here and to delay the argv[] conversion very late in the format scan, allowing %1$d to fetch argv[0] and use the numeric expression as a number, emit result, then later on fetch argv[1] again and this time use it as a string.

The ksh algo is way broken here, and need a lot of band aid around the existing code. It looks like the code was designed for non $ at all (ksh archeologist may confirm) then one decided to implement x$ like this.

On first $ occurence in a format string, it trig the build of yet another argv[] kind called fp[] (where may be the 'p' stand for position) and this one is bugged, it grab the first occurence of an index (say 1 that is argv[0]) and enter in fp[0].fmt to the %format associated with 1$ for instance %1$d, fd[0].fmt='d' and fp[0].value=numeval(argv[0]), and that's it for 1$, next occurence of 1$ is considered 'cached' as dejavu, and next %1$s will not re-eval argv[0] will use the previous number as a string, with all the crash we got.

What I describe here is indeed worsened by x$ occurences in width/precision. The term mix'n'match in this discussion is not limited to %1$d %1$s, but indeed apply to width/prec as %1$*d' where 1$ is used for width and non $ access is used for 'd', it is a mix'n'match because of $ and non $, but %1$*s' is super mix'n'match because not only we mix $ and non $ but 1$ is used as a number for width and as a string of 's' (assuming 's' is the first occurence of non $ in the format string)

And when even this is solved (mix'n'match), then there must be an agreement about how to handle non $ access following $ access for width/precision. like here

• ksh/zsh

f='%*2$.*2$d %d %d\n'     ; printf  "$f" 1 2 3
03 0 0

• C

 echo $'#include <stdio.h>
int main(int c, char **v){printf("%*2$.*2$d %d %d\\\\n",1,2,3);}
' | cc -x c - -o yo.$$ && ./yo.$$  && rm yo.$$
<stdin>: In function ‘main’:
<stdin>:2:34: warning: missing $ operand number in format [-Wformat=]
01 2 3

As you see here the warning is due to the mix'n'match of $ vs non $, yet this is just a warning and the semantic applied here (and logical to me) is that any non $ are evaluated like x$ with x starting at 1 and incremented on each non $ so
'%*2$.*2$d %d %d\n' is equivalent to '%1$*2$.*2$d' %d %d\n' because 'd' is the 1st non $ in the format string, and then the 01 output is logical.

ksh/zsh god knows why decided the non $ 'd' is right after the last x$ for width/prec i.e %*2$d is %3$*2$d, this is a big problem because existing ksh script may rely on this and fixing it the C way will break them. And even worst I bet the %*2$.*2$d is probalbly the most common used when start dealing is %. even though with shell it is simpler to expand width/prec inside the format string "%$n.${n}d" is probably better than '%*2$.*2$d' with n in position 2. I suspect ksh having open the wrong door of 03 and zsh may have adapted to that ...

May be implementing the C way would trig some regression in our current QA suite (I am not that far at the moment)

So I still query to vox populi, if we go the fix it path, should we put back the C semantic for tolerated (warning) mix'n'match,.

My vote is C way but I may be the only one :)

[hyenias]

From my ksh viewpoint, ksh is a shell not C. 1st priority is to try to align ksh with POSIX shell standards. Next, as feature sets go; evaluate what the other Bourne shell variants do and see what the merits of aligning with them; this helps reduce the learning curve as well as compatibility. But please be aware that I am 100% for the KornShell, so KSH RULES!

To that end, I wish ksh to be faster and better than all the rest. This means, in general, I want to keep ksh's feature set. So, for me, I would like to keep the printf parameter indexing as repeating parameters (the argv[...] references) or reordering their output is clearly beneficial.

With the plain sequential parameter access via %s this is a must to keep. It is the basics. From my viewpoint, I wish to keep both of those options. You have pointed out various complexities and failures of ksh's implementations of these options.

hyenias summary:

1. ksh is a shell. In general, lean towards following other shells over following compiled C. Preference for alignment of shells is mksh then zsh then bash.
2. I am against, "withdraw any x$ (bash)". I would like to keep x$ parameter indexing. If the solution prevents mixing those formatting options with %s or such then limit it but still keep that functionality.
3. For "Working mix'n'match This is doable, I got a patch almost working, but if the majority goes with one of the 2 above, there is no need to get to this complexity." option then seems to be the best of all worlds at this time. As I am not the one coding the solution then the only thing I would like to evaluate is its performance. Remember, I always want ksh to be faster and better than all the rest. I am greedy that way. Is this enough for you to continue working on your best of all worlds approach? Not sure, but you got my vote if that helps justify your time.

[phidebian]

Thanx @hyenias
OK I guess I understand what you mean, ksh is closer to C than bash is because we have typed argv[], i.e %d expect an integer argv[], yet ksh is not C so we have format-string repeat until all argv[] are consumed, something that C don't have, and potentially, we can do with ksh/zsh (bash don't have x$) integer i=1;printf '%1$s %1$d\n' i i i.e departing from C.

• zsh

$ integer i=1;printf '%1$s %1$d\n' i i
i 1
i 1

• ksh

$ integer i=1;printf '%1$s %1$d\n' i i
i 491374945
i 491374977

Despite ksh is wrong here this is easily fixable and would produce zsh output after fix, this is a case where we can consider zsh correct, and establish the norm.

Now this one is problematic to me %*1$.*2$d

      +--<--<--<-- Indexed access     argv[0]
      |   +--<--<- Indexed access     argv[1]
      |   | +--<-- Non-indexed access argv[?]
      v   v v
    %*1$.*2$d

• zsh

 $ f='%*1$.*2$d %d %d\n'     ; printf  "$f" 1 2 3
03 0 0

$ f='%*2$.*1$d %d %d\n'     ; printf  "$f" 1 2 3
 2 3 0

Why 3 on first instance and 2 on the 2nd, we infere the non indexed access is 1 past the last indexed access, dunno why we can consider that logic, and it would require extensive docco to explain.

• ksh

$ f='%*1$.*2$d %d %d\n'     ; printf  "$f" 1 2 3 
03 0 0

$ f='%*2$.*1$d %d %d\n'     ; printf  "$f" 1 2 3
d d d  # would be 3 0 0 if fixed...

Ksh logic (even though bugged here), would favor the max of the indexed access so here would be 3 for both.
Again not too logical, departed from zsh (and C) , require docco.

To me it is not logical there is no reasons to skip 1 for first non indexed access, beside a fully qualified indexed access is always possible i.e %1$*2$.*1$d (last 'd' is qualified by 1st '1$' and both ksh/zsh would provide the same correct answer.

So I'd like a decision on this last point, do we go the erratic zsh way, do they establish the norm? do we wait POSIX, do we add a printf flag -z behave like zsh (last indexed) , -k behave like ksh (max indexed) -s (sequential are respected) [-p] posix, adapt to what posix will say after the y3k bug ? (kidding)

[hyenias]

Hmmmm... I prefer not to add more flags to printf to perform differing behavior of indexed/sequential parameter access. Consistency is key. Now, if there is a performance impact to doing something uncommon then sure put that uncommon stuff/behavior/special case behind a flag.

I have not looked at the code. As I think I understand it from your input, ksh's version of printf consumes the parameters passed along it probably turning them into a token and popping them one out at a time. So as ksh printf processes along the parameter tokens, it consumes them thus putting us into the hard to resolve situation for out-of-order parameter access as the format string is parsed.

Is this what is going on? If the argv array is being consumed, can we not rewrite the code to use pointers to the argv array to keep all the original parameters intact thus allowing all the references either absolute or relative to be accessible and understood?

I do not like erratic anything. So, not the zsh, ksh, or bash way of doing things as none of it is clear much less seemingly correct.

We do not wait on POSIX.

If I could have it my way (not necessarily the correct way but you asked me), I keep the whole process KISS. This means, I would keep all passed parameter values intact so that they can be referenced positionally either through direct absolute indexed referencing or via next sequential value after the last used indexed position. Is this your "last indexed" way?

Not sure how one would move the sequential access along to prevent infinite loop.

Did I help answer your questions?

[phidebian]

Thanx @hyenias , I'll take all this in consideration. I think I will come with a 'fix' candidate, trying to keep all thins in mind.

I'll try to make it 'configurable', i.e will come up with one implementation of the %*X$.*Y$F ( indexed for width/prec but not for F format letter) i.e either pure seuential for F access, or one past the last seen width/prec ndx, or one past the max of witdth/prec/ndx) will do this in a way we can change at the last minute the implementation we prefere, and if nobody agry, i.e two ways are acceptable then will see....

The fix I envision as no perf impact on 'regular' path, will trig a format rescan on ambigous indexed access, i.e all the strange questionable mix'n'match allowed to exist (compared to a compiler kind that would reject such mix'n'match)

I'll give it a try and see what I come up with.

Note that I guess this fix is marginal, I mean nobody do those mix'n'match, most people simply do %s sometime with hardcoded width/prec, and if really needed do var expansion in the format string to get a parameterizable prec/width i.e '%3.4d' or "%$w.${p}s", what really bugged me here is how easily one can segv or loop on the wrongly setup format string.

[phidebian]

Made a candidate fix

The patch is posted at
https://github.com/phidebian/ksh/tree/bug-324-1
and is ksh93/ksh PR
#617

This is a call for testers, i.e verify and accept/reject the new output, and a call for code review if accepted.

Here is the script I use to generate outputs, and generate the QA suite.

#!/bin/bash
# ./printf.sh [-o] [ShellName]

o=0 ; [ "$1" = "-o" ] && o=1 && shift
K=/d1/ksh/arch/linux.i386-64/bin/ksh # Fix candidate
[ "$1" ] && K="$1"                   # production zsh | ksh
 
for f in \
'%s %s\n'  \
'%s %1$s\n' \
'%1$s %s\n' \
'%1$s %1$s\n' \
'%1$s %s %s %2$s\n' \
'%s %1$s %s %s\n' \
'%s %1$s %2$s %s\n' \
'%2$s %s\n' \
'%2$s %1$s %2$s %1$s\n' \
'%2$s %1$s %s %2$s %1$s %s\n' \
'%*2$s %s\n' \
'%*2$.*3$s %s\n' \
'%*3$.*2$s %s\n' \
'%*s %*.*s\n' \
'%*d %*.*d\n' \
'%*f %*.*f\n' \
'%3$*2$.*1$s %2$s %1$s\n' \
'%s %5$s\n' \
; do
echo

((o)) &&
{ echo "printf '$f' 1 2 3 4 5 6 7 8 9 0"
  $K -c "printf '$f' 1 2 3 4 5 6 7 8 9 0"
  echo
  continue
}

x=$($K -c "printf '$f' 1 2 3 4 5 6 7 8 9 0")
echo "x='$x";echo "'"
echo "f='$f'"
echo 'T $LINENO "$f" "$x" # err_exit src/cmd/ksh93/tests/shtests grep -c accounting'
done

For testers it can be used like this ksh-b stand for bad, i.e my distro ksh, ksh-f for fixed.

./printf.sh -o >ksh-f.lst     2>&1
./printf.sh -o ksh >ksh-b.lst 2>&1
./printf.sh -o zsh >zsh.lst   2>&1

The current ksh-f.lst is

printf '%s %s\n' 1 2 3 4 5 6 7 8 9 0
1 2
3 4
5 6
7 8
9 0


printf '%s %1$s\n' 1 2 3 4 5 6 7 8 9 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
0 0


printf '%1$s %s\n' 1 2 3 4 5 6 7 8 9 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
0 0


printf '%1$s %1$s\n' 1 2 3 4 5 6 7 8 9 0
1 1
2 2
3 3
4 4
5 5
6 6
7 7
8 8
9 9
0 0


printf '%1$s %s %s %2$s\n' 1 2 3 4 5 6 7 8 9 0
1 1 2 2
3 3 4 4
5 5 6 6
7 7 8 8
9 9 0 0


printf '%s %1$s %s %s\n' 1 2 3 4 5 6 7 8 9 0
1 1 2 3
4 4 5 6
7 7 8 9
0 0  


printf '%s %1$s %2$s %s\n' 1 2 3 4 5 6 7 8 9 0
1 1 2 2
3 3 4 4
5 5 6 6
7 7 8 8
9 9 0 0


printf '%2$s %s\n' 1 2 3 4 5 6 7 8 9 0
2 1
4 3
6 5
8 7
0 9


printf '%2$s %1$s %2$s %1$s\n' 1 2 3 4 5 6 7 8 9 0
2 1 2 1
4 3 4 3
6 5 6 5
8 7 8 7
0 9 0 9


printf '%2$s %1$s %s %2$s %1$s %s\n' 1 2 3 4 5 6 7 8 9 0
2 1 1 2 1 2
4 3 3 4 3 4
6 5 5 6 5 6
8 7 7 8 7 8
0 9 9 0 9 0


printf '%*2$s %s\n' 1 2 3 4 5 6 7 8 9 0
 1 2
   3 4
     5 6
       7 8
9 0


printf '%*2$.*3$s %s\n' 1 2 3 4 5 6 7 8 9 0
 1 2
    4 5
       7 8
 


printf '%*3$.*2$s %s\n' 1 2 3 4 5 6 7 8 9 0
  1 2
     4 5
        7 8
 


printf '%*s %*.*s\n' 1 2 3 4 5 6 7 8 9 0
2   5
     7        0


printf '%*d %*.*d\n' 1 2 3 4 5 6 7 8 9 0
2 0005
     7 000000000


printf '%*f %*.*f\n' 1 2 3 4 5 6 7 8 9 0
2.000000 5.0000
7.000000 0.000000000


printf '%3$*2$.*1$s %2$s %1$s\n' 1 2 3 4 5 6 7 8 9 0
 3 2 1
    6 5 4
       9 8 7
  0


printf '%s %5$s\n' 1 2 3 4 5 6 7 8 9 0
1 5
6 0

The output is almost on par with zsh the diffs are

KSH
printf '%*2$s %s\n' 1 2 3 4 5 6 7 8 9 0
 1 2
   3 4
     5 6
       7 8
9 0                                     

ZSH
printf '%*2$s %s\n' 1 2 3 4 5 6 7 8 9 0 
 3 4
     7 8
 
===========================================
     
KSH
printf '%*2$.*3$s %s\n' 1 2 3 4 5 6 7 8 9 0
 1 2
    4 5
       7 8

ZSH
printf '%*2$.*3$s %s\n' 1 2 3 4 5 6 7 8 9 0
 4 5
      9 0

===========================================

KSH
printf '%3$*2$.*1$s %2$s %1$s\n' 1 2 3 4 5 6 7 8 9 0
 3 2 1
    6 5 4
       9 8 7
  0

ZSH
printf '%3$*2$.*1$s %2$s %1$s\n' 1 2 3 4 5 6 7 8 9 0
zsh:printf:1: 3: argument specifier out of range
 3 2 1
    6 5 4
       9 8 7

IMHO all zsh diff are wrong, unless someone explain.
Bear in mind that man printf(3) sez

 Thus,   printf("%*d", width, num);    and     printf("%2$*1$d", width, num);  are equivalent

meaning non indexed format (i.e seq format % and *) are like auto numbered indexed format where the first non indexed format start at index one and so on.