Why are expansions in array subscripts not processed?

[ormaaj]

 ~ $ echo; ksh /dev/fd/9 9<<\EOF
typeset -a a
printf -v 'a[${#a[@]}]' %s hi
{ IFS= read -ru "${a[-1]}" 'a[${#a[@]}]'; } {a[${#a[@]}]}<<<hello
typeset -p a
EOF

ksh[2]: printf: ${#a[@]}: arithmetic syntax error
ksh: line 2: ${#a[@]}: arithmetic syntax error
typeset -a a

This is just one contrived scenario but this class of problem crops up all over the place. It generally assumes the subscript is a raw arithmetic expression for indexed arrays and a literal string in the case of associative arrays. I've run into variations on this problem countless times and sometimes it's impossible to work around.

I believe ksh has always been like this but I haven't seen any discussion about it. This seems like a generic problem with the way variables are handled.

[McDutchie]

I fail to see why this is a problem. To process $ expansions, simply use double quotes (") instead of single quotes (').

If we added another expansion pass in the processing of subscripts itself, we'd get a double evaluation problem – and a code injection vulnerability when untrusted values are used as array subscripts.

[ormaaj]

I guess this deserves further elaboration as this is a rather nuanced subject. Importantly, printf -v "$string" where string is not deterministically controlled by the program is a programmer error. The example you've written therefore isn't necessarily a problem and in fact that example is correct when the resulting string is generated correctly. That isn't really a code injection unless the subscript variable in your example isn't constructed properly.

There's an additional layer to the problem in situations where the shell forces a pre-expansion pass. (()) is a common trouble spot for that. The correct way to pass an array into an arithmetic expression in cases where an expansion is required is (in bash 5.1 and earlier, and zsh) e.g.:

(( arr[\${var}] + 1 ))

This escapes the initial expansion in order to allow arr[$var] to be properly passed to the variable resolver by the arithmetic evaluator. In ksh, it is impossible to write this correctly. Instead one might be tempted to not escape $var, thereby injecting the string into the expression prior to the arithmetic evaluation stage, which is a rather big problem.

[McDutchie]

You've double quoted the printf argument, introducing an injection vulnerability.

Sorry, but that is nonsense. Double quotes don't cause an injection vulnerability and are the safe way to pass data coming from variables. E.g.:

$ export var='$(echo INJECTION! >&2)'
$ bash -c 'echo "a[$var]"'
a[$(echo INJECTION! >&2)]

The command substitution is not executed.

$ bash -c 'printf -v "a[$var]" x'
INJECTION!

The command substitution is executed.

This is a vulnerability in array subscripts in bash, not in double quotes.

The problem with ksh is that it in many cases forces you to perform an improper injection into an array subscript rather than safely allowing the subscript to be resolved as the variable is looked up.

It does no such thing.

[ormaaj]

Sorry, but that is nonsense. Double quotes don't cause an injection vulnerability and are the safe way to pass data coming from variables. E.g.:

...that's not true. What I wrote is absolutely correct.
It is not a vulnerability. It is a programming error to use uncontrolled variables in this manner.

This is not as simple as you think.

[McDutchie]

What you wrote is absolutely wrong.

It is that simple on ksh, and I'm keeping it that way.

[ormaaj]

...In any event, there are so many scenarios where this is a problem that it's difficult to enumerate them and I figured I'd have to fix this myself anyway because there are a lot of details that are not widely understood and hard to communicate. Just probing to see if anyone out there with considerable experience with this has some extra insight.

I suppose your feedback is ironically useful as If the real reason for omitting these expansions is as stated then this is straightforward and my understanding of the issue is complete.

My actual reason for asking is that I suspect the real motivation for the current behavior had more to do with future design directions that wouldn't be particularly evident in the present state..

[ormaaj]

 $ ( bool=([1]=); for sh in zsh bash /dotfiles/bin/ksh93v ksh; do printf '\n%s:\n' "${sh##*/}"; [[ $sh != zsh ]]; "$sh" ${bool[$?]+'--emulate'} ${bool[$?]+'ksh'} /proc/self/fd/9 9<<\EOF9; done )
[[ -v BASH_VERSION ]] && shopt -u assoc_expand_once

function f {
        typeset x=\] y=5
        typeset -A arr=([$x]=10)
        let 'y += arr[$x]'
        typeset -p arr y
}

f
EOF9

zsh:
typeset -A arr=( [']']=10 )
typeset y=15

bash:
declare -A arr=(["]"]="10" )
declare -- y="15"

ksh93v:
typeset -A arr=([']']=10)
y=5

ksh:
typeset -A arr=(['$x']= [']']=10)
y=5

Ugh. Awful.

I have solutions in mind that benefit from other nice properties of ksh conducive to making this work cleanly.

[McDutchie]

ksh 93u+m is the only shell in your examples that acts in a sane manner. At least bash now acts sane by default, though really they shouldn't have that assoc_expand_once option at all.

The command let 'y += arr[$x]' uses single quotes. It is basic, fundamental shell grammar that single quotes do not perform any expansions such as $x. So in this context, $x is, and absolutely must be, a literal associative array index '$x'.

Any expansion of single-quoted strings is a flagrant violation of POSIX, security, user expectations, and basic sanity, and I am not having any of that dangerous nonsense in my shell. We fixed that vulnerability in b48e5b3 (see #152) and it is staying fixed. This is not negotiable. Scripts that depend on that vulnerability are simply broken.

To use arbitrary associative array subscripts, you need a lexical analysis pass before the expansion of $x. So, compatibility with pathological corner-case array subscripts such as ] precludes the use of let in this case, even when using double instead of single quotes.

Which means your reproducer above is easily fixed by using ((y += arr[$x])) or y=$((y + arr[$x])) instead of that let command.

[ormaaj]

To use arbitrary associative array subscripts, you need a lexical analysis pass before the expansion of $x.

Horray this sentence I agree with.

A lexical analysis pass is needed to find the subscript boundaries. Since the arithmetic grammar is effectively separate from everything else it shouldn't be very hard avoid parsing shell code at all unless it's actually required for snipping out variables with subscripts that contain a potential token initiating an expansion.

I might agree with something else if things were radically different than they are.

[ormaaj]

Changed my mind about closing discussions... apparently the thread-starter is responsible for moderating. Kind of odd.