Refactor argument parsing and passing to a json-based approach instead of code injection

.github/workflows/sbt_test.yml

@@ -50,14 +50,34 @@ jobs:
     - name: Set up sbt
       uses: sbt/setup-sbt@v1
 
-    - name: Set up Scala
+    - name: Set up Scala (via Coursier)
+      uses: coursier/setup-action@v2
+      with:
+        apps: scala:3.6.2 bloop
+
+    - name: Warm up scala
       run: |
-        if [[ "${{ matrix.config.os }}" =~ ^macos.*$ ]]; then
-          brew install scala
-        else
-          sudo apt-get update
-          sudo apt-get install -y scala
-        fi
+        # start the bloop server, it'll download a lot of dependencies, so do it early
+        # also we need to start it before running any scala code, otherwise it will often timeout and cause the whole job to fail
+        bloop server &
+        for i in $(seq 1 30); do
+          timeout 5 bloop about 2>/dev/null && echo "Bloop is ready!" && break
+          echo "Waiting for Bloop... ($i/30)"
+          sleep 2
+        done
+
+        scala -version
+        scala -e 'println("Hello, Scala!")'
+
+    - name: Set up .NET SDK
+      if: ${{ runner.os == 'Linux' }}
+      uses: actions/setup-dotnet@v4
+      with:
+        dotnet-version: '9.0.x'
+
+    - name: Set up dotnet-script
+      if: ${{ runner.os == 'Linux' }}
+      run: dotnet tool install -g dotnet-script
 
     - name: Set up Python
       uses: actions/setup-python@v6
@@ -91,7 +111,7 @@ jobs:
 
     - name: Upload check results on fail
       if: failure()
-      uses: actions/upload-artifact@master
+      uses: actions/upload-artifact@v6
       with:
         name: ${{ matrix.config.name }}_results
         path: check

CHANGELOGS/CHANGELOG_0.10.md

@@ -10,6 +10,11 @@ TODO add summary
   This allows for running a component without having to build it first.
   Example: `viash run vsh://toolbox@v0.1.0/yq -- --input input.yaml --output output.yaml`.
 
+* `Parameter passing`: Add support for unsetting argument values and computational requirements at runtime (PR #762, fixes #375).
+  * Pass the literal `UNDEFINED` (unquoted) to set a single-value argument to undefined/null: `./my_component --arg UNDEFINED`
+  * Pass `UNDEFINED_ITEM` as a value in a multi-value argument to represent a missing item: `./my_component --args "value1;UNDEFINED_ITEM;value3"`
+  * Unset computational requirements with `---cpus UNDEFINED` or `---memory UNDEFINED`
+  * Quote the values (`'"UNDEFINED"'` or `"'UNDEFINED'"`) to pass the literal string `UNDEFINED` instead of null.
 
 ## BREAKING CHANGES
 
@@ -19,10 +24,25 @@ TODO add summary
 
 * Remove deprecated functionality `functionality` and `platforms` (PR #832).
 
+* `Bash scripts`: Arguments with `multiple: true` are now stored as bash arrays instead of semicolon-separated strings (PR #762).
+  Update scripts to use array syntax: `for item in "${par_inputs[@]}"; do ...` instead of IFS splitting.
+
 ## BUG FIXES
 
 * `NextflowRunner`: Automatically convert integers to doubles when argument type is `double` (port of PR #824, PR #825).
 
+* `Parameter passing`: Fix handling of special characters in argument values (PR #762, fixes #619, #705, #763, #821, #840).
+  * Backticks in argument values no longer cause command substitution
+  * Backslash-quote sequences (`\'`) no longer break Python syntax
+  * Dollar signs, newlines, and other special characters are properly preserved
+
 ## MINOR FIXES
 
 * `Executable`: Add more info to the --help (PR #802).
+
+## INTERNAL CHANGES
+
+* `Parameter passing`: Switch from code injection to JSON-based parameter passing (PR #762).
+  Instead of injecting argument values directly into script code, values are now stored in a JSON file
+  (`params.json`) and parsed at runtime using language-specific JSON parsers. This approach is more
+  robust, easier to debug, and handles special characters (backticks, quotes, newlines) correctly.

Makefile.in

@@ -10,7 +10,11 @@ $(obj): $(shell find src/ -name "*.scala")
 	cat src/stub.sh target/viash.jar > $(obj)
 	chmod +x $(obj)
 	@echo "==================================> $(obj)"
-
+
+touch: $(obj)
+	@echo "Touching src/main/scala/io/viash/Main.scala to trigger rebuild"
+	touch src/main/scala/io/viash/Main.scala
+
 tools: $(obj) $(shell find src/viash)
 	bin/viash ns build -s src/viash -t bin --flatten -c ".version := '$(VERSION)'"
 	@echo "==================================> tools"

docs/reference/viash_code_block/index.qmd

@@ -95,6 +95,28 @@ The `par` object (or `par_` environment variables in Bash) will contain argument
 Try adding more [arguments]({{< var reference.arguments >}}) with different types to see what effect this has on the resulting placeholder.
 :::
 
+### Special values: undefined and missing items
+
+When calling a Viash component, you can use special values to explicitly set arguments to undefined or represent missing items in multi-value arguments:
+
+* **Unsetting a single-value argument**: Pass the literal `UNDEFINED` (unquoted) to set a single-value argument to undefined/null:
+  ```bash
+  ./my_component --arg UNDEFINED
+  ```
+  In the script, `par["arg"]` will be `None` (Python), `NULL` (R), `null` (JavaScript), or unset (Bash).
+
+* **Missing items in multi-value arguments**: When passing multiple values via semicolon-separated syntax, use `UNDEFINED_ITEM` to represent a missing element:
+  ```bash
+  ./my_component --values "item1;UNDEFINED_ITEM;item3"
+  ```
+  In the script, `par["values"]` will be `["item1", None, "item3"]` (Python) or equivalent.
+
+* **Passing the literal string "UNDEFINED"**: Quote the value to pass it as a literal string:
+  ```bash
+  ./my_component --arg '"UNDEFINED"'     # par["arg"] = "UNDEFINED"
+  ./my_component --arg "'UNDEFINED'"     # par["arg"] = "UNDEFINED"
+  ```
+
 ## Meta variables in `meta`
 
 Meta-variables offer information on the runtime environment which you can use from within your script.
@@ -215,6 +237,9 @@ viash test config.vsh.yaml --cpus 10
 viash build config.vsh.yaml -o output
 output/my_executable ---cpus 10
 #                     ↑ notice the triple dash
+
+# to unset a default cpu value, pass UNDEFINED
+output/my_executable ---cpus UNDEFINED
 ```
 
 ### `config` (string)
@@ -267,6 +292,9 @@ viash test config.vsh.yaml --memory 2GB
 viash build config.vsh.yaml -o output
 output/my_executable ---memory 2GB
 #                     ↑ notice the triple dash
+
+# to unset a default memory value, pass UNDEFINED
+output/my_executable ---memory UNDEFINED
 ```
 
 ### `resources_dir` (string)

src/main/resources/io/viash/helpers/bashutils/ViashCleanupRegistry.sh

@@ -0,0 +1,30 @@
+# Cleanup handler registry
+# Allows multiple cleanup functions to be registered and called on exit
+
+# Array to store registered cleanup function names
+VIASH_CLEANUP_HANDLERS=()
+
+# ViashRegisterCleanup: Register a cleanup function to be called on exit
+# $1: Name of the function to call during cleanup
+# usage: ViashRegisterCleanup my_cleanup_function
+function ViashRegisterCleanup {
+  local handler="$1"
+  VIASH_CLEANUP_HANDLERS+=("$handler")
+}
+
+# ViashRunCleanupHandlers: Run all registered cleanup handlers in reverse order
+# This function is meant to be used as the EXIT trap handler
+function ViashRunCleanupHandlers {
+  # Run handlers in reverse order (LIFO - last registered runs first)
+  local i
+  for (( i=${#VIASH_CLEANUP_HANDLERS[@]}-1 ; i>=0 ; i-- )); do
+    local handler="${VIASH_CLEANUP_HANDLERS[$i]}"
+    if type "$handler" &>/dev/null; then
+      ViashDebug "Running cleanup handler: $handler"
+      "$handler"
+    fi
+  done
+}
+
+# Set up the master EXIT trap that runs all registered handlers
+trap ViashRunCleanupHandlers EXIT

src/main/resources/io/viash/helpers/bashutils/ViashDockerFuns.sh

@@ -189,7 +189,7 @@ function ViashDockerSetup {
 # $1                  : image identifier with format `[registry/]image[:tag]`
 # $@                  : commands to verify being present
 # examples:
-#   ViashDockerCheckCommands bash:4.0 bash ps foo
+#   ViashDockerCheckCommands bash:3.2 bash ps foo
 function ViashDockerCheckCommands {
   local image_id="$1"
   shift 1
@@ -220,10 +220,13 @@ function ViashDockerBuild {
   # create temporary directory to store dockerfile & optional resources in
   local tmpdir=$(mktemp -d "$VIASH_META_TEMP_DIR/dockerbuild-$VIASH_META_NAME-XXXXXX")
   local dockerfile="$tmpdir/Dockerfile"
-  function clean_up {
-    rm -rf "$tmpdir"
+
+  # Use a unique cleanup function name to avoid conflicts
+  VIASH_DOCKER_BUILD_TMPDIR="$tmpdir"
+  function ViashDockerBuildCleanup {
+    rm -rf "$VIASH_DOCKER_BUILD_TMPDIR"
   }
-  trap clean_up EXIT
+  ViashRegisterCleanup ViashDockerBuildCleanup
 
   # store dockerfile and resources
   ViashDockerfile "$VIASH_ENGINE_ID" > "$dockerfile"

src/main/resources/io/viash/helpers/bashutils/ViashParseArgumentValue.sh

@@ -0,0 +1,191 @@
+# ViashParseArgumentValue: Parse the value of an argument
+# 
+# This script is used by to parse the value of an argument and set
+# the corresponding environment variable.
+# 
+# If the argument is multiple: false:
+# 
+# * If the variable is already set, an error is thrown.
+# * If the value is equal to UNDEFINED, it is replaced with
+#   @@VIASH_UNDEFINED@@.
+# * If the value is quoted, the quotes are removed.
+#
+# If the argument is multiple: true:
+#
+# * If the value is equal to UNDEFINED, it is replaced with
+#   @@VIASH_UNDEFINED@@.
+# * If the value is quoted, the quotes are removed.
+# * If the value is a list of values, the values are split by semicolons.
+# * If the value contains escaped characters '"\;, they are unescaped.
+#
+# Arguments:
+# $1: The name of an argument (used for error messages)
+# $2: The name of the environment variable to set
+# $3: Whether the argument can be passed multiple times (true/false)
+# $4: The value of the argument
+# return: None, but sets the environment variable
+# 
+# Examples:
+#
+# ViashParseArgumentValue "--input" "par_input" "true" "'UNDEFINED'" "false"
+#
+# See ViashParseArgumentValue.test.sh for additional examples.
+#
+# Note: This function is written to be compatible with bash 3.2 (macOS default)
+# and avoids bash 4+ features like declare -g, local -n, ${var@Q}, and readarray.
+function ViashParseArgumentValue {
+  local flag="$1"
+  local env_name="$2"
+  local multiple="$3"
+  local value="$4"
+
+  if [ $# -lt 4 ]; then
+    ViashError "Not enough arguments passed to ${flag}. Use '--help' to get more information on the parameters."
+    exit 1
+  fi
+
+  if [ "$multiple" == "false" ]; then
+    # check whether the variable is already set (bash 3.2 compatible)
+    eval "local is_set=\${${env_name}+x}"
+    if [ ! -z "$is_set" ]; then
+      eval "local prev_value=\"\${$env_name}\""
+      ViashError "Pass only one argument to argument '${flag}'. Found: '${prev_value}' & '${value}'"
+      exit 1
+    fi
+
+    value=$(ViashParseSingleString "$value")
+
+    # set the variable globally (bash 3.2 compatible - using eval instead of declare -g)
+    # use printf %q to safely escape the value for eval to prevent backtick/command substitution
+    local escaped_value
+    escaped_value=$(printf '%q' "$value")
+    eval "$env_name=$escaped_value"
+  else
+    # Get existing array values (bash 3.2 compatible)
+    eval "local prev_values=(\"\${${env_name}[@]}\")"
+
+    # Parse new values into array (bash 3.2 compatible - using while loop instead of readarray)
+    local new_values=()
+    while IFS= read -r line || [ -n "$line" ]; do
+      new_values+=("$line")
+    done < <(ViashParseMultipleStringAsArray "$value")
+
+    local combined_values=( "${prev_values[@]}" "${new_values[@]}" )
+
+    # if length is larger than 1 and some element is @@VIASH_UNDEFINED@@, throw error
+    if [ ${#combined_values[@]} -gt 1 ]; then
+      for element in "${combined_values[@]}"; do
+        if [ "$element" == "@@VIASH_UNDEFINED@@" ]; then
+          # bash 3.2 compatible quoting (using printf %q instead of ${var@Q})
+          local combined_quoted=$(printf '%q ' "${combined_values[@]}")
+          ViashError "Argument '${flag}': If argument value 'UNDEFINED' is passed, no other values should be provided.\nFound: ${combined_quoted}"
+          exit 1
+        fi
+      done
+    fi
+
+    # Set the global array (bash 3.2 compatible - using eval instead of declare -g -a)
+    eval "$env_name=(\"\${combined_values[@]}\")"
+  fi
+}
+
+function ViashParseSingleString() {
+  local value="$1"
+
+  # if value is equal to UNDEFINED, replace with @@VIASH_UNDEFINED@@
+  if [ "$value" == "UNDEFINED" ]; then
+    value="@@VIASH_UNDEFINED@@"
+  fi
+
+  # if value is quoted, remove the quotes
+  if [[ "$value" =~ ^\".*\"$ ]]; then
+    value="${value:1:${#value}-2}"
+  elif [[ "$value" =~ ^\'.*\'$ ]]; then
+    value="${value:1:${#value}-2}"
+  fi
+
+  echo "$value"
+}
+
+function ViashParseMultipleStringAsArray() {
+  local value="$1"
+
+  # if value is equal to UNDEFINED, replace with @@VIASH_UNDEFINED@@
+  if [ "$value" == "UNDEFINED" ]; then
+    echo "@@VIASH_UNDEFINED@@"
+    return
+  fi
+
+  # if value is empty, return nothing (results in 0-length array)
+  if [ -z "$value" ]; then
+    return
+  fi
+
+  # Parse semicolon-separated values with proper quote handling
+  # This is a bash-native implementation that doesn't rely on gawk's FPAT
+  # (which isn't available in BSD awk on macOS or BusyBox awk)
+  local i=0
+  local len=${#value}
+  local in_double_quote=false
+  local in_single_quote=false
+  local escape_next=false
+  local current_field=""
+  local field_was_quoted=false  # Track if field started with a quote
+  local char
+
+  while [ $i -lt $len ]; do
+    char="${value:$i:1}"
+
+    if $escape_next; then
+      # Previous char was backslash - add escaped char to field
+      current_field+="$char"
+      escape_next=false
+    elif [ "$char" = "\\" ]; then
+      # Backslash - escape next character
+      escape_next=true
+    elif [ "$char" = '"' ] && ! $in_single_quote; then
+      # Toggle double quote state (unless we're in single quotes)
+      if $in_double_quote; then
+        in_double_quote=false
+      else
+        in_double_quote=true
+        # Mark field as quoted if this is the first char
+        if [ -z "$current_field" ]; then
+          field_was_quoted=true
+        fi
+      fi
+    elif [ "$char" = "'" ] && ! $in_double_quote; then
+      # Toggle single quote state (unless we're in double quotes)
+      if $in_single_quote; then
+        in_single_quote=false
+      else
+        in_single_quote=true
+        # Mark field as quoted if this is the first char
+        if [ -z "$current_field" ]; then
+          field_was_quoted=true
+        fi
+      fi
+    elif [ "$char" = ";" ] && ! $in_double_quote && ! $in_single_quote; then
+      # Semicolon outside quotes - end of field
+      # Handle UNDEFINED_ITEM replacement (only if not quoted)
+      if [ "$current_field" == "UNDEFINED_ITEM" ] && ! $field_was_quoted; then
+        current_field="@@VIASH_UNDEFINED_ITEM@@"
+      fi
+      echo "$current_field"
+      current_field=""
+      field_was_quoted=false
+    else
+      # Regular character - add to field
+      current_field+="$char"
+    fi
+
+    i=$((i + 1))
+  done
+
+  # Don't forget the last field (after the last semicolon or if no semicolon)
+  # Handle UNDEFINED_ITEM replacement (only if not quoted)
+  if [ "$current_field" == "UNDEFINED_ITEM" ] && ! $field_was_quoted; then
+    current_field="@@VIASH_UNDEFINED_ITEM@@"
+  fi
+  echo "$current_field"
+}