StepFunctions parser: jsonpath_ng backend doesn't implement AWS JSONPath filter dialect (&&/|| parse error; empty filter raises instead of [])

[tinovyatkin]

Summary

The Step Functions parser backend evaluates ASL JSONPath through jsonpath_ng.ext, which does not implement AWS Step Functions' JSONPath dialect. Two gaps make the classic service integrations unusable for real-world (CDK-generated) state machines:

1. && / || filter conjunctions. AWS SFN filter expressions use && / || (e.g. $.items[?(@.a == true && @.b == true)]). jsonpath_ng only accepts a single & / | and raises JsonPathParserError: Parse error ... near token & on the doubled form. This shape is emitted directly by aws-cdk-lib's StateMachine (e.g. inside States.JsonToString(...)), so it appears in real deployed ASL.

2. Empty filter result raises instead of []. On AWS, a [?()] filter that matches nothing yields []. moto's extract_json raises NoSuchJsonPathError for it (it returns [] only for slice/wildcard paths, not filters), surfacing as States.Runtime.

Both block the arn:aws:states:::states:startExecution.sync path for any non-trivial scenario, because the always-present input-preparation states filter file lists with exactly these constructs.

Reproduce

&& parse failure (end-to-end):

import json, time, os
import boto3
from moto.server import ThreadedMotoServer
from moto.core.config import default_user_config

default_user_config["stepfunctions"]["execute_state_machine"] = True
os.environ["MOTO_PORT"] = "5000"
srv = ThreadedMotoServer(port=5000); srv.start()
sfn = boto3.client("stepfunctions", endpoint_url="http://localhost:5000",
                   region_name="us-east-1", aws_access_key_id="x", aws_secret_access_key="x")
definition = {
    "StartAt": "Filter",
    "States": {"Filter": {
        "Type": "Pass",
        "Parameters": {"selected.$": "$.items[?(@.keep == true && @.ready == true)]"},
        "End": True,
    }},
}
arn = sfn.create_state_machine(name="f", roleArn="arn:aws:iam::123456789012:role/sfn",
                               definition=json.dumps(definition))["stateMachineArn"]
exe = sfn.start_execution(stateMachineArn=arn, name="r1",
                          input=json.dumps({"items": [{"keep": True, "ready": True}]}))["executionArn"]
for _ in range(40):
    d = sfn.describe_execution(executionArn=exe)
    if d["status"] != "RUNNING": break
    time.sleep(0.25)
print(d["status"], d.get("cause"))   # FAILED  JsonPathParserError(Parse error at 1:26 near token & (&))
srv.stop()

Empty-filter facet (isolated; single & so it parses, matches nothing):

from moto.stepfunctions.parser.asl.utils.json_path import extract_json
extract_json("$.items[?(@.k == 'NONE')]", {"items": [{"k": "a"}]})
# -> raises NoSuchJsonPathError; AWS Step Functions yields []

Suggested fix

Rather than special-case each divergence, consider swapping the JSONPath engine for one that implements the AWS dialect. python-jsonpath handles both cases out of the box:

import jsonpath  # python-jsonpath
data = {"items": [{"keep": True, "ready": True}, {"keep": True, "ready": False}]}
jsonpath.findall("$.items[?(@.keep == true && @.ready == true)]", data)  # -> [{'keep': True, 'ready': True}]
jsonpath.findall("$.items[?(@.k == 'NONE')]", data)                      # -> []  (no error)

extract_json's current AWS-specific post-processing (singleton-array unpack, the #7825 context-Index special case, scalar-vs-list) maps cleanly onto python-jsonpath's JSONPathEnvironment.compile(path).singular_query(): a singular query returns the single value (or "not found" for a missing definite path), a non-singular query returns the list of matches. That replaces the isinstance(match.path, Index) heuristics and _is_singleton_array_access regex with the library's own notion of a singular path.

(For now we monkeypatch json_path.extract_json onto python-jsonpath in our offline test harness; we'd happily drop it once moto evaluates AWS JSONPath natively.)

Environment

• moto 5.1.22 (parser backend, execute_state_machine=True)
• jsonpath-ng 1.8.0 (latest; does not implement AWS &&/||)
• Python 3.12