Skip to content

SFN parser: support AWS JSONPath filter dialect (&&/|| and empty filter results) - #10095

Open
koteshyelamati wants to merge 3 commits into
getmoto:masterfrom
koteshyelamati:fix-sfn-jsonpath-aws-dialect
Open

SFN parser: support AWS JSONPath filter dialect (&&/|| and empty filter results)#10095
koteshyelamati wants to merge 3 commits into
getmoto:masterfrom
koteshyelamati:fix-sfn-jsonpath-aws-dialect

Conversation

@koteshyelamati

Copy link
Copy Markdown
Contributor

Fixes #10078

The Step Functions parser backend evaluates ASL JSONPath through jsonpath_ng.ext, which does not implement two features of AWS Step Functions' JSONPath dialect. Both appear in real-world (CDK-generated) state machines and surface as States.Runtime errors:

  1. && / || filter conjunctions. AWS SFN filter expressions use &&/|| (e.g. $.items[?(@.a == true && @.b == true)]); jsonpath_ng only accepts single & and raises JsonPathParserError on the doubled form, and its filter grammar has no disjunction at all.
  2. Empty filter results raise instead of yielding []. On AWS, a filter that matches nothing produces []; extract_json raised NoSuchJsonPathError (it returned [] only for slice/wildcard paths).

Rather than swapping the JSONPath engine (a larger dependency decision), this implements the dialect on top of jsonpath_ng inside extract_json:

  • && is translated to jsonpath_ng's &, skipping string literals.
  • || is handled by expanding the filter into one path per disjunct (respecting AWS's precedence: && binds tighter than ||), evaluating each, and merging matches back into document order with duplicates removed. Nothing changes for paths without ||.
  • Filters that match nothing now return [], consistent with the existing slice/wildcard behaviour.

Operators inside string literals (e.g. [?(@.v == 'x && y')]) are preserved by quote-aware scanning in both transformations.

Added unit tests covering conjunctions, disjunctions (including precedence, chaining and document order), quoted literals, empty filter results, and the pre-existing behaviours (definite paths still raise NoSuchJsonPathError when missing, wildcard/index access unchanged). All existing parser test suites pass.

Together with #10093 and #10094 this completes the set of three issues (#10076, #10077, #10078) blocking the states:startExecution.sync path for CDK-generated state machines.

@codecov

codecov Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.87234% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.29%. Comparing base (9473300) to head (c495466).

Files with missing lines Patch % Lines
moto/stepfunctions/parser/asl/utils/json_path.py 97.87% 2 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master   #10095   +/-   ##
=======================================
  Coverage   93.28%   93.29%           
=======================================
  Files        1336     1336           
  Lines      121679   121770   +91     
=======================================
+ Hits       113508   113604   +96     
+ Misses       8171     8166    -5     
Flag Coverage Δ
servertests 27.98% <97.87%> (+0.08%) ⬆️
unittests 93.27% <97.87%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tinovyatkin

Copy link
Copy Markdown

I would not recommend to merge this as is - patching just these two cases without extending testing corpus is future error prone. I believe swapping whole jsonpath engine is more future proof solution here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StepFunctions parser: jsonpath_ng backend doesn't implement AWS JSONPath filter dialect (&&/|| parse error; empty filter raises instead of [])

3 participants