Binary codec should reject excessively nested STObject/STArray input before recursive parsing exhausts the host stack

[RaymondSeven]

The binary codec currently parses nested "STObject" and "STArray" values recursively without carrying a nesting-depth counter. A small, validly framed XRPL binary payload can therefore trigger very deep recursive deserialization.

This affects public decode paths that parse attacker-controlled or otherwise untrusted XRPL binary blobs, such as transaction blobs, ledger entries, or metadata returned by third-party endpoints.

The reference implementation already guards this case by rejecting nesting depth greater than 10. The SDK codecs should mirror that behavior and fail with a normal codec error before recursion reaches the language runtime stack limit.

Affected components

Confirmed against:

• "ripple-binary-codec" "2.7.0"
• "xrpl" "4.6.0"
• "xrpl-py" "4.5.0"
• "xrpl4j-core" "6.1.0-rc.1"

The same parser pattern appears in the "STObject" and "STArray" binary decode paths.

Root cause

The binary parser alternates recursively between object and array parsing:

1. The top-level decode reads an "STObject".
2. "STObject" reads each field and dispatches to "readFieldValue".
3. If the field is an "STArray", the parser enters "STArray".
4. Each array member is an "STObject", so parsing returns to "STObject".
5. This continues without a depth argument, maximum nesting constant, or recursion counter.

As a result, input nesting is limited only by the host language stack or runtime recursion limit.

Minimal payload shape

The payload uses defined XRPL field headers:

F9 = Memos, STArray field
EA = Memo, STObject array member
E1 = ObjectEndMarker
F1 = ArrayEndMarker

One level:

F9 EA E1 F1

N nested levels:

("F9EA" * N) + ("E1F1" * N)

At "N = 1", this decodes as:

{
  "Memos": [
    {
      "Memo": {}
    }
  ]
}

At larger depths, each level adds only 4 bytes but increases recursive parser depth.

Observed behavior

With a 300-level payload, which is only 1200 bytes:

• JavaScript can throw "RangeError: Maximum call stack size exceeded".
• Python can throw "RecursionError: maximum recursion depth exceeded".
• Java can terminate the parser path with "StackOverflowError".

In Python, the error is not raised as an "XRPLBinaryCodecException". In Java, "StackOverflowError" is a "java.lang.Error", not an "Exception", so it bypasses wrappers or handlers that only catch "Exception".

The JavaScript path can also become very slow before reaching stack exhaustion. In local testing with default Node stack settings, deeply nested input showed superlinear CPU cost because nested array bytes are repeatedly reprocessed during conversion.

Example timing from local testing:

100 levels   /  400 bytes   ->  ~101 ms
200 levels   /  800 bytes   ->  ~405 ms
400 levels   / 1600 bytes   ->  ~1951 ms
800 levels   / 3200 bytes   ->  ~11342 ms

Expected behavior

The codec should reject excessive nesting with a normal binary codec parse error before recursion can exhaust the host stack.

This should match the reference implementation behavior, which rejects nested "STObject" and "STArray" structures once the nesting depth exceeds 10.

Security impact

This is an application-level denial-of-service risk for software that decodes XRPL binary from untrusted sources.

Examples include:

• Wallets decoding user-supplied transaction blobs
• Explorers or indexers decoding binary data from third-party XRPL endpoints
• Custodians or exchanges decoding submitted transaction blobs before validation or compliance checks
• API gateways converting XRPL binary to JSON for downstream services

The input does not need to be a valid ledger transaction. The issue is in raw binary decoding before any on-chain acceptance rules apply.

Suggested fix

Mirror the reference implementation behavior:

• Add a maximum nesting constant, preferably "10" for parity with the reference implementation.
• Thread a "depth" argument through "STObject" and "STArray" parser entry points.
• Increment depth before parsing nested object or array values.
• Reject "depth > 10" using the SDK's normal binary codec exception type.
• Apply the same guard to JSON conversion paths that reparse stored array or object bytes.

Example shape:

static fromParser(parser: BinaryParser, depth = 0): STObject {
  if (depth > MAX_NESTING_DEPTH) {
    throw new BinaryCodecError("Maximum nesting depth exceeded")
  }

  // parse fields, passing depth + 1 into nested STObject/STArray parsing
}

The exact exception type should match each SDK's existing codec error model.

Why byte-bound checks are not enough

The payload is syntactically framed and small. Per-read bounds checks only prove that the next bytes exist. They do not limit how deeply the parser can recurse.

A separate nesting-depth guard is needed to bound parser recursion.