Skip to content

[bugfix] BinaryValueFromFile: honor shared-reference reference counting#6506

Open
joewiz wants to merge 1 commit into
eXist-db:developfrom
joewiz:bugfix/multipart-upload-store
Open

[bugfix] BinaryValueFromFile: honor shared-reference reference counting#6506
joewiz wants to merge 1 commit into
eXist-db:developfrom
joewiz:bugfix/multipart-upload-store

Conversation

@joewiz

@joewiz joewiz commented Jun 20, 2026

Copy link
Copy Markdown
Member

[This PR was prompted by Joe, drafted by Claude Code, and reviewed by Joe.]

Summary

A file-backed binary value (BinaryValueFromFile, e.g. from file:read-binary or request:get-uploaded-file-data) that is used in an element or document constructor (an enclosed expression) and then read again could have its file channel closed underneath it, after which the second read failed with "Underlying channel has been closed":

let $b := file:read-binary($path)
let $w := <a>{$b}</a>
return (count($w), $b)[2]   (: count($w) forces the constructor; -> "Underlying channel has been closed" without the fix :)

Root cause

BinaryValueFromFile did not honor the shared-reference reference counting that XQueryContext.enterEnclosedExpr() / exitEnclosedExpr() rely on:

  • incrementSharedReferences() was a no-op ("we don't need reference counting…"), and
  • close() released the FileChannel/RandomAccessFile unconditionally.

enterEnclosedExpr() increments the shared-reference count of each live binary value so a value that escapes an enclosed expression survives; the matching exitEnclosedExpr() then close()s each, intending only to decrement a reference. For BinaryValueFromFile the increment did nothing and the close() actually tore down the channel — so a value used in a constructor was closed immediately after the constructor, while it was still referenced. BinaryValueFromInputStream was unaffected because it already reference-counts through AbstractFilterInputStreamCache.

Confirmed by instrumentation: the close fires from XQueryContext.exitEnclosedExpr() right after the constructor reads the value, and the failing read is the subsequent use of the value.

Fix

Give BinaryValueFromFile the same reference counting as its input-stream sibling: a counter starting at 1, incremented by incrementSharedReferences() and decremented by close(), releasing the channel/file handle only when it reaches zero. Eager cleanup of values not shared out of a scope is preserved (their count goes 1 → 0 on cleanup).

What changed

  • exist-core/.../xquery/value/BinaryValueFromFile.java — add the sharedReferences counter; incrementSharedReferences() increments it; close() decrements and releases the channel only at zero (idempotent once released).

Test plan

  • BinaryValueFromFileSharedReferenceTest (unit): models enterEnclosedExpr()/exitEnclosedExpr() on a shared file-backed value; it must stay open and readable, then be released by the final close. Fails before the fix, passes after.
  • AbstractBinariesTest#readBinaryUsedInElementConstructorThenReadAgain (file module): the summary query, exercised over both the embedded (EmbeddedBinariesTest) and REST (RestBinariesTest) execution paths; both fail with "Underlying channel has been closed" before the fix and pass after.
  • Codacy/PMD clean on the changed files.

The regression test runs a main-module query (not an XQSuite test function) on purpose: inside a module function the bug is masked, because ModuleContext.registerBinaryValueInstance() delegates registration to the parent/root context, while enterEnclosedExpr()/exitEnclosedExpr() operate on the ModuleContext's own (empty) deque — so the constructor's exitEnclosedExpr() never sees the binary there and the close happens later via popLocalVariables (after the read), harmlessly. The bug surfaces only where the binary and the enclosed expression share a context, i.e., in a main-module query. (That asymmetry in ModuleContext is pre-existing and merely masks this bug; this PR does not change it.)

Notes

  • This PR was originally described as fixing a multipart-upload bug. Upon further research, that claim was misplaced. The multipart xmldb:store failure actually has a distinct root cause (Sequence.containsReference not recursing into nested map/array items, across five sequence types), now fixed separately in [bugfix] Sequence.containsReference: recurse into nested map/array items #6507. This PR fixes the enclosed-expression premature-close path – a sibling "file-backed binary closed while still referenced" bug.

@joewiz joewiz requested a review from a team as a code owner June 20, 2026 15:07
@joewiz joewiz marked this pull request as draft June 20, 2026 19:12
@joewiz joewiz force-pushed the bugfix/multipart-upload-store branch 2 times, most recently from 187d96c to 289b228 Compare June 21, 2026 03:52
BinaryValueFromFile did not honor the shared-reference reference counting
that XQueryContext.enterEnclosedExpr()/exitEnclosedExpr() rely on:
incrementSharedReferences() was a no-op and close() released the FileChannel
unconditionally. So when a file-backed binary value was used in an element
(or document) constructor -- an enclosed expression -- exitEnclosedExpr()
closed the value's channel immediately after the constructor, and any later
read of the value failed with "Underlying channel has been closed". For
example (count($w) forces the constructor to be evaluated before $b is read
again):

  let $b := file:read-binary($path)
  let $w := <a>{$b}</a>
  return (count($w), $b)[2]

BinaryValueFromInputStream was unaffected because it already reference-counts
through AbstractFilterInputStreamCache (close() releases only when the shared
reference count reaches zero).

Root cause confirmed by instrumentation: the close fires from
XQueryContext.exitEnclosedExpr() right after the constructor reads the value;
the failing read is the subsequent use of the value. It reproduces on both
the embedded and REST execution paths -- only the consumer of the second read
differs (the caller vs the response serializer).

Fix: give BinaryValueFromFile the same reference counting -- a counter
starting at 1, incremented by incrementSharedReferences() and decremented by
close(), releasing the channel/file handle only when it reaches zero. Eager
cleanup of values not shared out of a scope is preserved.

Tests:
- BinaryValueFromFileSharedReferenceTest (unit): models
  enterEnclosedExpr()/exitEnclosedExpr() on a shared file-backed value; it
  must stay open and readable, then be released by the final close.
- AbstractBinariesTest#readBinaryUsedInElementConstructorThenReadAgain
  (file module): the query above, run over BOTH the embedded
  (EmbeddedBinariesTest) and REST (RestBinariesTest) execution paths; both
  fail with "Underlying channel has been closed" before the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@joewiz joewiz force-pushed the bugfix/multipart-upload-store branch from 289b228 to 33d7cde Compare June 21, 2026 04:15
@joewiz joewiz changed the title [bugfix] Fix file-backed binary value closed across enclosed expressions [bugfix] BinaryValueFromFile: honor shared-reference reference counting Jun 21, 2026
@joewiz joewiz marked this pull request as ready for review June 21, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant