Skip to content

feat: add support for arrays with sequences as members#1416

Open
EPMatt wants to merge 28 commits intomasterfrom
ma/xquery-array-sequences-as-members
Open

feat: add support for arrays with sequences as members#1416
EPMatt wants to merge 28 commits intomasterfrom
ma/xquery-array-sequences-as-members

Conversation

@EPMatt
Copy link
Collaborator

@EPMatt EPMatt commented Mar 16, 2026

Overview

This branch targets XQuery 3.1 / F&O 3.1 array semantics in Rumble: each array member is a logical sequence of items (length 1 for typical cells, or longer when the source expression produces multiple items for that member). It adds SequenceArrayItem, extends the Item API for per-member sequences, wires construction and navigation through the compiler and runtime, registers a large array: function surface in BuiltinFunctionCatalogue, and implements those functions as dedicated iterators. It also adds v0 XQuery support for array tests and item tests in the translation pipeline, plus ArrayFunctionCallIterator so function items can be invoked in array contexts with appropriate typing checks.

On Item, array values expose member sequences via getSequences() (all members) and getSequenceAt(int) (zero-based, parallel to getItemAt). allowsNonSingletons() distinguishes representations that may store multi-item members from singleton-only arrays, so hot paths can avoid unnecessary sequence handling. ArrayRuntimeIterator builds arrays from member sequences (e.g. createArrayFromMemberSequences on ItemFactory), and array lookup, unboxing, members, and descendant code paths are updated so sequence members behave consistently end-to-end. array:sort is implemented with SortKeyComparison so ordering aligns with the engine’s comparison/sort-key rules for the supported sort signatures.

Changelog

1. Array members as sequences (data model and API)

  • Add SequenceArrayItem and implement sequence-aware behavior alongside ArrayItem.
  • On Item: getSequences(), getSequenceAt(int), and allowsNonSingletons() for array items; refactor of older “member sequence” accessors toward this getSequence* naming (see commit f76efc7b9).
  • ItemFactory: support for creating arrays from lists of member sequences and related factory paths used by runtime construction.

2. Wiring construction, types, and visitors

  • ArrayConstructorExpression and runtime array construction updated so each member can materialize to a sequence, not only a single item.
  • Compiler passes are adjusted where needed—among others XQueryTranslationVisitor, InferTypeVisitor, RuntimeIteratorVisitor, CloneVisitor, ComparisonVisitor, and shared visitors such as AbstractNodeVisitor—so array typing, cloning, and execution stay consistent with sequence members.
  • ArrayItemType (and related typing) updated so static knowledge of arrays can reflect non-singleton members where the type system models that.

3. Runtime navigation and closures

  • Array lookup, unboxing, members, and descendant iterators and closures updated to use member sequences (getSequences / per-member lists) instead of assuming one item per slot everywhere.

4. array:* functions (F&O-oriented iterators)

Registered and implemented (exact arity/overloads per BuiltinFunctionCatalogue and iterator classes under runtime/functions/arrays/), including:

  • Structural / access: array:size, array:get, array:head, array:tail, array:subarray, array:insert-before, array:remove, array:reverse, array:append, array:join, array:flatten.
  • Higher-order (some marked v0 in history): array:fold-left, array:fold-right, array:for-each, array:for-each-pair, array:filter, array:sort (including variants with collation / key function where implemented).

5. Dynamic calls on arrays

  • ArrayFunctionCallIterator: invoke function items in array contexts with the extra type checking and runtime behavior described in commit f5ca3360b.

6. XQuery surface

  • v0 array tests (commit 7e7b1ea87) and item tests (commit aa3a19305) in the XQuery translation stack.

@EPMatt EPMatt changed the base branch from ma/xml-atomic-type-system to master March 16, 2026 22:18
@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 192 362 182 135 871
SerTest 37 246 60 0 343
Prod1Test 5591 728 1470 449 8238
Fn1Test 2716 632 1585 362 5295
Fn2Test 3204 941 1239 456 5840
AppTest 1062 62 1000 33 2157
Prod2Test 1858 523 1094 494 3969
ArrayTest 0 45 155 9 209
XsTest 99 18 4 43 164
OpTest 4044 126 155 41 4366
MapTest 4 23 193 0 220
Total 18954 3706 7139 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 0 0 0 209 209
XsTest 99 4 0 61 164
Fn1Test 2431 179 143 2542 5295
SerTest 6 1 0 336 343
Prod1Test 4502 252 394 3090 8238
Fn2Test 2686 299 117 2738 5840
MapTest 3 1 16 200 220
AppTest 1011 22 18 1106 2157
Prod2Test 1348 229 147 2245 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16163 1304 998 13356 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 192 362 182 135 871
SerTest 37 246 60 0 343
Prod1Test 5597 728 1464 449 8238
Fn1Test 2716 632 1585 362 5295
Fn2Test 3204 941 1239 456 5840
AppTest 1067 62 995 33 2157
Prod2Test 1858 523 1094 494 3969
ArrayTest 0 45 155 9 209
XsTest 99 18 4 43 164
OpTest 4044 126 155 41 4366
MapTest 4 23 193 0 220
Total 18965 3706 7128 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 0 0 0 209 209
XsTest 99 4 0 61 164
Fn1Test 2431 179 143 2542 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 389 3090 8238
Fn2Test 2686 299 117 2738 5840
MapTest 3 1 16 200 220
AppTest 1011 22 18 1106 2157
Prod2Test 1348 229 147 2245 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16168 1304 993 13356 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 192 362 182 135 871
SerTest 37 246 60 0 343
Prod1Test 5617 734 1438 449 8238
Fn1Test 2716 632 1585 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1077 62 985 33 2157
Prod2Test 1860 523 1092 494 3969
ArrayTest 7 45 148 9 209
XsTest 99 18 4 43 164
OpTest 4044 126 155 41 4366
MapTest 4 23 193 0 220
Total 19012 3715 7072 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 6 1 0 202 209
XsTest 99 4 0 61 164
Fn1Test 2431 179 143 2542 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 389 3090 8238
Fn2Test 2687 299 117 2737 5840
MapTest 3 1 16 200 220
AppTest 1019 22 19 1097 2157
Prod2Test 1348 231 147 2243 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16183 1307 994 13337 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 192 362 182 135 871
SerTest 37 246 60 0 343
Prod1Test 5617 734 1438 449 8238
Fn1Test 2716 632 1585 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1085 60 979 33 2157
Prod2Test 1860 523 1092 494 3969
ArrayTest 23 36 141 9 209
XsTest 99 18 4 43 164
OpTest 4044 126 155 41 4366
MapTest 4 23 193 0 220
Total 19036 3704 7059 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 20 1 1 187 209
XsTest 99 4 0 61 164
Fn1Test 2431 179 143 2542 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 389 3090 8238
Fn2Test 2687 299 117 2737 5840
MapTest 3 1 16 200 220
AppTest 1027 22 19 1089 2157
Prod2Test 1348 231 147 2243 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16205 1307 995 13314 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 196 367 173 135 871
SerTest 37 246 60 0 343
Prod1Test 5637 725 1427 449 8238
Fn1Test 2717 632 1584 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1128 66 930 33 2157
Prod2Test 1882 514 1079 494 3969
ArrayTest 32 34 134 9 209
XsTest 99 18 4 43 164
OpTest 4044 126 155 41 4366
MapTest 4 23 193 0 220
Total 19135 3695 6969 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 23 3 1 182 209
XsTest 99 4 0 61 164
Fn1Test 2431 179 143 2542 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 389 3090 8238
Fn2Test 2687 299 117 2737 5840
MapTest 3 1 16 200 220
AppTest 1032 23 19 1083 2157
Prod2Test 1348 231 147 2243 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16213 1310 995 13303 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 196 367 173 135 871
SerTest 37 246 60 0 343
Prod1Test 5637 725 1427 449 8238
Fn1Test 2719 630 1584 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1157 61 906 33 2157
Prod2Test 1882 514 1079 494 3969
ArrayTest 66 24 110 9 209
XsTest 99 18 4 43 164
OpTest 4043 127 155 41 4366
MapTest 4 23 193 0 220
Total 19199 3679 6921 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 52 8 7 142 209
XsTest 99 4 0 61 164
Fn1Test 2433 179 154 2529 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 389 3090 8238
Fn2Test 2687 299 117 2737 5840
MapTest 3 1 16 200 220
AppTest 1058 26 23 1050 2157
Prod2Test 1348 231 147 2243 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16270 1318 1016 13217 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 196 367 173 135 871
SerTest 37 246 60 0 343
Prod1Test 5637 725 1427 449 8238
Fn1Test 2725 624 1584 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1185 60 879 33 2157
Prod2Test 1882 514 1079 494 3969
ArrayTest 107 17 76 9 209
XsTest 99 18 4 43 164
OpTest 4044 126 155 41 4366
MapTest 4 23 193 0 220
Total 19275 3664 6860 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 89 12 15 93 209
XsTest 99 4 0 61 164
Fn1Test 2439 181 177 2498 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 389 3090 8238
Fn2Test 2687 299 117 2737 5840
MapTest 3 1 16 200 220
AppTest 1084 28 26 1019 2157
Prod2Test 1348 231 147 2243 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16339 1326 1050 13106 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 196 367 173 135 871
SerTest 37 246 60 0 343
Prod1Test 5637 725 1427 449 8238
Fn1Test 2728 622 1583 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1198 60 866 33 2157
Prod2Test 1882 514 1079 494 3969
ArrayTest 142 13 45 9 209
XsTest 99 18 4 43 164
OpTest 4043 127 155 41 4366
MapTest 4 23 193 0 220
Total 19325 3659 6815 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 114 20 24 51 209
XsTest 99 4 0 61 164
Fn1Test 2441 183 182 2489 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 392 3087 8238
Fn2Test 2687 300 117 2736 5840
MapTest 3 1 16 200 220
AppTest 1095 30 35 997 2157
Prod2Test 1348 231 147 2243 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16377 1339 1076 13029 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 196 367 173 135 871
SerTest 37 246 60 0 343
Prod1Test 5637 725 1427 449 8238
Fn1Test 2730 620 1583 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1201 60 863 33 2157
Prod2Test 1882 514 1079 494 3969
ArrayTest 145 13 42 9 209
XsTest 99 18 4 43 164
OpTest 4044 126 155 41 4366
MapTest 4 23 193 0 220
Total 19334 3656 6809 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 117 22 28 42 209
XsTest 99 4 0 61 164
Fn1Test 2443 185 190 2477 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 392 3087 8238
Fn2Test 2687 300 117 2736 5840
MapTest 3 1 16 200 220
AppTest 1096 31 39 991 2157
Prod2Test 1348 231 148 2242 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16383 1344 1093 13001 31821

Download detailed test results

@github-actions
Copy link

Test Results (qt3tests)

RumbleDB, XQuery parser
Test Suite Passing Failing Errors Skipped Total
MathTest 147 0 2 0 149
MiscTest 196 367 173 135 871
SerTest 37 246 60 0 343
Prod1Test 5637 725 1427 449 8238
Fn1Test 2730 620 1583 362 5295
Fn2Test 3212 944 1228 456 5840
AppTest 1201 60 863 33 2157
Prod2Test 1882 514 1079 494 3969
ArrayTest 145 13 42 9 209
XsTest 99 18 4 43 164
OpTest 4042 128 155 41 4366
MapTest 4 23 193 0 220
Total 19332 3658 6809 2022 31821
RumbleDB, JSONiq parser
Test Suite Passing Failing Errors Skipped Total
MiscTest 164 286 115 306 871
ArrayTest 117 22 28 42 209
XsTest 99 4 0 61 164
Fn1Test 2443 185 190 2477 5295
SerTest 6 1 0 336 343
Prod1Test 4507 252 392 3087 8238
Fn2Test 2687 300 117 2736 5840
MapTest 3 1 16 200 220
AppTest 1096 31 39 991 2157
Prod2Test 1348 231 148 2242 3969
OpTest 3766 31 47 522 4366
MathTest 147 0 1 1 149
Total 16383 1344 1093 13001 31821

Download detailed test results

@EPMatt EPMatt changed the base branch from master to ma/xml-atomic-type-system March 21, 2026 18:26
@EPMatt EPMatt force-pushed the ma/xquery-array-sequences-as-members branch from b3ff173 to 54c6314 Compare March 21, 2026 18:30
Copy link
Collaborator Author

@EPMatt EPMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ready for review. For consistency, please rebase on main and merge only after #1409 is merged.

@EPMatt EPMatt marked this pull request as ready for review March 22, 2026 08:47
@EPMatt EPMatt requested a review from ghislainfourny as a code owner March 22, 2026 08:47
Base automatically changed from ma/xml-atomic-type-system to master March 25, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant