Skip to content

ART Index: Support compound key scans#7

Merged
mach-kernel merged 6 commits into
spiceai-1.4.1from
mach/compound-art-scan
Oct 30, 2025
Merged

ART Index: Support compound key scans#7
mach-kernel merged 6 commits into
spiceai-1.4.1from
mach/compound-art-scan

Conversation

@mach-kernel

@mach-kernel mach-kernel commented Oct 29, 2025

Copy link
Copy Markdown

🗣 Description

  • The ART index already supports compound keys in order to enforce uniqueness constraints.
  • ARTKey has a concatenation mechanism that it uses to generate one value key for multiple column exprs.
  • This wires it up for the query side, with
    • ARTIndexCompoundKeyScanState, which is the same as ARTIndexScanState but with vectors for an arbitrary number of values
    • ART::CompoundKeyScan, which generates the concatenated key, locks, does an equality scan
D explain analyze select * from test where x = '525' and y = '2525';
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││    Query Profiling Information    ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
explain analyze select * from test where x = '525' and y = '2525';
┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││              Total Time: 0.0251s             ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘
┌───────────────────────────┐
│           QUERY           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│      EXPLAIN_ANALYZE      │
│    ────────────────────   │
│           0 rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         TABLE_SCAN        │
│    ────────────────────   │
│        Table: test        │
│      Type: Index Scan     │
│                           │
│        Projections:       │
│             x             │
│             y             │
│             z             │
│                           │
│          Filters:         │
│          x='525'          │
│          y='2525'         │
│                           │
│           1 row           │
│          (0.00s)          │
└───────────────────────────┘

For completeness (break before passing back to TableScanInitGlobal
image

rebind projected columns in ALL index exprs
do not bail out early if more than one index expr
hook up composite key scan
…s we made earlier), fix single-ref-per-expr predicate to correctly walk expr tree and yank refs (allowing nesting in fns, etc)
@mach-kernel

Copy link
Copy Markdown
Author

All "non slow" ART index tests pass:

Filters: test/sql/index/art*
[0/92] (0%): test/sql/index/art/types/test_art_varchar.test     [1/92] (1%): test/sql/index/art/types/test_art_integer_types.tes[2/92] (2%): test/sql/index/art/types/test_art_union.test       [3/92] (3%): test/sql/index/art/types/test_art_real.test        [4/92] (4%): test/sql/index/art/types/test_art_expression_key.te[5/92] (5%): test/sql/index/art/types/test_art_coverage_types.te[6/92] (6%): test/sql/index/art/types/test_art_boolean.test     [7/92] (7%): test/sql/index/art/types/test_art_real_pk.test     [8/92] (8%): test/sql/index/art/types/test_art_double.test      [9/92] (9%): test/sql/index/art/constraints/test_art_upsert_dupl[10/92] (10%): test/sql/index/art/constraints/test_art_tx_upsert[11/92] (11%): test/sql/index/art/constraints/test_art_tx_update[12/92] (13%): test/sql/index/art/constraints/test_art_compound_[13/92] (14%): test/sql/index/art/constraints/test_art_tx_delete[14/92] (15%): test/sql/index/art/constraints/test_art_tx_delete[15/92] (16%): test/sql/index/art/constraints/test_art_simple_up[16/92] (17%): test/sql/index/art/constraints/test_art_upsert_ot[17/92] (18%): test/sql/index/art/constraints/test_art_tx_upsert[18/92] (19%): test/sql/index/art/constraints/test_art_tx_update[19/92] (20%): test/sql/index/art/constraints/test_art_tx_delete[20/92] (21%): test/sql/index/art/constraints/test_art_tx_update[21/92] (22%): test/sql/index/art/constraints/test_art_large_abo[92/92] (100%): test/sql/index/art/create_drop/test_art_invalid_create_index....
===============================================================================
All tests passed (1 skipped test, 2211 assertions in 91 test cases)

We can test in-situ now

@mach-kernel

Copy link
Copy Markdown
Author

Full suite:

-------------------------------------------------------------------------------
test/sql/storage/shutdown_create_index.test
-------------------------------------------------------------------------------
/Users/mach/workspace/spice/duckdb/test/sqlite/test_sqllogictest.cpp:247
...............................................................................

test/sql/storage/shutdown_create_index.test:45: FAILED:
explicitly with message:
  0

[3971/3971] (100%): test/sql/table_function/icu_range_timestamptz.test
===============================================================================
test cases:    3971 |    3970 passed | 1 failed | 283 skipped
assertions: 1035329 | 1035328 passed | 1 failed | 283 skipped

@mach-kernel mach-kernel changed the title ART Index: Support compound scans ART Index: Support compound key scans Oct 30, 2025
@mach-kernel

Copy link
Copy Markdown
Author

All set! Merging.

===============================================================================
test cases:   3972 |   3972 passed | 0 failed | 283 skipped
assertions: 943239 | 943239 passed | 0 failed | 283 skipped

Skipped tests for the following reasons:
require autocomplete: 54
require block_size: 2
require exact_vector_size: 2
require httpfs: 2
require icu: 49
require inet: 1
require json: 116
require longdouble: 2
require spatial: 7
require tpcds: 6
require tpch: 30
require windows: 3
require-env LOCAL_EXTENSION_REPO: 7
require-env RUN_EXTENSION_UPDATE_TEST: 1
require-env TEST_PERSISTENT_SECRETS_AVAILABLE: 1

@mach-kernel mach-kernel merged commit ef35b00 into spiceai-1.4.1 Oct 30, 2025
49 of 53 checks passed
mach-kernel added a commit that referenced this pull request Oct 30, 2025
ART Index: Support compound key scans

Squashed commit of the following:

commit a22a430
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 16:37:30 2025 -0400

    simplify filter expression storage index bindings (just reuse the ones we made earlier), fix single-ref-per-expr predicate to correctly walk expr tree and yank refs (allowing nesting in fns, etc)

commit 9c8c1ed
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 15:11:23 2025 -0400

    copy index expressions before rewriting column refs

commit aff2c98
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 14:36:33 2025 -0400

    table scan:

    rebind projected columns in ALL index exprs
    do not bail out early if more than one index expr
    hook up composite key scan

commit bfc6f02
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 14:35:09 2025 -0400

    make specialized compound key scan state for eq compares, specialized scan using ARTKey::Concat
mach-kernel added a commit that referenced this pull request Nov 3, 2025
Squashed commit of the following:

commit 36ffa5b
Author: David Stancu <david@spice.ai>
Date:   Mon Nov 3 12:30:28 2025 -0500

    tryscanindex sanity check: indexed_columns / art column ids may not need remapping if the scan is not a view scan

commit 525f9c7
Author: David Stancu <david@spice.ai>
Date:   Thu Oct 30 10:42:17 2025 -0400

    do not do index scan if there are other non index filters in the predicate (fix shutdown_create_index.test)

commit b0a6e2d
Author: David Stancu <david@spice.ai>
Date:   Thu Oct 30 10:04:54 2025 -0400

    add test, bail out for eg composite query with IN () list

commit a22a430
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 16:37:30 2025 -0400

    simplify filter expression storage index bindings (just reuse the ones we made earlier), fix single-ref-per-expr predicate to correctly walk expr tree and yank refs (allowing nesting in fns, etc)

commit 9c8c1ed
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 15:11:23 2025 -0400

    copy index expressions before rewriting column refs

commit aff2c98
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 14:36:33 2025 -0400

    table scan:

    rebind projected columns in ALL index exprs
    do not bail out early if more than one index expr
    hook up composite key scan

commit bfc6f02
Author: David Stancu <david@spice.ai>
Date:   Wed Oct 29 14:35:09 2025 -0400

    make specialized compound key scan state for eq compares, specialized scan using ARTKey::Concat
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant