Skip to content

Commit 8a9f4a3

Browse files
eywalkereywalker
authored
Merge branch 'main' into eywalker/itl-627-extension-type-metadata-dropped-for-list-backed-logical
2 parents d0a9044 + a19b814 commit 8a9f4a3

14 files changed

Lines changed: 1199 additions & 32 deletions

‎.github/workflows/_license-check.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,13 @@ jobs:
1414
name: License check
1515
runs-on: ubuntu-latest
1616
steps:
17-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
17+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
1818
with:
1919
fetch-depth: 0 # required: hatch-vcs runs during uv sync
2020
ref: ${{ inputs.branch }}
2121

2222
- name: Install uv
23-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
23+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
2424

2525
- name: Install system dependencies
2626
run: sudo apt-get update && sudo apt-get install -y graphviz libgraphviz-dev

‎.github/workflows/release-sync.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ jobs:
1313
contents: read
1414
steps:
1515
- name: Checkout
16-
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
16+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
1717
with:
1818
fetch-depth: 0
1919
- name: Sync Linear release

‎.github/workflows/release.yml‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -41,13 +41,13 @@ jobs:
4141
matrix:
4242
python-version: ["3.11", "3.12"]
4343
steps:
44-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
44+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
4545
with:
4646
fetch-depth: 0 # required: hatch-vcs needs full tag history
4747
ref: ${{ inputs.branch }}
4848

4949
- name: Install uv
50-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
50+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
5151

5252
- name: Install system dependencies
5353
run: sudo apt-get update && sudo apt-get install -y graphviz libgraphviz-dev
@@ -81,7 +81,7 @@ jobs:
8181
VERSION="${VERSION#v}"
8282
echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
8383
84-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
84+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
8585
with:
8686
fetch-depth: 0 # required: hatch-vcs reads tag to set version
8787
ref: ${{ inputs.branch }}
@@ -95,7 +95,7 @@ jobs:
9595
run: git tag "v${{ steps.normalize.outputs.version }}"
9696

9797
- name: Install uv
98-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
98+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
9999

100100
- name: Build wheel and sdist
101101
run: uv build
@@ -129,7 +129,7 @@ jobs:
129129
id-token: write # required for OIDC Trusted Publishing
130130
steps:
131131
- name: Install uv
132-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
132+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
133133

134134
- name: Download dist artifact
135135
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
@@ -153,7 +153,7 @@ jobs:
153153
contents: write # required for creating GitHub Release
154154
steps:
155155
- name: Install uv
156-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
156+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
157157

158158
- name: Download dist artifact
159159
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
@@ -180,7 +180,7 @@ jobs:
180180
contents: read
181181
steps:
182182
- name: Checkout
183-
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
183+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
184184
with:
185185
fetch-depth: 0
186186
- name: Sync Linear release
@@ -199,7 +199,7 @@ jobs:
199199
contents: read
200200
steps:
201201
- name: Checkout
202-
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
202+
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
203203
with:
204204
fetch-depth: 0
205205
- name: Complete Linear release

‎.github/workflows/run-objective-tests.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,13 @@ jobs:
1414
python-version: ["3.11", "3.12"]
1515

1616
steps:
17-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
17+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
1818

1919
- name: Install uv
20-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
20+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
2121

2222
- name: Set up Python ${{ matrix.python-version }}
23-
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
23+
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
2424
with:
2525
python-version: ${{ matrix.python-version }}
2626

@@ -34,6 +34,6 @@ jobs:
3434
run: uv run --python ${{ matrix.python-version }} pytest test-objective/ -v --cov=src --cov-report=term-missing --cov-report=xml
3535

3636
- name: Upload coverage reports to Codecov
37-
uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5.5.5
37+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
3838
with:
3939
token: ${{ secrets.CODECOV_TOKEN }}

‎.github/workflows/run-postgres-tests.yml‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,13 @@ jobs:
2929
--health-retries 5
3030
3131
steps:
32-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
32+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
3333

3434
- name: Install uv
35-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
35+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
3636

3737
- name: Set up Python ${{ matrix.python-version }}
38-
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
38+
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
3939
with:
4040
python-version: ${{ matrix.python-version }}
4141

@@ -55,6 +55,6 @@ jobs:
5555
PGDATABASE: testdb
5656

5757
- name: Upload coverage reports to Codecov
58-
uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5.5.5
58+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
5959
with:
6060
token: ${{ secrets.CODECOV_TOKEN }}

‎.github/workflows/run-tests.yml‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,15 +15,15 @@ jobs:
1515
python-version: ["3.11", "3.12"]
1616

1717
steps:
18-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
18+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
1919
with:
2020
fetch-depth: 0 # required: hatch-vcs needs full tag history
2121

2222
- name: Install uv
23-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
23+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
2424

2525
- name: Set up Python ${{ matrix.python-version }}
26-
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
26+
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
2727
with:
2828
python-version: ${{ matrix.python-version }}
2929

@@ -37,7 +37,7 @@ jobs:
3737
run: uv run --python ${{ matrix.python-version }} pytest -v -m "not postgres" --cov=src --cov-report=term-missing --cov-report=xml
3838

3939
- name: Upload coverage reports to Codecov
40-
uses: codecov/codecov-action@0fb7174895f61a3b6b78fc075e0cd60383518dac # v5.5.5
40+
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
4141
with:
4242
token: ${{ secrets.CODECOV_TOKEN }}
4343

@@ -65,15 +65,15 @@ jobs:
6565
echo "::error::SPIRAL_WORKLOAD_ID secret is not set. SpiralDB integration tests cannot run. Configure the secret or investigate why it is missing."
6666
exit 1
6767
68-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
68+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
6969
with:
7070
fetch-depth: 0 # required: hatch-vcs needs full tag history
7171

7272
- name: Install uv
73-
uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
73+
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
7474

7575
- name: Set up Python 3.11
76-
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
76+
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
7777
with:
7878
python-version: "3.11"
7979

@@ -94,10 +94,10 @@ jobs:
9494
runs-on: ubuntu-latest
9595
if: github.event_name == 'pull_request'
9696
steps:
97-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
97+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
9898

9999
- name: Dependency review
100-
uses: actions/dependency-review-action@2031cfc080254a8a887f58cffee85186f0e49e48 # v4.9.0
100+
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
101101
with:
102102
deny-licenses: >-
103103
GPL-2.0-only, GPL-2.0-or-later,

‎.github/workflows/tests.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ jobs:
1010
runs-on: ubuntu-latest
1111

1212
steps:
13-
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
13+
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
1414

1515
- name: Start MinIO
1616
run: |
@@ -24,7 +24,7 @@ jobs:
2424
- name: Install system dependencies
2525
run: sudo apt-get install -y libgraphviz-dev graphviz
2626

27-
- uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
27+
- uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
2828
with:
2929
enable-cache: true
3030

‎DESIGN_ISSUES.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,32 @@ lock before iterating, so later writes to the stream don't affect the snapshot m
124124

125125
---
126126

127+
### PS3 — `_combine` leaks `_content_hash` into the data schema on the second accumulating fetch
128+
**Status:** resolved
129+
**Severity:** high
130+
**Issue:** ITL-616
131+
132+
`_combine` calls `as_table(all_info=True)` on both streams before concatenating them.
133+
`all_info=True` resolves to `ColumnConfig.all()`, which includes `content_hash=True`.
134+
In `ArrowTableStream.as_table()`, `content_hash=True` dynamically appends a `_content_hash`
135+
column to the output table. This is a synthetic column — computed on demand, not stored in
136+
`ArrowTableStream._table`.
137+
138+
`pa.concat_tables` then includes `_content_hash` in the combined table, which is passed
139+
directly to `ArrowTableStream.__init__`. Since `_content_hash` has no recognized prefix
140+
(`_tag::`, `_source_`, `_context_key`), it lands in `_data_columns` as if it were user data.
141+
142+
On the next `_combine` call, `_validate_combining_schemas` compares:
143+
- `existing.keys()` → includes `_content_hash` in data keys (baked in from previous combine)
144+
- `new_stream.keys()` → no `_content_hash` (freshly built from raw fetched data)
145+
146+
This raises `SchemaInconsistencyError`. A polling source emitting one new row per poll will
147+
change its data schema on the second new-data poll and crash on the third.
148+
149+
**Fix:** Added `_STREAM_COMBINE_COLUMNS = ColumnConfig(system_tags=True, source=True, context=True)` constant and replaced `as_table(all_info=True)` with `as_table(columns=_STREAM_COMBINE_COLUMNS)` in `_combine`. `content_hash` is intentionally excluded — it is a synthetic output column, never a stored one.
150+
151+
---
152+
127153
## `src/orcapod/core/nodes/function_node.py`
128154

129155
### FN1 — `FunctionNodeBase.as_table()` returned empty schema when no data existed

‎src/orcapod/core/sources/polling_source.py‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,16 @@ def _assert_schema_match(
121121
)
122122

123123

124+
# ColumnConfig used when concatenating streams in _combine.
125+
# Includes the provenance columns (system_tags, source, context) that
126+
# ArrowTableStream.__init__ knows how to parse and split into their
127+
# respective internal tables.
128+
# content_hash is intentionally absent: it is a synthetic, on-demand
129+
# column produced by as_table(); including it would bake it into stored
130+
# data and corrupt the data schema on the next combine.
131+
_STREAM_COMBINE_COLUMNS = ColumnConfig(system_tags=True, source=True, context=True)
132+
133+
124134
# ---------------------------------------------------------------------------
125135
# PollingSource
126136
# ---------------------------------------------------------------------------
@@ -652,8 +662,8 @@ def _combine(
652662

653663
combined = pa.concat_tables(
654664
[
655-
existing.as_table(all_info=True),
656-
new_stream.as_table(all_info=True),
665+
existing.as_table(columns=_STREAM_COMBINE_COLUMNS),
666+
new_stream.as_table(columns=_STREAM_COMBINE_COLUMNS),
657667
],
658668
promote_options="default",
659669
)

0 commit comments

Comments
 (0)