You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- Spec doc: correct three stale inaccuracies about arrow-ipc metadata key
ordering (same corrections already applied to the Rust repo's copy):
IPC intro no longer claims insertion order is preserved; key_reorder
section now says byte-identical IPC blobs; risk section corrected to say
arrow-ipc sorts keys alphabetically, not preserves insertion order
- golden-sync-check: add job-level if guard so the job is skipped for PRs
from forks (secrets not available to forked workflows)
- release.yml: strip leading 'v' from the version input before tagging to
prevent 'vv0.3.0' tags when operator includes the prefix
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Arrow IPC is used for `ipc_b64` because it captures the exact bytes — including metadata key
50
-
insertion order — that were fed to the Rust hasher. This eliminates any risk of Python
51
-
constructing subtly different Arrow data.
49
+
Arrow IPC is used for `ipc_b64` because it provides a stable, self-contained encoding of the
50
+
Arrow schema (including all metadata) that both Rust and Python can deserialize identically.
51
+
Note: `arrow-ipc`'s `metadata_to_fb` sorts metadata keys alphabetically before FlatBuffers
52
+
encoding (`ordered_keys.sort()` in convert.rs), so the IPC byte stream is deterministic
53
+
regardless of HashMap insertion order at the producer side.
52
54
53
55
---
54
56
@@ -67,8 +69,12 @@ constructing subtly different Arrow data.
67
69
|`empty_metadata_invariant`| No metadata at all — tested with `include_metadata=false`; `expected_hash` must equal that of the same schema hashed with `include_metadata=true`|`false`|
68
70
69
71
The `key_reorder_canonical` / `key_reorder_shuffled` pair encodes the key-ordering determinism
70
-
invariant directly in the fixture: two different IPC blobs (different insertion orders) map to
71
-
the same `expected_hash`.
72
+
invariant directly in the fixture. Because `arrow-ipc` sorts metadata keys alphabetically before
73
+
FlatBuffers encoding, both vectors produce **byte-identical IPC blobs** — the insertion-order
74
+
invariant is enforced at the IPC level, not the hasher level. Both vectors therefore share the
75
+
same `ipc_b64` and the same `expected_hash`. The test verifies that the hasher also produces
76
+
matching output when the live hasher is called directly on schemas built with different insertion
77
+
orders.
72
78
73
79
The `empty_metadata_invariant` entry pins the empty-metadata fixed point: a schema with no
74
80
metadata must produce the same hash regardless of `include_metadata`. Only one entry is needed
@@ -374,10 +380,12 @@ Tag push fires the existing `publish.yml` → pure-Python package published to P
374
380
375
381
## Risks
376
382
377
-
- **IPC metadata order:** Arrow IPC preserves key insertion order in its FlatBuffers encoding.
378
-
This is load-bearing for the `key_reorder_*` vectors. If a future Arrow version changes this
379
-
behaviour the vectors would need to be regenerated, but the fixture format itself remains
380
-
valid.
383
+
- **IPC metadata order:** Arrow IPC does **not** preserve key insertion order — `metadata_to_fb`
384
+
in arrow-ipc sorts keys alphabetically before FlatBuffers encoding (`ordered_keys.sort()` in
385
+
convert.rs). As a result, the `key_reorder_canonical` and `key_reorder_shuffled` vectors
386
+
produce byte-identical IPC blobs. If a future Arrow version changes this sorting behaviour,
387
+
the `key_reorder_*` IPC blobs would diverge and the vectors would need to be regenerated;
388
+
the fixture format itself remains valid.
381
389
- **Fixture drift:** Mitigated by the `golden-sync-check` CI job. If the GitHub App secret
382
390
expires or is revoked, the drift check will fail loudly rather than silently passing.
383
391
- **Version/tag sync:** The `verify-version-tag-sync` CI job enforces the invariant on every
0 commit comments