Skip to content

perf: restore typed load performance - #187

Merged
eitsupi merged 4 commits into
mainfrom
fix/manifest-load-regression
Aug 30, 2026
Merged

perf: restore typed load performance#187
eitsupi merged 4 commits into
mainfrom
fix/manifest-load-regression

Conversation

@eitsupi

@eitsupi eitsupi commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • Decode typed manifest reports in one streaming pass instead of building a full serde_json::Value tree and converting it with from_value.
  • Preserve forward-compatibility observations and diagnostics without scanning the JSON twice.
  • Decode nodes incrementally so diagnostics from valid entries survive later typed decode failures.
  • Split the manifest report implementation into report/mod.rs, report/decode.rs, and report/tests.rs.

Problem

The 0.2.5 release candidate introduced a full generic JSON tree before typed manifest decoding. On manifests with thousands of nodes, commands that require the typed manifest paid a node-linear regression even when the requested lineage was tiny. A reported 4,400-node production manifest added about 52 ms to column upstream and check-manifest; synthetic manifests showed the gap growing to about 80 ms at 16,000 nodes on macOS.

Design

The decoder now assigns known top-level fields directly into Manifest. Only values required for compatibility reporting remain generic JSON: metadata, untyped resource maps, and unknown top-level fields. A compile-time exhaustive Manifest pattern requires the visitor to be reviewed whenever a field is added.

nodes use an incremental typed visitor. Each successfully decoded node is classified once and any unsupported-resource observation is retained immediately. This keeps diagnostics available if a later node or top-level field is malformed, without another pass or a per-node Value round trip.

Known duplicate top-level fields remain errors, matching the stable 0.2.4 typed loader. Duplicate IDs inside the nodes map retain HashMap's last-value-wins behavior.

Performance

Linux x86_64, dist profile, Hyperfine 1.19.0, 3 warmups and 10 measured runs. The comparison uses the verified GitHub Release binary for 0.2.4, the 0.2.5 release-candidate baseline, and this branch.

check-manifest -q exercises the typed loader in a real dbt project layout:

Synthetic manifest 0.2.4 0.2.5 RC This PR
2,000 nodes 22.3 ms 75.8 ms 24.1 ms
8,000 nodes 70.0 ms 292.3 ms 79.6 ms
16,000 nodes 154.8 ms 589.8 ms 159.2 ms

The 0.2.5 RC overhead of 53.5/222.3/435.0 ms is reduced to 1.8/9.6/4.4 ms relative to 0.2.4.

summary --no-cache is also faster than both baselines in this environment. column upstream m5 --no-cache improves from 81.8/344.7/742.4 ms median in the RC to 39.5/94.4/201.6 ms. The remaining difference from 0.2.4 is not present in check-manifest, separating it from the typed-loader regression.

Compatibility

Tests cover:

  • missing, null, empty, and populated resource maps;
  • invalid and future producer metadata;
  • unknown top-level fields and unsupported resource types;
  • diagnostics retained across later top-level and node-entry decode failures;
  • syntax/EOF errors remaining parse-error-only;
  • stable duplicate-key behavior and diagnostic ordering;
  • permissive, strict, and legacy loading APIs.

Verification

  • cargo fmt --all -- --check
  • cargo test --workspace --locked (955 passed, 10 ignored; plus 94 CLI integration and 9 column-lineage tests)
  • cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • cargo build --profile dist --locked
  • column-lineage artifact and oracle validation

@eitsupi eitsupi changed the title perf(manifest): restore typed load performance perf: restore typed load performance Aug 30, 2026
@eitsupi
eitsupi merged commit 585920b into main Aug 30, 2026
16 checks passed
@eitsupi
eitsupi deleted the fix/manifest-load-regression branch August 30, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant