Skip to content

fix: avoid linear objectHas scans for large inline imports - #1070

Merged
stephenamar-db merged 1 commit into
databricks:masterfrom
He-Pin:codex/pr1061-rework
Jun 30, 2026
Merged

fix: avoid linear objectHas scans for large inline imports#1070
stephenamar-db merged 1 commit into
databricks:masterfrom
He-Pin:codex/pr1061-rework

Conversation

@He-Pin

@He-Pin He-Pin commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Motivation:
PR #1061 was closed against the obsolete fixjson base, but the issue still needs a clean PR on the current upstream master. PR #1060 added the InlineScanMax fallback for valueRaw, yet repeated std.objectHas / std.objectHasAll checks over large imported strict-JSON objects still reached containsVisibleKey / containsKey and scanned uncapped inline field arrays. That kept membership-heavy workloads at O(N^2).

Modification:

  • Rebase the PR fix: avoid linear objectHas scans for large inline imports #1061 follow-up onto current upstream/master (1db39dde).
  • Apply Obj.InlineScanMax to Val.Obj.containsKey and Val.Obj.containsVisibleKey, so large inline objects fall through to the existing lazy getValue0 LinkedHashMap path.
  • Reuse Obj.InlineScanMax in inline-object producers, keeping construction and lookup thresholds on one source of truth.
  • Add a benchmark regression covering repeated std.objectHas / std.objectHasAll calls over imported JSON object fields.

Result:

Benchmark Current master This branch Change
repeated std.objectHas / std.objectHasAll over large imported JSON object fields 97.942 ms/op 21.117 ms/op 4.64x faster (-78%)

Large inline imported objects now pay one lazy map build above the scan threshold, then reuse O(1) map probes for repeated membership checks. Small inline objects keep the cheap linear scan path.

References:

Motivation:
Large imported strict-JSON objects can use uncapped inline field arrays. PR databricks#1060 routes value lookups through a lazy map above the inline scan threshold, but objectHas/objectHasAll still used linear contains scans.

Modification:
Reuse Obj.InlineScanMax in containsKey, containsVisibleKey, and inline-object producers so large inline objects fall through to getValue0 while all inline thresholds share one source of truth. Add a bench regression that repeatedly calls objectHas/objectHasAll over imported JSON fields.

Result:
Keeps small inline-object scans while avoiding O(N^2) repeated membership checks on large imported JSON objects.

References: databricks#1061
@He-Pin
He-Pin force-pushed the codex/pr1061-rework branch from 1342f67 to 0f177a9 Compare June 30, 2026 02:55
@He-Pin
He-Pin marked this pull request as draft June 30, 2026 03:22
@He-Pin
He-Pin marked this pull request as ready for review June 30, 2026 04:35
@stephenamar-db
stephenamar-db merged commit c59f86b into databricks:master Jun 30, 2026
5 checks passed
@He-Pin
He-Pin deleted the codex/pr1061-rework branch June 30, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants