Skip to content

fix: Preserve filter type/int precision in doc updates - #5192

Merged
ChrisBQu merged 5 commits into
sourcenetwork:developfrom
ChrisBQu:issue-5191
Aug 25, 2026
Merged

fix: Preserve filter type/int precision in doc updates#5192
ChrisBQu merged 5 commits into
sourcenetwork:developfrom
ChrisBQu:issue-5191

Conversation

@ChrisBQu

@ChrisBQu ChrisBQu commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Relevant issue(s)

Resolves #5191

Description

Filters sent to UpdateDocumentsWithFilter/DeleteDocumentsWithFilter through any client that round-trips them as JSONcould silently target the wrong documents. This included the C bindings, as well as the HTTP client, CLI, and JS clients.

There were actually a number of issues going wrong here. immutable.Some(request.Filter{...}) was marshaled to {"Conditions": ...}, which decoded back into a filter with a literal Conditions field instead of the intended conditions. Further, raw map[string]any filters lost integer precision on decode (encoding/json decodes numbers as float64), so a condition on an integer above 2^53 could match a different document than intended. This was an issue that appeared, and was addressed in, several other recent PRs.

This PR introduces new functions, NormalizeFilterrForJSON and DecodeJSONFilter used on the encoding and decoding sides, respectively, to properly wire the values into the clients code.

To attain parity across the clients as much as possible, the JS client's filter parameter now also accepts an object as well as a filter string. However, because JavaScript does not support large integers, we are limited on that particular part off the issue. See: #5176

This PR contains new integration tests to illustrate the behavior. This includes two on which JS client support is marked as a to-do.

Tasks

  • I made sure the code is well commented, particularly hard-to-understand areas.
  • I made sure the repository-held documentation is changed accordingly.
  • I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
  • I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

Specify the platform(s) on which this was tested:

  • WSL

@ChrisBQu ChrisBQu self-assigned this Aug 24, 2026
@ChrisBQu ChrisBQu added the bug Something isn't working label Aug 24, 2026
@ChrisBQu ChrisBQu changed the title First pass fix: Preserve filter type/int precision in doc updates Aug 24, 2026
@ChrisBQu
ChrisBQu marked this pull request as ready for review August 24, 2026 21:08
@ChrisBQu
ChrisBQu requested a review from a team August 24, 2026 21:09
@coderabbitai

coderabbitai Bot commented Aug 24, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a6f722b2-e09c-4de9-aa12-b98e2a73edf8

📥 Commits

Reviewing files that changed from the base of the PR and between 20562c1 and 9791be5.

📒 Files selected for processing (1)
  • tests/integration/mutation/delete/with_filter_action_test.go
💤 Files with no reviewable changes (1)
  • tests/integration/mutation/delete/with_filter_action_test.go

Included review availability: Your plan provides up to 4 included reviews per hour; 2 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (41)
  • GitHub Check: Test macos job
  • GitHub Check: Test Limited Resource job
  • GitHub Check: Check wire format changes job
  • GitHub Check: Test coverage job (cli, memory, collection-save)
  • GitHub Check: Test coverage leveldb job
  • GitHub Check: Test coverage document acp job (cli, source-hub)
  • GitHub Check: Test coverage document acp job (c, source-hub)
  • GitHub Check: Test coverage job (http, file, collection-named)
  • GitHub Check: Test coverage encryption job
  • GitHub Check: Test coverage job (cli, file, collection-save)
  • GitHub Check: Test coverage job (c, memory, collection-named)
  • GitHub Check: Test coverage job (go, file, gql)
  • GitHub Check: Test coverage JS job
  • GitHub Check: Test coverage job (c, file, collection-save)
  • GitHub Check: Test coverage job (cli, file, collection-named)
  • GitHub Check: Test coverage document acp job (go, source-hub)
  • GitHub Check: Test coverage job (c, file, collection-named)
  • GitHub Check: Test coverage job (c, file, gql)
  • GitHub Check: Test coverage job (c, memory, gql)
  • GitHub Check: Test coverage job (http, file, gql)
  • GitHub Check: Test coverage job (cli, file, gql)
  • GitHub Check: Test coverage job (http, memory, gql)
  • GitHub Check: Test coverage job (http, file, collection-save)
  • GitHub Check: Test coverage job (http, memory, collection-save)
  • GitHub Check: Test coverage job (c, memory, collection-save)
  • GitHub Check: Test coverage job (go, file, collection-save)
  • GitHub Check: Test coverage job (go, memory, collection-save)
  • GitHub Check: Test coverage job (http, memory, collection-named)
  • GitHub Check: Test coverage secondary index job
  • GitHub Check: Test coverage job (go, file, collection-named)
  • GitHub Check: Test coverage lens job (wazero)
  • GitHub Check: Test coverage document acp job (http, source-hub)
  • GitHub Check: Test coverage job (go, memory, collection-named)
  • GitHub Check: Test coverage job (cli, memory, collection-named)
  • GitHub Check: Test coverage signed docs job
  • GitHub Check: Test coverage job (cli, memory, gql)
  • GitHub Check: Test coverage telemetry job
  • GitHub Check: Test coverage job (go, memory, gql)
  • GitHub Check: Test coverage view job
  • GitHub Check: Check data format changes job
  • GitHub Check: Lint GoLang job
⚠️ CI failures not shown inline (7)

GitHub Actions: Build Dependencies Workflow / 0_Build dependencies job.txt: fix: Preserve filter type/int precision in doc updates

Conclusion: failure

View job details

##[group]Run make deps
 �[36;1mmake deps�[0m
 shell: /usr/bin/bash -e {0}
 env:
   GOTOOLCHAIN: local
 ##[endgroup]
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 go mod download
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 go install golang.org/x/perf/cmd/benchstat@latest
 go: downloading golang.org/x/perf v0.0.0-20260819171926-ebcb4798430d
 go: golang.org/x/perf/cmd/benchstat@latest: golang.org/x/perf@v0.0.0-20260819171926-ebcb4798430d requires go >= 1.26.0 (running go 1.25.9; GOTOOLCHAIN=local)
 make[1]: *** [Makefile:166: deps:bench] Error 1
 make: *** [Makefile:196: deps] Error 2
 ##[error]Process completed with exit code 2.

GitHub Actions: Build Dependencies Workflow / Build dependencies job: fix: Preserve filter type/int precision in doc updates

Conclusion: failure

View job details

##[group]Run make deps
 �[36;1mmake deps�[0m
 shell: /usr/bin/bash -e {0}
 env:
   GOTOOLCHAIN: local
 ##[endgroup]
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 go mod download
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 go install golang.org/x/perf/cmd/benchstat@latest
 go: downloading golang.org/x/perf v0.0.0-20260819171926-ebcb4798430d
 go: golang.org/x/perf/cmd/benchstat@latest: golang.org/x/perf@v0.0.0-20260819171926-ebcb4798430d requires go >= 1.26.0 (running go 1.25.9; GOTOOLCHAIN=local)
 make[1]: *** [Makefile:166: deps:bench] Error 1
 make: *** [Makefile:196: deps] Error 2
 ##[error]Process completed with exit code 2.

GitHub Actions: Check Vulnerabilities Workflow / Check vulnerabilities job: fix: Preserve filter type/int precision in doc updates

Conclusion: failure

View job details

##[group]Run govulncheck -C . -format text ./... | tee govulncheck.txt
 �[36;1mgovulncheck -C . -format text ./... | tee govulncheck.txt�[0m
 shell: /usr/bin/bash -e {0}
 env:
   GOTOOLCHAIN: local
 ##[endgroup]
 === Symbol Results ===
 Vulnerability `#1`: GO-2026-6218
     Avoid quadratic complexity in resolvePath in net/url
   More info: https://pkg.go.dev/vuln/GO-2026-6218
   Standard library
     Found in: net/url@go1.25.9
     Fixed in: net/url@go1.25.13
     Example traces found:
 ##[error]      `#1`: http/client.go:580:30: http.Client.ExecRequest calls http.Client.Do, which eventually calls url.URL.Parse

GitHub Actions: Check Vulnerabilities Workflow / Check vulnerabilities job: fix: Preserve filter type/int precision in doc updates

Conclusion: failure

View job details

##[group]Run cat govulncheck.txt | grep "Your code is affected by 2 vulnerabilities from 1 module."
 �[36;1mcat govulncheck.txt | grep "Your code is affected by 2 vulnerabilities from 1 module."�[0m
 shell: /usr/bin/bash -e {0}
 env:
   GOTOOLCHAIN: local
 ##[endgroup]
 ##[error]Process completed with exit code 1.

GitHub Actions: Check Vulnerabilities Workflow / 0_Check vulnerabilities job.txt: fix: Preserve filter type/int precision in doc updates

Conclusion: failure

View job details

##[group]Run govulncheck -C . -format text ./... | tee govulncheck.txt
 �[36;1mgovulncheck -C . -format text ./... | tee govulncheck.txt�[0m
 shell: /usr/bin/bash -e {0}
 env:
   GOTOOLCHAIN: local
 ##[endgroup]
 === Symbol Results ===
 Vulnerability `#1`: GO-2026-6218
     Avoid quadratic complexity in resolvePath in net/url
   More info: https://pkg.go.dev/vuln/GO-2026-6218
   Standard library
     Found in: net/url@go1.25.9
     Fixed in: net/url@go1.25.13
     Example traces found:
 ##[error]      `#1`: http/client.go:580:30: http.Client.ExecRequest calls http.Client.Do, which eventually calls url.URL.Parse

GitHub Actions: Test Introspection JS Build Workflow / Test NPX_JS build job: fix: Preserve filter type/int precision in doc updates

Conclusion: failure

View job details

##[group]Run make test:npx
 �[36;1mmake test:npx�[0m
 shell: /usr/bin/bash -e {0}
 env:
   GOTOOLCHAIN: local
 ##[endgroup]
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 �[31m✖�[0m  internal/request/graphql/schema (19.04s)
 �[31m✖�[0m  cli (23.88s)
 === �[31mFailed�[0m
 === �[31mFAIL�[0m: cli TestSDLGenerate_SingleFileInput_Succeeds (16.65s)
     sdl_generate_npx_test.go:78: graphql-inspector output:
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-tools/code-file-loader/cjs/load-from-module.js'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/docs-command/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/code-loader/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/tslib/modules/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/coverage-command/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/graphql-loader/typings/index.d.ts'
 make: *** [Makefile:405: test:npx] Error 123
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/commands/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or ...

GitHub Actions: Test Introspection JS Build Workflow / 0_Test NPX_JS build job.txt: fix: Preserve filter type/int precision in doc updates

Conclusion: failure

View job details

##[group]Run make test:npx
 �[36;1mmake test:npx�[0m
 shell: /usr/bin/bash -e {0}
 env:
   GOTOOLCHAIN: local
 ##[endgroup]
 ----------------------------------------
 OS = Linux
 PACKAGE_MANAGER = apt
 GOINFO = go version go1.25.9 linux/amd64
 GITCOMMIT = ba5c6bce127188af544fc0ae96c86323d5d5af6f
 GITCOMMITDATE = 2026-08-25
 GITRELEASE = dev-
 ----------------------------------------
 �[31m✖�[0m  internal/request/graphql/schema (19.04s)
 �[31m✖�[0m  cli (23.88s)
 === �[31mFailed�[0m
 === �[31mFAIL�[0m: cli TestSDLGenerate_SingleFileInput_Succeeds (16.65s)
     sdl_generate_npx_test.go:78: graphql-inspector output:
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-tools/code-file-loader/cjs/load-from-module.js'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/docs-command/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/code-loader/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/tslib/modules/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/coverage-command/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/graphql-loader/typings/index.d.ts'
 make: *** [Makefile:405: test:npx] Error 123
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or directory, open '/home/runner/.npm/_npx/af070519648768fb/node_modules/@graphql-inspector/commands/typings/index.d.ts'
         npm warn tar TAR_ENTRY_ERROR ENOENT: no such file or ...

📝 Walkthrough

Walkthrough

Update and delete filters now use shared precision-preserving JSON decoding and optional-filter normalization across Go, HTTP, C, CLI, and JavaScript paths. Integration tests cover structured, optional, null, and large-integer filters.

Changes

Filter handling

Layer / File(s) Summary
Shared filter decoding and normalization
internal/utils/util.go
Adds DecodeJSONFilter with integer precision preservation and NormalizeFilterForJSON for optional request filters.
Go, C, and CLI filter paths
cbindings/document_delete.go, cbindings/document_update.go, cbindings/wrapper_document.go, cli/document_delete.go, cli/document_update.go, tests/clients/cli/wrapper_document.go
Update and delete paths use shared decoding and normalize absent or optional filters before processing or serialization.
HTTP request decoding and transport
http/handler_collection.go, http/client_document.go
HTTP requests decode filters with precision preservation, treat missing or null filters as match-all filters, and normalize outgoing filters.
JavaScript filter arguments
js/utils.go, js/client_document.go, tests/clients/js/wrapper_document.go
JavaScript update and delete operations accept filter strings, structured objects, and nullish values.
Cross-client update and delete validation
tests/integration/mutation/delete/with_filter_action_test.go, tests/integration/mutation/update/with_filter_action_test.go
Adds coverage for map filters, optional filters, match-all behavior, and exact large-integer matching.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to 9791b

The PR updates filter serialization and decoding across clients to preserve filter structure and integer precision; no actionable merge-blocking risk remains beyond normal checks and review.

Assessment against linked issues

Objective Addressed Explanation
Investigate update and delete filter parsing across affected clients [#5191]
Preserve optional filter semantics, including Some and None [#5191]
Accept map filters and preserve exact large integers [#5191]
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 53.48837% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.31%. Comparing base (c0c47f4) to head (9791be5).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
http/handler_collection.go 54.84% 7 Missing and 7 partials ⚠️
js/utils.go 44.44% 7 Missing and 3 partials ⚠️
internal/utils/util.go 73.33% 2 Missing and 2 partials ⚠️
cbindings/document_delete.go 25.00% 1 Missing and 2 partials ⚠️
cbindings/document_update.go 25.00% 1 Missing and 2 partials ⚠️
cli/document_delete.go 25.00% 1 Missing and 2 partials ⚠️
cli/document_update.go 25.00% 1 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff            @@
##           develop    #5192   +/-   ##
========================================
  Coverage    76.31%   76.31%           
========================================
  Files          645      645           
  Lines        51050    51122   +72     
========================================
+ Hits         38956    39011   +55     
- Misses        8827     8832    +5     
- Partials      3267     3279   +12     
Flag Coverage Δ
all-tests 76.31% <53.49%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
cbindings/wrapper_document.go 80.97% <100.00%> (ø)
http/client_document.go 72.45% <100.00%> (ø)
js/client_document.go 35.40% <100.00%> (ø)
cbindings/document_delete.go 53.19% <25.00%> (-2.36%) ⬇️
cbindings/document_update.go 55.56% <25.00%> (-2.14%) ⬇️
cli/document_delete.go 70.37% <25.00%> (-2.71%) ⬇️
cli/document_update.go 69.70% <25.00%> (-2.18%) ⬇️
internal/utils/util.go 72.06% <73.33%> (+0.36%) ⬆️
js/utils.go 49.57% <44.44%> (-0.95%) ⬇️
http/handler_collection.go 85.49% <54.84%> (-2.68%) ⬇️

... and 9 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c0c47f4...9791be5. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AndrewSisley AndrewSisley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks Chris :)

state.HTTPClientType,
state.CLIClientType,
state.CClientType,
state.JSClientType,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: This looks like all of our client types are listed here, why have you done this?

todo: Please document why, or remove it. Please apply this to the other similar tests below.

@ChrisBQu
ChrisBQu merged commit 67f59ab into sourcenetwork:develop Aug 25, 2026
84 of 94 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update/delete filters may be parsed incorrectly

2 participants