Skip to content

[bug] QueryPackagesListForScan fails with PostgreSQL parameter limit error on large datasets #2825

@musapelm

Description

@musapelm

Describe the bug
When using the visualizer (v0.4.10) with large datasets, queryPackagesListForScan fails with the following error:

failed package query with error: pq: got 172313 parameters but PostgreSQL only supports 65535 parameters

This occurs when the GraphQL endpoint receives a large list of package IDs that need to be queried from the PostgreSQL backend.

To Reproduce

  1. Ingest a large dataset with >60,000 packages into GUAC
  2. Use the visualizer (v0.4.10) to query package data
  3. The visualizer calls queryPackagesListForScan with a large pkgIDs array
  4. Backend attempts to execute: Where(packageversion.IDIn(shortenedQueryList...))
  5. Error occurs when shortenedQueryList exceeds ~65,000 UUIDs

Expected behavior
The query should succeed by automatically batching large ID lists internally to stay within PostgreSQL's parameter limits.

Screenshots
N/A

GUAC version

Additional context

  • PR fix issues with certifier querying running into postgres parameter limit #2184 split the query into findPackagesThatNeedScanning and queryPackagesListForScan
  • However, when clients (like the visualizer) pass large pkgIDs arrays directly to queryPackagesListForScan, the backend still tries to query all IDs at once
  • PostgreSQL's parameter limit is 65,535
  • Each UUID in IDIn() counts as one parameter
  • The issue manifests when pkgIDs array > ~60,000 items

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions