Skip to content

Add "Is not in" (notOneOf) filter operator#18932

Merged
melohagan merged 7 commits into
Budibase:masterfrom
Stropdasman:filterNotIn
Jun 15, 2026
Merged

Add "Is not in" (notOneOf) filter operator#18932
melohagan merged 7 commits into
Budibase:masterfrom
Stropdasman:filterNotIn

Conversation

@Stropdasman

@Stropdasman Stropdasman commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Addresses: #16460

Mirror the existing "Is in" (oneOf) operator with a negated "Is not in" (notOneOf) operator across the whole filter stack:

  • types: ArrayOperator.NOT_ONE_OF enum, SearchFilters field, Zod + Joi validation schemas
  • shared-core: operator option, valid operators per field type, in-memory runQuery matcher, numeric value parsing
  • backend-core: SQL builder (whereNotIn + orWhereNull so rows with empty values are returned), lucene builder
  • server: external row _id decoding
  • client/frontend-core/builder: multiselect value input, datetime exclusion, conditional-UI editors

Rows with an empty/null value are returned by "Is not in" (they are not in the list), matching intuitive expectations.


Summary by cubic

Adds an “Is not in” (notOneOf) filter operator across the stack. Mirrors oneOf in SQL, Lucene, and in-memory filters, and returns rows with empty/null values.

  • New Features

    • Types/validation: added ArrayOperator.NOT_ONE_OF, FilterType.NOT_ONE_OF, and SearchFilters.notOneOf in @budibase/types; Zod + Joi schemas; OpenAPI specs regenerated.
    • Query engines: SQL uses whereNotIn + orWhereNull (SQLite date-only via LIKE; Oracle CLOBs handled); Lucene builds (*:* -field:(...)); in-memory runQuery inverses oneOf and includes nulls.
    • Server: decodes _id values for notOneOf in external row search; validators and generated OpenAPI types include notOneOf.
    • UI: multiselect inputs support notOneOf; hidden for DateTime; type selectors disabled for oneOf/notOneOf; conditional-UI editors updated.
    • Operator availability: enabled for OPTIONS, single refs, and external SQL _id; added to no-empty list; numeric CSV values parsed for oneOf/notOneOf.
    • Tests: Postgres notOneOf uses whereNotIn + is null; oneOf uses whereIn; server route coverage; shared-core runQuery cases.
  • Bug Fixes

    • Server: prevent crash on external SQL datasources by skipping notOneOf arrays in cleanupConfig and adding FilterType.NOT_ONE_OF.

Written for commit 8bfa5fc. Summary will update on new commits.

Review in cubic

Mirror the existing "Is in" (oneOf) operator with a negated "Is not in"
(notOneOf) operator across the whole filter stack:

- types: ArrayOperator.NOT_ONE_OF enum, SearchFilters field, Zod + Joi
  validation schemas
- shared-core: operator option, valid operators per field type, in-memory
  runQuery matcher, numeric value parsing
- backend-core: SQL builder (whereNotIn + orWhereNull so rows with empty
  values are returned), lucene builder
- server: external row _id decoding
- client/frontend-core/builder: multiselect value input, datetime exclusion,
  conditional-UI editors

Rows with an empty/null value are returned by "Is not in" (they are not in
the list), matching intuitive expectations.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Stropdasman

Copy link
Copy Markdown
Contributor Author

@cubic-dev-ai

@cubic-dev-ai

cubic-dev-ai Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai

@Stropdasman I can’t start this review because this PR was opened by an external contributor in a public repository. Ask an installation member or outside collaborator to comment @cubic review this.

@NDCallahan

NDCallahan commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

@Stropdasman - in addition to this, would you add Greater than and less than in the operands please?

@Stropdasman

Copy link
Copy Markdown
Contributor Author

@NDCallahan, this option is already available for numeric columns, or when you choose “Number” as the condition type.

@melohagan melohagan self-requested a review June 15, 2026 12:32

@melohagan melohagan left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I pulled it down locally and gave it a test with BudibaseDB and Postgres, and appeared to be working as intended.

Nice addition, and the tests are appreciated too. 🙂

@melohagan

Copy link
Copy Markdown
Collaborator

Ah looks like there's some test failures to fix!

Error: Expected status 200 but got 500

Body:
⏐ {
⏐   "message": "filter[key].map is not a function",
⏐   "status": 500
⏐ }

Stack from request handler:
⏐ TypeError: filter[key].map is not a function
⏐     at InternalBuilder.map [as parseFilters] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:523:35)
⏐     at InternalBuilder.parseFilters [as addFilters] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:696:20)
⏐     at InternalBuilder.addFilters [as read] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:1860:18)
⏐     at MySQLIntegration.read [as _query] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:1956:25)
⏐     at MySQLIntegration._query [as queryWithReturning] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:2032:24)
⏐     at MySQLIntegration.queryWithReturning [as query] (/home/runner/work/budibase/budibase/packages/server/src/integrations/mysql.ts:422:25)
⏐     at processTicksAndRejections (node:internal/process/task_queues:103:5)
⏐     at AliasTables.queryWithAliasing (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/sqlAlias.ts:241:20)
⏐     at ExternalRequest.run (/home/runner/work/budibase/budibase/packages/server/src/api/controllers/row/ExternalRequest.ts:765:11)
⏐     at async Promise.all (index 0)
⏐     at Object.search (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search/external.ts:117:52)
⏐     at /home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search.ts:138:16
⏐     at Object.search (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search.ts:42:10)
⏐     at search (/home/runner/work/budibase/budibase/packages/server/src/api/controllers/row/index.ts:368:14)
⏐     at cleanupMiddleware (/home/runner/work/budibase/budibase/packages/server/src/middleware/cleanup.ts:7:16)
⏐     at contentSecurityPolicy (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/contentSecurityPolicy.ts:201:3)
⏐     at /home/runner/work/budibase/budibase/node_modules/koa-compress/lib/index.js:38:5
⏐     at /home/runner/work/budibase/budibase/packages/backend-core/src/middleware/featureFlagCookie.ts:11:5
⏐     at doInFeatureFlagOverrideContext (/home/runner/work/budibase/budibase/packages/backend-core/src/context/mainContext.ts:491:10)
⏐     at featureFlagCookie (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/featureFlagCookie.ts:10:3)
⏐     at errorHandling (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/errorHandling.ts:9:5)
⏐     at userAgent (/home/runner/work/budibase/budibase/node_modules/koa-useragent/dist/index.js:12:5)
⏐     at ip (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/ip.ts:6:12)
⏐     at /home/runner/work/budibase/budibase/packages/server/src/koa.ts:56:7
⏐     at /home/runner/work/budibase/budibase/packages/server/src/koa.ts:47:12
    at RowAPI._checkResponse (src/tests/utilities/api/base.ts:211:15)
    at RowAPI._checkResponse [as _request] (src/tests/utilities/api/base.ts:239:17)
    at RowAPI._post (src/tests/utilities/api/base.ts:65:12)
    at RowAPI.search (src/tests/utilities/api/row.ts:158:12)
    at SearchAssertion.toContainExactly (src/api/routes/tests/search.spec.ts:340:34)
    at SearchAssertion.toFindNothing (src/api/routes/tests/search.spec.ts:408:17)
    at Object.<anonymous> (src/api/routes/tests/search.spec.ts:1080:19)

Errors thrown in src/api/routes/tests/search.spec.ts
    ● search (mariadb) › in-memory: false › from table › string › notOneOf › successfully finds rows not in the list
  
      Expected status 200 but got 500
  
      Body:
      ⏐ {
      ⏐   "message": "filter[key].map is not a function",
      ⏐   "status": 500
      ⏐ }
  
      Stack from request handler:
      ⏐ TypeError: filter[key].map is not a function
      ⏐     at InternalBuilder.map [as parseFilters] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:523:35)
      ⏐     at InternalBuilder.parseFilters [as addFilters] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:696:20)
      ⏐     at InternalBuilder.addFilters [as read] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:1860:18)
      ⏐     at MySQLIntegration.read [as _query] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:1956:25)
      ⏐     at MySQLIntegration._query [as queryWithReturning] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:2032:24)
      ⏐     at MySQLIntegration.queryWithReturning [as query] (/home/runner/work/budibase/budibase/packages/server/src/integrations/mysql.ts:422:25)
      ⏐     at processTicksAndRejections (node:internal/process/task_queues:103:5)
      ⏐     at AliasTables.queryWithAliasing (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/sqlAlias.ts:241:20)
      ⏐     at ExternalRequest.run (/home/runner/work/budibase/budibase/packages/server/src/api/controllers/row/ExternalRequest.ts:765:11)
      ⏐     at async Promise.all (index 0)
      ⏐     at Object.search (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search/external.ts:117:52)
      ⏐     at /home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search.ts:138:16
      ⏐     at Object.search (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search.ts:42:10)
      ⏐     at search (/home/runner/work/budibase/budibase/packages/server/src/api/controllers/row/index.ts:368:14)
      ⏐     at cleanupMiddleware (/home/runner/work/budibase/budibase/packages/server/src/middleware/cleanup.ts:7:16)
      ⏐     at contentSecurityPolicy (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/contentSecurityPolicy.ts:201:3)
      ⏐     at /home/runner/work/budibase/budibase/node_modules/koa-compress/lib/index.js:38:5
      ⏐     at /home/runner/work/budibase/budibase/packages/backend-core/src/middleware/featureFlagCookie.ts:11:5
      ⏐     at doInFeatureFlagOverrideContext (/home/runner/work/budibase/budibase/packages/backend-core/src/context/mainContext.ts:491:10)
      ⏐     at featureFlagCookie (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/featureFlagCookie.ts:10:3)
      ⏐     at errorHandling (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/errorHandling.ts:9:5)
      ⏐     at userAgent (/home/runner/work/budibase/budibase/node_modules/koa-useragent/dist/index.js:12:5)
      ⏐     at ip (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/ip.ts:6:12)
      ⏐     at /home/runner/work/budibase/budibase/packages/server/src/koa.ts:56:7
      ⏐     at /home/runner/work/budibase/budibase/packages/server/src/koa.ts:47:12
  
        209 |           response.error.message = response.error.text
        210 |         }
      > 211 |         throw new Error(message, { cause: response.error })
            |               ^
        212 |       } else {
        213 |         throw new Error(message)
        214 |       }
  
        at RowAPI._checkResponse (src/tests/utilities/api/base.ts:211:15)
        at RowAPI._checkResponse [as _request] (src/tests/utilities/api/base.ts:239:17)
        at RowAPI._post (src/tests/utilities/api/base.ts:65:12)
        at RowAPI.search (src/tests/utilities/api/row.ts:158:12)
        at SearchAssertion.toContainExactly (src/api/routes/tests/search.spec.ts:340:34)
        at Object.<anonymous> (src/api/routes/tests/search.spec.ts:1060:19)
  
      Cause:
      {"message":"filter[key].map is not a function","status":500,"stack":"TypeError: filter[key].map is not a function\n    at InternalBuilder.map [as parseFilters] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:523:35)\n    at InternalBuilder.parseFilters [as addFilters] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:696:20)\n    at InternalBuilder.addFilters [as read] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:1860:18)\n    at MySQLIntegration.read [as _query] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:1956:25)\n    at MySQLIntegration._query [as queryWithReturning] (/home/runner/work/budibase/budibase/packages/backend-core/src/sql/sql.ts:2032:24)\n    at MySQLIntegration.queryWithReturning [as query] (/home/runner/work/budibase/budibase/packages/server/src/integrations/mysql.ts:422:25)\n    at processTicksAndRejections (node:internal/process/task_queues:103:5)\n    at AliasTables.queryWithAliasing (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/sqlAlias.ts:241:20)\n    at ExternalRequest.run (/home/runner/work/budibase/budibase/packages/server/src/api/controllers/row/ExternalRequest.ts:765:11)\n    at async Promise.all (index 0)\n    at Object.search (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search/external.ts:117:52)\n    at /home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search.ts:138:16\n    at Object.search (/home/runner/work/budibase/budibase/packages/server/src/sdk/workspace/rows/search.ts:42:10)\n    at search (/home/runner/work/budibase/budibase/packages/server/src/api/controllers/row/index.ts:368:14)\n    at cleanupMiddleware (/home/runner/work/budibase/budibase/packages/server/src/middleware/cleanup.ts:7:16)\n    at contentSecurityPolicy (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/contentSecurityPolicy.ts:201:3)\n    at /home/runner/work/budibase/budibase/node_modules/koa-compress/lib/index.js:38:5\n    at /home/runner/work/budibase/budibase/packages/backend-core/src/middleware/featureFlagCookie.ts:11:5\n    at doInFeatureFlagOverrideContext (/home/runner/work/budibase/budibase/packages/backend-core/src/context/mainContext.ts:491:10)\n    at featureFlagCookie (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/featureFlagCookie.ts:10:3)\n    at errorHandling (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/errorHandling.ts:9:5)\n    at userAgent (/home/runner/work/budibase/budibase/node_modules/koa-useragent/dist/index.js:12:5)\n    at ip (/home/runner/work/budibase/budibase/packages/backend-core/src/middleware/ip.ts:6:12)\n    at /home/runner/work/budibase/budibase/packages/server/src/koa.ts:56:7\n    at /home/runner/work/budibase/budibase/packages/server/src/koa.ts:47:12"}

@Stropdasman

Copy link
Copy Markdown
Contributor Author

@melohagan Thanks for the review. Will push changes soon.

Stropdasman and others added 2 commits June 15, 2026 16:53
cleanupConfig only skipped oneOf, so notOneOf array values were mangled by convertRowId into scalars, crashing parseFilters. Skip notOneOf too and add FilterType.NOT_ONE_OF.
@Stropdasman Stropdasman requested a review from melohagan June 15, 2026 14:55
@melohagan melohagan merged commit f092666 into Budibase:master Jun 15, 2026
34 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants