Add chips and filtering to AI Search by ellenmuller · Pull Request #4752 · guardian/grid

ellenmuller · 2026-06-02T16:09:08Z

What does this change?

This PR makes AI search respect Grid’s existing chip/filter behaviour and adds structured support for similar: queries.

Previously, 'Use AI search' embedded the raw query, including any chips, so structured chips such as file type and other filters did not work as one might expect. This branch changes the frontend and backend flow so AI search separates semantic text from structured filters, applies those filters to Elasticsearch, and handles similar: as part of the structured query model rather than via ad hoc string parsing.

Changes

Added similarity 'chip' to the structured query parser.
Switched AI mode detection to use structured query parsing.
Split AI queries into:
- semantic text to embed
- structured filter conditions to apply to Elasticsearch
Reused QueryBuilder logic so AI mode inherits the same chip semantics as standard search.
Wired filter application through hybrid search and similar-image KNN search:
- added inline KNN filtering to improve recall for filtered similar-image searches.
- kept top-level query filtering so final hybrid results still obey filters on the lexical branch.
Added backend guards for invalid AI queries:
- chips-only AI text queries just say that the user needs to add a query
- mixed similar: plus positive semantic text also doesn't work
Added matching frontend guards so unsupported AI queries don't trickle end up in backend. (Copilot helped here!)
Updated the search UI so filters are visible again in AI mode, except for sorting date. (Copilot helped here!)

How should a reviewer test this change?

How can success be measured?

Who should look at this?

Tested? Documented?

locally by committer
locally by Guardian reviewer
on the Guardian's TEST environment
relevant documentation added or amended (if needed)

github-actions · 2026-06-02T16:16:59Z

Deploy build 14540 of `media-service::grid::all` to TEST

All deployment options

From guardian/actions-riff-raff.

…ch-filtering

Copilot

Pull request overview

This PR updates Grid’s AI search flow so it respects the existing structured query “chips”/filters, and adds first-class structured parsing support for similar: queries. It splits AI queries into (a) semantic free text to embed and (b) structured filter conditions to apply to Elasticsearch, reusing existing query-building/filter logic so AI mode behaves consistently with standard search.

Changes:

Added similar: as a structured query filter in the server-side parser + model, with corresponding parser tests.
Introduced AiQueryParts to split structured queries into semantic text, filter conditions, and optional similar-image ID; updated Media API AI search to use these parts and apply filters to hybrid/KNN queries.
Updated Kahuna UI to re-enable filters in AI mode and add frontend guards for unsupported AI query shapes.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
media-api/test/lib/querysyntax/ParserTest.scala	Adds coverage to ensure `similar:` is parsed into structured conditions.
media-api/app/lib/querysyntax/QuerySyntax.scala	Extends parser grammar to recognize `similar:` as a filter.
media-api/app/lib/querysyntax/model.scala	Adds `SimilarField` / `SimilarValue` to the structured query model.
media-api/app/lib/elasticsearch/QueryBuilder.scala	Handles `SimilarField` explicitly and centralizes construction of request-level filter queries via `buildFilterOpt`.
media-api/app/lib/elasticsearch/ElasticSearchModel.scala	Adds `AiQueryParts` and exposes `SearchParams.aiQueryParts` for AI query splitting.
media-api/app/lib/elasticsearch/ElasticSearch.scala	Threads `filterOpt` through KNN + hybrid queries; uses `QueryBuilder.buildFilterOpt` during standard search.
media-api/app/controllers/MediaApi.scala	Switches AI mode detection and filtering to use `SearchParams.aiQueryParts`; blocks unsupported `similar:+semantic` combinations.
kahuna/public/js/search/structured-query/syntax.ts	Adds low-level query validation helpers for AI mode (positive unfielded text vs `similar:`).
kahuna/public/js/search/results.js	Updates AI “needs query” guard logic using the new structured-query helpers.
kahuna/public/js/search/query.html	Makes filter UI visible in AI mode (except sort UI remains hidden for AI).
kahuna/public/js/components/gr-chips/gr-chips.js	Updates AI placeholder text to reflect that filters are now available.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  def search(params: SearchParams)(implicit ex: ExecutionContext, request: AuthenticatedRequest[AnyContent, Principal], logMarker: LogMarker = MarkerMap()): Future[SearchResults] = {
    val query: Query = queryBuilder.makeQuery(params.structuredQuery)

-    val uploadTimeFilter = filters.date("uploadTime", params.since, params.until)
-    val lastModTimeFilter = filters.date("lastModified", params.modifiedSince, params.modifiedUntil)
-    val takenTimeFilter = filters.date("metadata.dateTaken", params.takenSince, params.takenUntil)
-    // we only inject filters if there are actual date parameters
-    val dateFilterList = List(uploadTimeFilter, lastModTimeFilter, takenTimeFilter).flatten.toNel
-    val dateFilter = dateFilterList.map(dateFilters => filters.and(dateFilters.list.toList: _*))
-
-    val idsFilter = params.ids.map(filters.ids)
-    val labelFilter = params.labels.toNel.map(filters.terms("labels", _))
-    val metadataFilter = params.hasMetadata.map(metadataField).toNel.map(filters.exists)
-    val archivedFilter = params.archived.map(filters.existsOrMissing(editsField("archived"), _))
-    val hasExports = params.hasExports.map(filters.existsOrMissing("exports", _))
-    val hasIdentifier = params.hasIdentifier.map(idName => filters.exists(NonEmptyList(identifierField(idName))))
-    val missingIdentifier = params.missingIdentifier.map(idName => filters.missing(NonEmptyList(identifierField(idName))))
-    val uploadedByFilter = params.uploadedBy.map(uploadedBy => filters.terms("uploadedBy", NonEmptyList(uploadedBy)))
-    val simpleCostFilter = params.free.flatMap(free => if (free) searchFilters.freeFilter else searchFilters.nonFreeFilter)
-    val costFilter = params.payType match {
-      case Some(PayType.Free) => searchFilters.freeFilter
-      case Some(PayType.MaybeFree) => searchFilters.maybeFreeFilter
-      case Some(PayType.Pay) => searchFilters.nonFreeFilter
-      case _ => None
-    }
-
-    val printUsageFilter = params.printUsageFilters.map(searchFilters.printUsageFilters)
-
-    val hasRightsCategory = params.hasRightsCategory.filter(_ == true).map(_ => searchFilters.hasRightsCategoryFilter)
-
-    val validityFilter = params.valid.map(valid => if (valid) searchFilters.validFilter else searchFilters.invalidFilter)
-
-    val persistFilter = params.persisted map {
-      case true => searchFilters.persistedFilter
-      case false => searchFilters.nonPersistedFilter
-    }
-
-    val usageFilter: Iterable[Query] =
-      params.usageStatus.toNel.map(status => filters.terms("usagesStatus", status.map(_.toString))).toOption ++
-        params.usagePlatform.toNel.map(filters.terms("usagesPlatform", _)).toOption
-
-    val syndicationStatusFilter = params.syndicationStatus.map(status => syndicationFilter.statusFilter(status))
-
-    // Port of special case code in elastic1 sorts. Using the dateAddedToCollection sort implies an additional filter for reasons unknown
-    val dateAddedToCollectionFilter = {
-      params.orderBy match {
-        case Some("dateAddedToCollection") => {
-          val pathHierarchyOpt = params.structuredQuery.flatMap {
-            case Match(HierarchyField, Phrase(value)) => Some(value)
-            case _ => None
-          }.headOption
-
-          pathHierarchyOpt.map { pathHierarchy =>
-            termQuery("collections.pathHierarchy", pathHierarchy)
-          }
-        }
-        case _ => None
-      }
-    }
-
-    val filterOpt = (
-      metadataFilter.toOption.toList
-        ++ persistFilter
-        ++ labelFilter.toOption
-        ++ archivedFilter
-        ++ uploadedByFilter
-        ++ idsFilter
-        ++ validityFilter
-        ++ simpleCostFilter
-        ++ costFilter
-        ++ hasExports
-        ++ hasIdentifier
-        ++ missingIdentifier
-        ++ dateFilter.toOption
-        ++ usageFilter
-        ++ hasRightsCategory
-        ++ searchFilters.tierFilter(params.tier)
-        ++ syndicationStatusFilter
-        ++ dateAddedToCollectionFilter
-        ++ printUsageFilter
-      ).toNel.map(filter => filter.list.toList.reduceLeft(filters.and(_, _))).toOption
+    val filterOpt: Option[Query] = queryBuilder.buildFilterOpt(params, searchFilters, syndicationFilter)



+        ctrl.needsQuery = $stateParams.useAISearch && (
+          !hasPositiveAiTextQuery($stateParams.query) ||
+          hasSimilarAndPositiveAiTextQuery($stateParams.query)
+        );


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

joelochlann · 2026-06-05T13:52:50Z

This looks really nice!

A first thought on the UI: now that we have filters & chips, I didn't expect the search to be blank until I enter a query. I thought that I would be able to mess around with filters, seeing how it shapes my dataset and the number of hits etc, before I added in the AI search layer on top.

I feel that this would make it clearer what's actually happening with the pre-filtering, especially if we give the right information in the UI.

Even if we think we want it blank until it enters a query, I think there has to be some information about what's going on with the pre-filtering + knn, otherwise always seeing "200 results" no matter what I do with the filters (unless I get it below 200!) is confusing.

Here's a (fairly clumsy) attempt

x images matching filters => (of which) showing 200 most similar to AI search

paperboyo · 2026-06-05T14:06:26Z

🎉🎉🎉 👏

I didn't expect the search to be blank until I enter a query

+1 to not blank, yeah.

Agreed signalling why 200 good idea. Changing Sort to Relevance may or (more likely) may not be enough. Maybe just crude text?

Whatever we decide, let’s not mimic tickers looks as these are clickable (to filter)?

Unrelatedly, I would also split More like this from the Use AI checkbox completely. Currently, this deadend is possible:

joelochlann · 2026-06-05T16:33:36Z

Yeah it's tricky. The problem with "200 relevant" is we don't know that all 200 are relevant! They're just the best 200 we could manage. In fact it's very rare they will all be relevant.

paperboyo · 2026-06-05T18:01:52Z

Yeah it's tricky. The problem with "200 relevant" is we don't know that all 200 are relevant! They're just the best 200 we could manage. In fact it's very rare they will all be relevant.

Yes, of course. I struggled the come up with a word (just like for “Sort” order: I think they should be the same word). But what matters here more than the fact that the word isn’t perfect is that it’s there at all: it signals a) it’s a bound subset b) of a corpus you can actively shape, c) has to do with this AI thing and d) is about ordering.

Especially b) (of 2,129) is particularly revealing about the very UX issue we discussed above related to post-filtering behaving counter-intuitively (versus “normal” filtering).

And 200 AI-similarity-ordered of 2,129 was too long ;-)

[EDIT: sorry deleted doubled comment]

gu-prout · 2026-06-10T15:05:42Z

Seen on auth, image-loader, metadata-editor, cropper, collections, kahuna (merged by @ellenmuller 9 minutes and 38 seconds ago) Please check your changes!

gu-prout · 2026-06-10T15:05:49Z

Seen on thrall, usage, media-api (merged by @ellenmuller 9 minutes and 47 seconds ago) Please check your changes!

gu-prout · 2026-06-10T15:05:54Z

Seen on leases (merged by @ellenmuller 9 minutes and 51 seconds ago) Please check your changes!

ellenmuller added 4 commits June 1, 2026 15:59

split out query from chips and only embed query

84fbfc1

make similar part of model and split out filteropt

f1f589b

add chip filter opt

98a1ee7

wire filteropt through to elasticsearch query

3f32dec

ellenmuller added the feature Departmental tracking: work on a new feature label Jun 2, 2026

ellenmuller added 9 commits June 3, 2026 14:09

require a semantic text query in semantic mode

efbeb33

tidy up extracting semantic text query

6ef668b

add case where there's a similar search + semantic text query

21cbcf9

filter at knn stage to maximise knn results

3246742

don't collapse filters when in ai mode

50db9a5

refactor the filter checking and move into lib

658eaab

tidy up remove comments etc

dd7faf4

Merge branch 'main' of github.com:guardian/grid into em-semantic-sear…

434c4aa

…ch-filtering

add extra comment

eb27ea3

ellenmuller changed the title ~~Em semantic search filtering~~ Add chips and filtering to 'Use AI Search` Jun 5, 2026

ellenmuller changed the title ~~Add chips and filtering to 'Use AI Search`~~ Add chips and filtering to 'Use AI Search' Jun 5, 2026

ellenmuller changed the title ~~Add chips and filtering to 'Use AI Search'~~ Add chips and filtering to AI Search Jun 5, 2026

ellenmuller marked this pull request as ready for review June 5, 2026 10:33

ellenmuller requested a review from a team as a code owner June 5, 2026 10:33

ellenmuller requested review from Copilot and joelochlann June 5, 2026 10:33

Copilot started reviewing on behalf of ellenmuller June 5, 2026 10:33 View session

Copilot AI reviewed Jun 5, 2026

View reviewed changes

ellenmuller and others added 2 commits June 5, 2026 12:23

Apply suggestions from code review

ac30ea9

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

tidy up

ad855fc

gu-prout Bot added Seen-on-usage Seen-on-media-api Seen-on-thrall and removed Pending-on-media-api Pending-on-usage Pending-on-thrall labels Jun 10, 2026

gu-prout Bot added Seen-on-leases and removed Pending-on-leases labels Jun 10, 2026

paperboyo added a commit that referenced this pull request Jun 15, 2026

Doc updates (and amendments to media-api plan post #4752)

cf701ad

joelochlann mentioned this pull request Jul 9, 2026

Display filteredPool when AI search query is absent #4791

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add chips and filtering to AI Search#4752

Add chips and filtering to AI Search#4752
ellenmuller merged 21 commits into
mainfrom
em-semantic-search-filtering

ellenmuller commented Jun 2, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joelochlann commented Jun 5, 2026 •

edited

Loading

Uh oh!

paperboyo commented Jun 5, 2026 •

edited

Loading

Uh oh!

joelochlann commented Jun 5, 2026

Uh oh!

paperboyo commented Jun 5, 2026 •

edited

Loading

Uh oh!

gu-prout Bot commented Jun 10, 2026

Uh oh!

gu-prout Bot commented Jun 10, 2026

Uh oh!

gu-prout Bot commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ellenmuller commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this change?

How should a reviewer test this change?

How can success be measured?

Who should look at this?

Tested? Documented?

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy build 14540 of media-service::grid::all to TEST

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joelochlann commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

paperboyo commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joelochlann commented Jun 5, 2026

Uh oh!

paperboyo commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gu-prout Bot commented Jun 10, 2026

Uh oh!

gu-prout Bot commented Jun 10, 2026

Uh oh!

gu-prout Bot commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ellenmuller commented Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading

Deploy build 14540 of `media-service::grid::all` to TEST

joelochlann commented Jun 5, 2026 •

edited

Loading

paperboyo commented Jun 5, 2026 •

edited

Loading

paperboyo commented Jun 5, 2026 •

edited

Loading