feat/gql-proxy: new approach for GraphQL local scale out #3074

vrajashkr · 2025-04-06T19:43:55Z

What type of PR is this?
feature

Which issue does this PR fix:
Towards #2434

What does this PR do / Why do we need it:
Previously, only dist-spec APIs were supported for scale-out as in a shared storage environment, the metadata was shared and any instance could correctly respond to the GQL queries as all the data is available.

In a local scale-out cluster deployment, the metadata store, in addition to the file storage is isolated to each member in the cluster. Due to this, there is a need to proxy the GQL queries as well for UI and client requests to work as expected.

This change introduces a new GQL proxy + a generic fan-out handler for GQL requests.

Testing done on this change:
Just manual testing with local setup for now. Real testing TODO.

Will this break upgrades or downgrades?
No

Does this PR introduce any user-facing change?:
TODO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

vrajashkr · 2025-04-06T19:49:24Z

A previous attempt at this was made in #2588

How is this different?

Instead of a dedicated handler for each operation, in this PR, an approach is explored where there are generic and specific handlers. (Refer to [Feat]: Scale-out cluster support for independent per-instance storage deployments #2434 (comment))
Generic handlers cater to common use cases such as fan-out proxy for GQL or single target Proxying
Specific handlers cater to specific use cases which do not align with a generic handler
GQL schema-merging was done manually in the previous PR, however, in this PR, a generic approach has been taken to dynamically merge the responses by treating them as maps instead of dedicated structs. This allows the code to be reusable across all types of handlers and reduces the amount of code to be maintained.

Note: Pagination is still pending
Note: The code itself is just in a Proof of concept state. Quite a bit of clean up is needed.

vrajashkr · 2025-04-06T20:00:32Z

pkg/extensions/search/gql_proxy/gql_proxy.go

+	log log.Logger,
+	gqlSchema *ast.Schema,
+) func(handler http.Handler) http.Handler {
+	proxyFunctionalityMap := map[string]GqlScaleOutHandlerFunc{


@rchincha this map statically stores which GQL operation needs which kind of handler.

There will be a couple of generic handlers such as fanout and some specific handlers if any of the operations need custom behavior.

What do you think about this approach?

It's better than last time since we don't need to maintain a separate handler for each operation type, but I'm open to more ideas on making this better.

vrajashkr · 2025-04-06T20:02:04Z

pkg/extensions/search/gql_proxy/handler_utils.go

+	}
+}
+
+func deepMergeMaps(a, b map[string]any) map[string]any {


@rchincha this is the new approach for aggregating the data. Since the response is a JSON, we can aggregate the data as a map type with individual logic for the embedded types - nested map, numeric types, and arrays.

This should be common across all handlers and may change when pagination comes into picture.

This sounds good. Only concern is whether we can validate schema so that we never emit garbage out.

pkg/extensions/search/search_test.go

rchincha · 2025-04-15T15:53:45Z

examples/scale-out-cluster-local/config-cluster-member0.json

+{
+  "distSpecVersion": "1.1.0",
+  "storage": {
+    "rootDirectory": "./workspace/zot/data/mem1",


We have to figure out a scheme to append a member path.
Would like to have a single zot configuration that folks don't have to tweak.

I agree. The reason for this is mostly because I was starting 2 binaries on the same host for development (so I had to change the path and port). For an actual deployment, the config files would be identical.

Signed-off-by: Vishwas Rajashekar <[email protected]>

vrajashkr · 2025-04-28T18:24:39Z

Latest analysis of GQL queries:

CVEListForImage

Proxy-once to target
uses "repository:tag" or "repository@digest"

CVEDiffListForImages

image and compared image names in "repository:tag" or "repository@digest" format.
challenging as the images may be on different servers.

ImageListForCVE

fan-out to all members

ImageListWithCVEFixed

fan-out to all members

ImageListForDigest

fan-out to all members

RepoListWithNewestImage

fan-out to all members

ImageList

use the repo param to proxy once to the target server.

ExpandedRepoInfo

use the repo param to proxy once to the target server.

GlobalSearch

fan-out to all members
future optimization: if the repo names are known, these don't need to be fanned-out.

DerivedImageList

uses "repository:tag" format.
however, expected output is all images using the given one so a fan-out is needed.

BaseImageList

image uses "repository:tag" format.
as expectation is to get all images, a fan-out is needed.

Image

image uses "repository:tag" format
use repo value to proxy once to target

Referrers

has a repo as an argument.
use repo value to proxy once to target

StarredRepos

fan-out to all members

BookmarkedRepos

fan-out to all members

Signed-off-by: Vishwas Rajashekar <[email protected]>

…poInfo Signed-off-by: Vishwas Rajashekar <[email protected]>

codecov · 2025-04-29T15:26:26Z

Codecov Report

Attention: Patch coverage is 69.03915% with 87 lines in your changes missing coverage. Please review.

Project coverage is 90.64%. Comparing base (293f424) to head (c991fe4).

Files with missing lines	Patch %	Lines
pkg/extensions/search/gql_proxy/gql_proxy.go	64.47%	23 Missing and 4 partials ⚠️
...search/gql_proxy/generic_proxy_once_gql_handler.go	40.00%	17 Missing and 4 partials ⚠️
...ns/search/gql_proxy/generic_fan_out_gql_handler.go	50.00%	12 Missing and 3 partials ⚠️
pkg/extensions/search/gql_proxy/handler_utils.go	73.46%	11 Missing and 2 partials ⚠️
pkg/api/cluster_proxy.go	79.24%	9 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3074      +/-   ##
==========================================
- Coverage   90.79%   90.64%   -0.15%     
==========================================
  Files         172      177       +5     
  Lines       32385    32584     +199     
==========================================
+ Hits        29404    29536     +132     
- Misses       2242     2298      +56     
- Partials      739      750      +11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rchincha · 2025-04-29T19:47:33Z

pkg/extensions/search/gql_proxy/generic_fan_out_gql_handler.go

+
+	for _, targetMember := range config.Cluster.Members {
+		proxyResponse, err := proxy.ProxyHTTPRequest(request.Context(), request, targetMember, config)
+		if err != nil {


Do we want to return failure even if just one member fails?
I think if one member responds, we should return that and swallow errors maybe with some indicator somewhere. Maybe logs? HTTP status - 206 Partial Content could work also since this is our own API.

That sounds like a good idea. One thought that I had in mind is that instead of swallowing the error, perhaps we could append an error to the Errors list key in the GQL response and send it to the client so there is awareness of some error in the system.

The client can choose to ignore the error and use the valid data in the response, or ideally, show both the valid data as well as indicate that there were some errors in processing. With this approach status 206 could be the return status as you've suggested.

What do you think?

vrajashkr commented Apr 6, 2025

View reviewed changes

pkg/extensions/search/search_test.go Show resolved Hide resolved

vrajashkr force-pushed the feat/gql-scale-out-v2 branch from a5dbb43 to ac85801 Compare April 6, 2025 20:17

rchincha reviewed Apr 15, 2025

View reviewed changes

feat(scale-out): support for GQL scale-out with local metadata

dd17e77

Signed-off-by: Vishwas Rajashekar <[email protected]>

vrajashkr force-pushed the feat/gql-scale-out-v2 branch from ac85801 to 13f6084 Compare April 28, 2025 19:58

vrajashkr added 3 commits April 28, 2025 20:02

feat(scale-out): add support for ImageList GQL

8aca3c6

Signed-off-by: Vishwas Rajashekar <[email protected]>

feat(scale-out): add support for ExtendedRepoInfo GQL

befbf5a

Signed-off-by: Vishwas Rajashekar <[email protected]>

feat(scale-out): refactor test setup + additional test for ExpandedRe…

c991fe4

…poInfo Signed-off-by: Vishwas Rajashekar <[email protected]>

vrajashkr force-pushed the feat/gql-scale-out-v2 branch from 13f6084 to c991fe4 Compare April 29, 2025 15:07

rchincha reviewed Apr 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat/gql-proxy: new approach for GraphQL local scale out #3074

feat/gql-proxy: new approach for GraphQL local scale out #3074

vrajashkr commented Apr 6, 2025

vrajashkr commented Apr 6, 2025 •

edited

Loading

vrajashkr Apr 6, 2025

vrajashkr Apr 6, 2025

rchincha Apr 29, 2025

rchincha Apr 15, 2025

vrajashkr Apr 17, 2025 •

edited

Loading

vrajashkr commented Apr 28, 2025

codecov bot commented Apr 29, 2025

rchincha Apr 29, 2025

vrajashkr May 1, 2025

feat/gql-proxy: new approach for GraphQL local scale out #3074

Are you sure you want to change the base?

feat/gql-proxy: new approach for GraphQL local scale out #3074

Conversation

vrajashkr commented Apr 6, 2025

vrajashkr commented Apr 6, 2025 • edited Loading

vrajashkr Apr 6, 2025

Choose a reason for hiding this comment

vrajashkr Apr 6, 2025

Choose a reason for hiding this comment

rchincha Apr 29, 2025

Choose a reason for hiding this comment

rchincha Apr 15, 2025

Choose a reason for hiding this comment

vrajashkr Apr 17, 2025 • edited Loading

Choose a reason for hiding this comment

vrajashkr commented Apr 28, 2025

codecov bot commented Apr 29, 2025

Codecov Report

rchincha Apr 29, 2025

Choose a reason for hiding this comment

vrajashkr May 1, 2025

Choose a reason for hiding this comment

vrajashkr commented Apr 6, 2025 •

edited

Loading

vrajashkr Apr 17, 2025 •

edited

Loading