New gateway exporter #6393

jedevc · 2025-12-02T17:14:23Z

Follow-ups:

Secret passing. This PR is already going to be large, and adding support for secrets also intersects with secrets for cache exporter backends, so it's not super trivial to just add on.

jedevc · 2025-12-02T17:30:05Z

exporter/exporter_test.go

+	var foundFiles []string
+	var foundDescs []ocispecs.Descriptor
+	export := func(ctx context.Context, c gateway.Client, conn *grpc.ClientConn, _ exptypes.ExporterTarget, result *gateway.Result) error {
+		entries, err := result.Ref.ReadDir(ctx, gateway.ReadDirRequest{Path: "/"})
+		if err != nil {
+			return err
+		}
+		for _, entry := range entries {
+			foundFiles = append(foundFiles, entry.Path)
+		}
+
+		store := contentproxy.NewContentStore(conn)
+		descs, err := result.Ref.GetRemote(ctx)
+		if err != nil {
+			return err
+		}
+		foundDescs = filterAvailableDescriptors(ctx, store, descs)
+
+		return nil
+	}
+
+	_, err = c.BuildExport(sb.Context(), client.SolveOpt{}, "", frontend, export, nil)
+	require.NoError(t, err)


This feels like a fun interface to me - allowing the client to easily do its own local export. This actually works entirely without any server-side changes - it's just calling the exporter function right after a build, using the same gateway.

We have access to the entire buildkit content store (so there's no isolation here, but this is client-trusted code).

jedevc · 2025-12-02T17:36:02Z

exporter/exporter_test.go

+func buildSampleExporter(ctx context.Context, c *client.Client, dest string) error {
+	gatewayDir, err := fsutil.NewFS(integration.BuildkitSourcePath)


I can't work out a good test strategy that doesn't involve doing this 😢

To actually test the gateway exporter codepath + the isolation it provides, we actually need to build one so we can test it. It feels weird to do this in a test (as @tonistiigi points out in #3633 (comment)). But not sure what the alternative is 🤷 Some tests in exporter_test.go might not need it, but could otherwise just build it in bake, and connect it in?

jedevc · 2025-12-02T17:37:39Z

session/filesync/proxy.go

I seem to remember writing this code before - but can't remember when.

@tonistiigi is there another implementation of this that you're aware of?

jedevc · 2025-12-02T17:43:16Z

exporter/gateway/export.go

+	return exporter.NewConfig()
+}
+
+func (e *gatewayExporterInstance) Export(ctx context.Context, llbBridge frontend.FrontendLLBBridge, exec executor.Executor, src *exporter.Source, inlineCache exptypes.InlineCache, sessionID string) (_ map[string]string, descref exporter.DescriptorReference, err error) {


Should the gateway exporter allow returning an arbitrary exporter response map? I haven't added that yet.

Seems like it would be nice to allow it:

Could either reuse the existing Return gateway API

New exporter response field

Use the result metadata somehow

Or create a new API (hoping to avoid this)

jedevc · 2025-12-02T17:45:28Z

api/services/control/control.proto

+	// Target indicates the target type of the exporter
+	ExporterTarget Target = 3;


This is required to support the new file/dir keys (pointing out, since I didn't realize this in #3037 (comment)).

Without something like this, the exporter can't actually know what type was attached (since we strip out the attribute itself to avoid leaking it to the server).

jedevc · 2025-12-05T15:52:55Z

Getting really close now, the only things I need to get done are:

Tests 🎉
Some refactorings of code that got that little bit more complicated
Authoring a real commit history 👀

Edit: done! This PR is now ready for review.

Signed-off-by: Justin Chadwell <[email protected]>

This allows the client to be explicit as to what client-side inputs the client has provided (instead of implying this through the use of exporter attributes). Signed-off-by: Justin Chadwell <[email protected]>

Additionally, simplify some of the result loading/stashing logic into helpers. Signed-off-by: Justin Chadwell <[email protected]>

We'll need this in the exporter package, so break it up a little bit so it can be used there later. Signed-off-by: Justin Chadwell <[email protected]>

This wasn't previously used *at all*. However, now, we need to be able to do this, since exporters will speak the gateway api, and we want to be able to access provenance attestations. Signed-off-by: Justin Chadwell <[email protected]>

Signed-off-by: Justin Chadwell <[email protected]>

WYGIN · 2025-12-09T15:46:13Z

Will it be a breaking change? How can I figure out if a buildkit demon support BuildFromEnvironment or ExportFromEnvironment supported As a gateway frontend?

jedevc · 2025-12-09T16:55:16Z

This will not be a breaking change.

ExportFromEnvironment will only be callable when invoked from --output type=gateway,source=<image>. We should probably add a sanity check that it's running in the right environment though - so that if you try and put a frontend image instead of a exporter image, you get a sane error.

github-actions bot assigned jedevc Dec 2, 2025

github-actions bot added area/testing area/hack building buildkit itself area/client area/frontend area/util area/api area/exporter area/session area/solver area/worker area/buildctl labels Dec 2, 2025

jedevc force-pushed the gateway-exporter branch from 92b531c to 040ea7f Compare December 2, 2025 17:24

jedevc commented Dec 2, 2025

View reviewed changes

jedevc force-pushed the gateway-exporter branch from 040ea7f to c188224 Compare December 4, 2025 16:56

github-actions bot added the area/dockerfile label Dec 4, 2025

jedevc force-pushed the gateway-exporter branch from c188224 to af8ac15 Compare December 5, 2025 10:11

github-actions bot added area/dependencies Pull requests that update a dependency file area/ci area/docs labels Dec 5, 2025

jedevc force-pushed the gateway-exporter branch 2 times, most recently from 45e4d38 to a930686 Compare December 5, 2025 10:50

github-actions bot removed area/hack building buildkit itself area/dependencies Pull requests that update a dependency file area/ci area/docs labels Dec 5, 2025

jedevc force-pushed the gateway-exporter branch from a930686 to 9a0d0c7 Compare December 5, 2025 11:52

github-actions bot added area/hack building buildkit itself area/ci labels Dec 5, 2025

jedevc force-pushed the gateway-exporter branch from 9a0d0c7 to af16b82 Compare December 5, 2025 15:38

jedevc added 8 commits December 8, 2025 15:38

chore: add gitattributes for generated protobuf

116e6b1

Signed-off-by: Justin Chadwell <[email protected]>

control: add new exporter target api

22f3d7e

This allows the client to be explicit as to what client-side inputs the client has provided (instead of implying this through the use of exporter attributes). Signed-off-by: Justin Chadwell <[email protected]>

gateway: add GetReturn and GetRemote apis

4b294b8

Additionally, simplify some of the result loading/stashing logic into helpers. Signed-off-by: Justin Chadwell <[email protected]>

gateway: make LLBBridgeForwarder more reusable

893931f

We'll need this in the exporter package, so break it up a little bit so it can be used there later. Signed-off-by: Justin Chadwell <[email protected]>

exporter: add new gateway exporter

19725fd

Signed-off-by: Justin Chadwell <[email protected]>

test: mount buildkit code to test container

dcbb8ac

Signed-off-by: Justin Chadwell <[email protected]>

test: add gateway exporter tests

9e30a4c

Signed-off-by: Justin Chadwell <[email protected]>

jedevc force-pushed the gateway-exporter branch from af16b82 to 9e30a4c Compare December 8, 2025 17:01

jedevc marked this pull request as ready for review December 8, 2025 17:01

jedevc requested a review from tonistiigi December 8, 2025 17:02

jedevc mentioned this pull request Dec 8, 2025

Exporter plugin (like frontend but for exporters) #3037

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

New gateway exporter #6393

New gateway exporter #6393

jedevc commented Dec 2, 2025 •

edited

Loading

Uh oh!

jedevc Dec 2, 2025

Uh oh!

jedevc Dec 2, 2025

Uh oh!

jedevc Dec 2, 2025

Uh oh!

jedevc Dec 2, 2025 •

edited

Loading

Uh oh!

jedevc Dec 2, 2025 •

edited

Loading

Uh oh!

jedevc commented Dec 5, 2025 •

edited

Loading

Uh oh!

WYGIN commented Dec 9, 2025

Uh oh!

jedevc commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		func buildSampleExporter(ctx context.Context, c *client.Client, dest string) error {
		gatewayDir, err := fsutil.NewFS(integration.BuildkitSourcePath)

		// Target indicates the target type of the exporter
		ExporterTarget Target = 3;

New gateway exporter #6393

Are you sure you want to change the base?

New gateway exporter #6393

Conversation

jedevc commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jedevc Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

jedevc Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

jedevc Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

jedevc Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jedevc Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jedevc commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WYGIN commented Dec 9, 2025

Uh oh!

jedevc commented Dec 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jedevc commented Dec 2, 2025 •

edited

Loading

jedevc Dec 2, 2025 •

edited

Loading

jedevc Dec 2, 2025 •

edited

Loading

jedevc commented Dec 5, 2025 •

edited

Loading