Refactor NewImageSource to add a manifest type abstraction #5743

aaronlehmann · 2024-09-17T23:28:58Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Currently, NewImageSource creates a Docker schema2 manifest and an OCI manifest at the same time. This precludes functionality that isn't supported by both manifest types, for example zstd compression. Refactoring this to create only the desired manifest type solves this and also cleans up the code by separating manifest-type-specific code into distinct implementations of a "manifest builder".

How to verify it

Existing tests should suffice since this does not change any semantics, it only cleans up the implementation.

Which issue(s) this PR fixes:

None

Special notes for your reviewer:

See discussion in #5452.

Does this PR introduce a user-facing change?

None

aaronlehmann · 2024-09-19T14:39:20Z

@mtrmac: PTAL - thanks!

aaronlehmann · 2024-09-24T17:02:57Z

@rhatdan @mtrmac: Would it be possible to get some eyes on this when you have a moment? I was asked to do this to unblock #5452, so I would really appreciate if we could avoid letting it sit too long.

aaronlehmann · 2024-10-08T04:12:31Z

Any possibility of moving this forward? It's a bit frustrating that I was asked to come back and do this refactor, but the PR is not getting reviewed.

TomSweeneyRedHat · 2024-11-01T22:11:16Z

@mtrmac have you had a chance to look at this?

TomSweeneyRedHat · 2024-11-01T22:12:20Z

@aaronlehmann sorry for the delay. Many of the maintainers are Red Hat based, and we've all been under a big crunch the past few weeks.

mtrmac · 2024-11-04T16:19:49Z

@Honny1 Do you happen to have the time to take a look?

github-actions · 2024-12-05T00:14:26Z

A friendly reminder that this PR had no activity for 30 days.

aaronlehmann · 2025-02-27T13:37:54Z

@TomSweeneyRedHat @mtrmac @Honny1: Would any of you be able to look at this? I've been trying to get this through for a very long time.

mtrmac · 2025-02-27T16:20:36Z

@flouthoc do you happen to have time for this?

flouthoc · 2025-02-28T18:13:26Z

I will review this thanks.

flouthoc · 2025-02-28T18:16:41Z

@aaronlehmann Could you please rebase this first, just to be sure if every thing is good.

aaronlehmann · 2025-02-28T21:24:45Z

Rebased.

aaronlehmann · 2025-03-10T17:20:22Z

Hi @flouthoc, have you had a chance to look at this yet?

flouthoc

There is some duplication but PR LGTM. Could you please rebase again.

aaronlehmann · 2025-03-18T17:47:40Z

Rebased again.

aaronlehmann · 2025-03-19T16:03:06Z

@flouthoc: Hoping I can get this merged before it needs yet another rebase 😄

flouthoc

LGTM @nalind any comments ?

image.go

nalind · 2025-03-19T18:31:50Z

image.go

@@ -720,187 +1027,19 @@ func (i *containerImageRef) NewImageSource(_ context.Context, _ *types.SystemCon
 		}
 		// Add a note in the manifest about the layer.  The blobs are identified by their possibly-
 		// compressed blob digests.


This looks like a comment that addLayer() should have in it, or as part of its godoc.

image.go

aaronlehmann · 2025-03-19T19:16:58Z

@nalind: Thanks for the review - updated based on your feedback.

aaronlehmann · 2025-03-21T21:40:19Z

@nalind @flouthoc: Any further comments?

aaronlehmann · 2025-03-25T17:13:33Z

@nalind @flouthoc: Can we please move ahead with merging this? Thanks!

flouthoc · 2025-03-25T21:00:11Z

PR LGTM I am waiting for @nalind to ack on his last remarks.

aaronlehmann · 2025-03-31T15:34:57Z

@nalind: Sorry to be so persistent here, but I'd really like to get this merged, since it has been dragging on for so long, and it's hard to keep coming back to this after long pauses.

nalind

Minor nits, otherwise LGTM.

internal/config/override.go

image.go

Currently, NewImageSource creates a Docker schema2 manifest and an OCI manifest at the same time. This precludes functionality that isn't supported by both manifest types, for example zstd compression. Refactoring this to create only the desired manifest type solves this and also cleans up the code by separating manifest-type-specific code into distinct implementations of a "manifest builder". See discussion in containers#5452. Signed-off-by: Aaron Lehmann <[email protected]>

Signed-off-by: Aaron Lehmann <[email protected]>

aaronlehmann · 2025-04-10T20:02:12Z

Made the changes, thanks!

aaronlehmann · 2025-05-07T18:53:30Z

@nalind: Just checking on this again...

aaronlehmann · 2025-05-20T18:15:25Z

@nalind @flouthoc: I'm grateful for your help moving this forward, but I have to say this has been a very frustrating experience. I'm making one last request for a final review so I can get this merged.

mtrmac · 2025-05-20T18:22:12Z

I’m afraid I can’t spend much time on this now, and this is probably locally not helpful: I happened to notice parallel #6159 .

flouthoc · 2025-05-20T18:24:33Z

@aaronlehmann Apologies for the delay. The PR LGTM, I am rebasing this and getting this merged. This already has 2 LGTM.

flouthoc

LGTM
/lgtm
/approve
/hold

openshift-ci · 2025-05-20T18:25:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aaronlehmann, flouthoc, nalind

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nalind]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

packit-as-a-service · 2025-05-20T18:27:11Z

Ephemeral COPR build failed. @containers/packit-build please check.

aaronlehmann · 2025-05-20T22:19:48Z

Thanks so much @flouthoc, I really appreciate it. This refactor was motivated by #5452, which now becomes a trivial change to the OCI manifest code path. I've rebased that PR - if you could take a look that would be awesome!

openshift-ci bot added the kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. label Sep 17, 2024

aaronlehmann mentioned this pull request Sep 17, 2024

Support zstd compression in image commit #5452

Merged

aaronlehmann force-pushed the refactor-manifest-types branch 2 times, most recently from 9f1c553 to 81fea1d Compare September 17, 2024 23:39

github-actions bot added the stale-pr label Dec 5, 2024

github-actions bot removed the stale-pr label Feb 28, 2025

flouthoc self-assigned this Feb 28, 2025

aaronlehmann force-pushed the refactor-manifest-types branch from 81fea1d to b962939 Compare February 28, 2025 21:24

flouthoc added the No New Tests Allow PR to proceed without adding regression tests label Feb 28, 2025

flouthoc reviewed Mar 18, 2025

View reviewed changes

aaronlehmann force-pushed the refactor-manifest-types branch from b962939 to 72ea5a8 Compare March 18, 2025 17:47

flouthoc approved these changes Mar 19, 2025

View reviewed changes

nalind reviewed Mar 19, 2025

View reviewed changes

image.go Outdated Show resolved Hide resolved

nalind reviewed Mar 19, 2025

View reviewed changes

image.go Outdated Show resolved Hide resolved

nalind reviewed Mar 19, 2025

View reviewed changes

image.go Outdated Show resolved Hide resolved

nalind reviewed Mar 19, 2025

View reviewed changes

image.go Outdated Show resolved Hide resolved

aaronlehmann force-pushed the refactor-manifest-types branch from 5686f79 to 7fbdf26 Compare March 19, 2025 19:48

aaronlehmann force-pushed the refactor-manifest-types branch from 7fbdf26 to 8008e9c Compare March 31, 2025 15:37

nalind approved these changes Apr 10, 2025

View reviewed changes

internal/config/override.go Outdated Show resolved Hide resolved

image.go Outdated Show resolved Hide resolved

image.go Outdated Show resolved Hide resolved

openshift-ci bot added the approved label Apr 10, 2025

aaronlehmann added 3 commits April 10, 2025 20:00

Review feedback

ec10f73

Signed-off-by: Aaron Lehmann <[email protected]>

Review feedback, round 2

71859d1

Signed-off-by: Aaron Lehmann <[email protected]>

aaronlehmann force-pushed the refactor-manifest-types branch from 8008e9c to 71859d1 Compare April 10, 2025 20:02

mtrmac mentioned this pull request May 20, 2025

breakup containerImageRef.NewImageSource #6159

Open

Merge branch 'main' into refactor-manifest-types

23c787c

flouthoc approved these changes May 20, 2025

View reviewed changes

openshift-ci bot added do-not-merge/hold lgtm labels May 20, 2025

flouthoc merged commit cf4635e into containers:main May 20, 2025
31 of 37 checks passed

flouthoc removed the do-not-merge/hold label May 20, 2025

Refactor NewImageSource to add a manifest type abstraction #5743

Refactor NewImageSource to add a manifest type abstraction #5743

Uh oh!

Conversation

aaronlehmann commented Sep 17, 2024

What type of PR is this?

What this PR does / why we need it:

How to verify it

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

aaronlehmann commented Sep 19, 2024

Uh oh!

aaronlehmann commented Sep 24, 2024

Uh oh!

aaronlehmann commented Oct 8, 2024

Uh oh!

TomSweeneyRedHat commented Nov 1, 2024

Uh oh!

TomSweeneyRedHat commented Nov 1, 2024

Uh oh!

mtrmac commented Nov 4, 2024

Uh oh!

github-actions bot commented Dec 5, 2024

Uh oh!

aaronlehmann commented Feb 27, 2025

Uh oh!

mtrmac commented Feb 27, 2025

Uh oh!

flouthoc commented Feb 28, 2025

Uh oh!

flouthoc commented Feb 28, 2025

Uh oh!

aaronlehmann commented Feb 28, 2025

Uh oh!

aaronlehmann commented Mar 10, 2025

Uh oh!

flouthoc left a comment

Choose a reason for hiding this comment

Uh oh!

aaronlehmann commented Mar 18, 2025

Uh oh!

aaronlehmann commented Mar 19, 2025

Uh oh!

flouthoc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nalind Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aaronlehmann commented Mar 19, 2025

Uh oh!

aaronlehmann commented Mar 21, 2025

Uh oh!

aaronlehmann commented Mar 25, 2025

Uh oh!

flouthoc commented Mar 25, 2025

Uh oh!

aaronlehmann commented Mar 31, 2025

Uh oh!

nalind left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronlehmann commented Apr 10, 2025

Uh oh!

aaronlehmann commented May 7, 2025

Uh oh!

aaronlehmann commented May 20, 2025

Uh oh!

mtrmac commented May 20, 2025

Uh oh!

flouthoc commented May 20, 2025