API: copy object will clone object metadata when possible #9500

nopcoder · 2025-09-09T12:23:01Z

Create a shallow copy (more efficient) of an object when requested.

This operation is limited to objects within the same repository and branch, subject to a configurable grace time.
When copying an object, the code will verify that the object’s creation time and the storage’s last update time fall within the grace time limit. If they don't, we use the standard copy behavior.

Closes #9499

github-actions · 2025-09-09T12:23:41Z

📚 Documentation preview at https://pr-9500.docs-lakefs-preview.io/

(Updated: 9/18/2025, 12:21:17 PM - Commit: 1594d8e)

api/swagger.yml

pkg/catalog/catalog.go

guy-har

LGTM
IMO the documentation should be clear what this does and clear that there is some risk that the caller should consider

api/swagger.yml

pkg/api/controller.go

pkg/catalog/catalog_test.go

Make a shallow copy of an object, copy metadata only. Limited to the same repository and branch. Experimental.

Co-authored-by: guy-har <[email protected]>

This reverts commit 5184f17.

This reverts commit 275dc9b.

This reverts commit 42c69c3.

nopcoder · 2025-09-18T13:04:29Z

@guy-har @itaiad200 re-implement the solution to enable physical copy as fallback to prefered clone.
limited clone just to same repository, branch and no custom user-metadata.

adding the underlying object time check in case of copy of copy.

we now check grace time using stat, so we get a different error from s3 when object not found

nopcoder · 2025-09-25T06:22:39Z

@guy-har @itaiad200 ready to review; had to fix some tests and added support to update metadata.

pkg/config/defaults.go

pkg/config/config.go

itaiad200 · 2025-09-25T10:58:13Z

pkg/catalog/catalog.go

-	} else {
-		dstEntry.Metadata = srcEntry.Metadata


Why did you remove this?

we don't like this code.
already copied the src entry into dst entry.
update dst metadata only if replaceSrcMetadata was true.

pkg/catalog/catalog.go

itaiad200 · 2025-09-25T11:09:09Z

pkg/catalog/catalog.go

+	if err != nil {
+		return nil, err
+	}
+	if !props.LastModified.IsZero() && time.Since(props.LastModified) > c.CloneGracePeriod {


When will props.LastModified.IsZero()==true?
Shouldn't we fail for that too?

My assumption was that no last modified property means that the underlying storage has no support, meaning that we can't use gc on them - like in memory block adapter.

itaiad200 · 2025-09-25T11:10:41Z

clients/python-wrapper/tests/integration/test_object.py

    assert copy_stat.mtime >= obj_stat.mtime
    assert copy_stat.size_bytes == obj_stat.size_bytes
    assert copy_stat.checksum == obj_stat.checksum
+    # do not check physical_address, as it can be a clone


I don't like can be a clone. The test should either always clone or always copy and we should assert on that

itaiad200 · 2025-09-25T11:11:46Z

pkg/api/controller_test.go

 		verifyResponseOK(t, copyResp, err)

-		// Verify the creation path, date and physical address are different
+		// Verify the creation path, date and physical address are the same


Date should be different

date can be the same or greater like copy (no change)

itaiad200 · 2025-09-25T11:12:36Z

pkg/api/controller_test.go

 	})

-	t.Run("committed", func(t *testing.T) {
+	t.Run("committed_clone", func(t *testing.T) {


Isn't clone just for when the source is in staging?

no - we said that even if we test, there can be a commit in parallel so you may clone a staged object. as long as the grace period is ok; it should be the same.

itaiad200 · 2025-09-25T11:14:49Z

pkg/gateway/operations/putobject.go

+		if errors.Is(err, block.ErrDataNotFound) || errors.Is(err, graveler.ErrNotFound) {
+			_ = o.EncodeError(w, req, err, gatewayErrors.Codes.ToAPIErr(gatewayErrors.ErrNoSuchKey))
+			return
+		}


Why is this part of the PR? Not sure how this is related

When we clone, we stat the source object - head request returns different error than copy api. we need to align that when the block adapter fail to find the object we return the right error in the gateway.

Co-authored-by: itaiad200 <[email protected]>

itaiad200

Approved. Still wondering about the branch limitation though

itaiad200 · 2025-09-27T12:18:36Z

pkg/catalog/catalog.go

+	if srcRepo.Name != destRepository {
+		return nil, fmt.Errorf("%w: not on the same repository", graveler.ErrCannotClone)
+	}
+	if srcRef != destBranch {


Comment still stands

nopcoder

Updated the code to support clone between branches

nopcoder · 2025-09-27T13:25:12Z

pkg/catalog/catalog.go

+	if srcRepo.Name != destRepository {
+		return nil, fmt.Errorf("%w: not on the same repository", graveler.ErrCannotClone)
+	}
+	if srcRef != destBranch {


sorry I missed this one! will remove this restriction it really doesn't matter for gc/lakefs.

nopcoder requested a review from guy-har September 9, 2025 12:23

nopcoder self-assigned this Sep 9, 2025

nopcoder added area/API Improvements or additions to the API include-changelog PR description should be included in next release changelog labels Sep 9, 2025

itaiad200 requested changes Sep 9, 2025

View reviewed changes

api/swagger.yml Outdated Show resolved Hide resolved

pkg/catalog/catalog.go Outdated Show resolved Hide resolved

pkg/catalog/catalog.go Outdated Show resolved Hide resolved

pkg/catalog/catalog.go Outdated Show resolved Hide resolved

guy-har approved these changes Sep 9, 2025

View reviewed changes

api/swagger.yml Outdated Show resolved Hide resolved

pkg/api/controller.go Outdated Show resolved Hide resolved

pkg/catalog/catalog_test.go Outdated Show resolved Hide resolved

nopcoder and others added 10 commits September 18, 2025 15:22

API: Add support to copy object logical mode

00cd639

Make a shallow copy of an object, copy metadata only. Limited to the same repository and branch. Experimental.

Update api/swagger.yml

42c69c3

Co-authored-by: guy-har <[email protected]>

Update pkg/api/controller.go

275dc9b

Co-authored-by: guy-har <[email protected]>

Update the API description

5184f17

clone functionality

32ce5f6

Revert "Update the API description"

7d5fc44

This reverts commit 5184f17.

Revert "Update pkg/api/controller.go"

6f8bd2d

This reverts commit 275dc9b.

Revert "Update api/swagger.yml"

8ea6755

This reverts commit 42c69c3.

Revert catalog test changes related to CloneEntry method

aaaa172

revert api changes

facd253

nopcoder force-pushed the task/catalog-clone branch from 1594d8e to facd253 Compare September 18, 2025 12:34

revert the controller.go changes

29d4a19

nopcoder changed the title ~~API: Add support to copy object logical mode~~ API: copy object will clone object metadata when possible Sep 18, 2025

test catalog copy and clone physical address

1792716

nopcoder requested review from guy-har and itaiad200 September 18, 2025 13:04

reset controller test changes

7732876

nopcoder marked this pull request as draft September 18, 2025 15:32

nopcoder added 4 commits September 20, 2025 16:11

fix controller tests to reflect clone behavior

47bbfc5

python-wrapper: fix copy object not to check physical_address

241b378

check the underlying object time for grace period

85dfd3d

adding the underlying object time check in case of copy of copy.

check mtime greater than or equal to original mtime

c595862

nopcoder added 9 commits September 21, 2025 22:25

Improve error handling in S3 copy object unit tests

e221c4e

we now check grace time using stat, so we get a different error from s3 when object not found

handle no properties as no data exists

446e615

revert the esti test changes

65df281

Merge branch 'master' into task/catalog-clone

0d4c410

map no data error to gateway error

e7c9da5

verify minio error code 404 on copy without source

cfb8314

no such key

007c649

test no such key as error code

cb8bac0

clone support for replace metadata

a10336f

nopcoder marked this pull request as ready for review September 25, 2025 06:20

itaiad200 requested changes Sep 25, 2025

View reviewed changes

nopcoder and others added 3 commits September 26, 2025 10:55

Update pkg/catalog/catalog.go

c643714

Co-authored-by: itaiad200 <[email protected]>

Apply suggestion from @itaiad200

f088e6b

Co-authored-by: itaiad200 <[email protected]>

apply code review changes

e2b606b

nopcoder requested a review from itaiad200 September 26, 2025 08:39

itaiad200 approved these changes Sep 27, 2025

View reviewed changes

nopcoder added 2 commits September 27, 2025 16:34

enable clone on different branches

7532112

remove unused code

d6126d0

nopcoder commented Sep 27, 2025

View reviewed changes

nopcoder merged commit 3439971 into master Sep 27, 2025
41 checks passed

nopcoder deleted the task/catalog-clone branch September 27, 2025 16:56

API: copy object will clone object metadata when possible #9500

API: copy object will clone object metadata when possible #9500

Uh oh!

Conversation

nopcoder commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guy-har left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nopcoder commented Sep 18, 2025

Uh oh!

nopcoder commented Sep 25, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

itaiad200 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nopcoder left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nopcoder commented Sep 9, 2025 •

edited

Loading

github-actions bot commented Sep 9, 2025 •

edited

Loading