feat: implement copy

[jiwangCHEN]

Summary

Implements a complete OCI copy engine for oras-py (a Python port of oras-go's
copy engine) and rewires Registry.push / Registry.pull to run on top of it.

Copies a content DAG between any two targets — registries, OCI layouts, or
in-memory stores — with concurrent traversal, per-node deduplication,
existence checks, optional cross-repo blob mounting, foreign-layer filtering,
and digest/size verification.

Design

1. Traversal: post-order concurrent DAG walk.
   A node's children must be fully committed to the destination before the node itself is pushed — a registry will reject a manifest whose layers don't yet exist, so this is a correctness invariant, not an ordering preference. To keep this deadlock-free at any concurrency level (including 1), a worker must release its semaphore slot before recursing into its children (region.end()) and reacquire it before copying itself (region.start()).

2. Deduplication: a first-class primitive.
   Every node passes through a StatusTracker before any other work. When N workers reach the same descriptor (e.g., a blob shared by multiple manifests in an index), exactly one becomes the doer and the other N−1 become waiters; without this, shared blobs would be fetched and pushed N times concurrently and race on dst.push().

3. Caching: decouple resolution from traversal.
   Wrap the source in a CacheProxy that absorbs manifest and index bytes on first fetch (during root resolution and successor discovery), so that when the traversal later copies the root it reads from memory instead of re-hitting the network. Provide a stop_caching flag: while map_root runs, it may fetch different content (e.g., platform selection), and that content must not pollute the cache.

4. Extensibility: hooks, not subclassing.
   Expose pre_copy, post_copy, on_copy_skipped, and on_mounted as injected callables. The tagging mechanism — guaranteeing dst_ref ends up pointing at the root — must be implemented entirely through these hooks, with no special-case branch in the traversal. A SkipNode exception raised from pre_copy must short-circuit the node's copy; this is the mechanism by which a ReferencePusher atomically pushes and tags the root in a single PUT.

Limitation identified

• ~8 MB/thread × N items in a single _go() batch — wide indexes could hit OS thread limits
• requests connection pool can throttle below the semaphore at high concurrency

Public API

Entry point	Use
Registry.copy(src, dst, opts=...)	Copy between repositories/registries
Registry.as_target(container)	Adapt a registry repo for oras.copy.copy
Layout.copy_to_registry / copy_from_registry	Layout ⇄ registry transfer
Layout.as_target()	Adapt an OCI layout for oras.copy.copy

Testing

• oras/tests/test_copy.py — descriptor utils, memory storage, cache proxy,
  status tracker, graph traversal, dedup, mounting, error origins, digest/size
  verification, path-traversal rejection, and LimitedRegion deadlock
  prevention (incl. concurrency=1).
• oras/tests/test_provider.py — mock-based copy/push/pull plus a
  live-registry registry-to-registry round-trip.

[vsoch]

Thanks @jiwangCHEN I can give a review when I am back at keyboard. To start, please follow existing conventions in the project. For example, a type like descriptor should go into types.py. Ensure you integrate your addition maximally with existing structure. Thanks!

[vsoch]

And another question. Copy was added to oras-go as a replacement for conventional push and pull - each is just a directional copy. If we add it here those functions need to be adapted to use it so we are not doing the same thing two different ways.

[jiwangCHEN]

Thanks. I have address the first comment:

• Added Descriptor to existing [oras/types.py]
• Updated descriptor.py to import from oras.types instead of defining locally
• Updated other files to import Descriptor from oras.types directly

[vsoch]

Thanks. There is still the issue that copy (in the Go library) was a replacement for push/pull - the push pull interfaces would use it under the hood. With your changes here, we have both. We need have one or the other.

[jiwangCHEN]

Understood. Layout.push_to_registry() and Layout.copy_to_registry() both push layout content to a registry — I'm working on a replacement for those. For Layout.pull_from_registry(), since it's a manual blob-download with no copy engine equivalent, it requires implementing Target (not just ReadOnlyTarget); I'll handle that as well.

[vsoch]

More specifically, the provider.py interfaces need to use copy under the hood.

https://github.com/oras-project/oras-py/blob/main/oras/provider.py

[jiwangCHEN]

Hi @vsoch, The provider and examples have been updated to support the copy engine.
To avoid breaking changes for existing users, the deprecation of the pull and push functionality should be deferred to a future pr/release.

[vsoch]

@jiwangCHEN push and pull interfaces should not go away. A push or pull is a directional copy. There should be no breaking changes in changing the underlying structure to use copy.

[jiwangCHEN]

Thanks. Let me know if this change plan looks solid to you: Rewrite pull() and push():
New push() flow

1. New helper _pack_files_to_layout(layout_target, files, ...) -> Descriptor — the existing push
   logic verbatim but each blob/config is written via layout_target.push(desc, io.BytesIO(data)) instead of self.upload_blob(...). Finally serialize the manifest, push its
   blob, and layout_target.tag(manifest_desc, tag).
2. In push(): create temp Layout(tmp, validate=False), _pack_files_to_layout(layout.as_target(), ...).
3. opts = CopyOptions(); opts.graph.do_chunked = do_chunked; opts.graph.chunk_size = chunk_size
4. layout.copy_to_registry(provider=self, target=target, tag=(container.digest or container.tag), opts=opts).
5. Return value: after copy, issue a GET on the manifest and return that requests.Response
6. finally: shutil.rmtree(tmp, ignore_errors=True).

[vsoch]

Please report the model name and version you used to generate this code, along with the prompts and how you, the person, have reviewed it. Please describe to me, in your own words (not using the LLM) the changes and the tests you have implemented.

[jiwangCHEN]

Please report the model name and version you used to generate this code, along with the prompts and how you, the person, have reviewed it. Please describe to me, in your own words (not using the LLM) the changes and the tests you have implemented.

I was using Claude Opus 4.8, following a prompt workflow aligned with the development cycle: planning, implementation, PEP 8 style check, and test case validation.

In the initial implementation, I focused on a specific prompt and refined the output through multiple iterations

Implements a complete OCI copy engine for oras-py (a Python port of oras-go's copy engine) 

Copy Overview

ACTION copy(src, src_ref, dst, dst_ref, opts):
    state: ValidatingInputs
    opts:  opts.clone() or default Options (mutation safety)

    if dst_ref == "": src_ref → dst_ref

    -- ValidatingInputs
    assert src ≠ null  else raise CopyError(SOURCE, "Resolve")
    assert dst ≠ null  else raise CopyError(DEST,   "Resolve")
    **state**:  CreatingProxy

    -- CreatingProxy
    proxy: CacheProxy(base=src, max_bytes=opts.max_metadata_bytes)
    **state**: ResolvingRoot

    -- ResolvingRoot
    root: resolve_root(src, src_ref, proxy)
    **state**: MappingRoot

    -- MappingRoot
    if opts.map_root ≠ null:
        proxy.stop_caching = true
        root = opts.map_root(proxy, root)
        proxy.stop_caching = false
    **state**: InstallingHooks

    -- InstallingHooks
    install_tagging_hooks(dst, dst_ref, proxy, root, opts)
    **state**: RunningGraph

    -- RunningGraph
    copy_graph(src=src, dst=dst, root=root, proxy=proxy, opts=opts)
    **state**: Done(root)
    return root

Concurrency

────────────────────────────────────────────────────
WORKER LAUNCHER (with semaphore gating)
────────────────────────────────────────────────────

ACTION launch_workers(sem, fn, descriptors, cancel):
    for each desc in descriptors:
        thread ← new daemon Thread:
            if cancel.is_set(): return
            region ← LimitedRegion(sem)
            region.start()              -- acquire slot (blocks if full)
            if cancel.is_set():
                region.end(); return
            try:
                fn(region, desc)
            catch e:
                first_error ← e         -- capture first error only
                cancel.set()            -- signal all workers to stop
            finally:
                region.end()            -- always release slot
        thread.start()


────────────────────────────────────────────────────
COPY NODE
────────────────────────────────────────────────────

Per-node:      Claimed → CheckingExistence
               → [Skipped | FindingSuccessors]
               → [YieldingSlot → WaitingForChildren → ReacquiringSlot]?
               → Copying → NodeDone

Single copy:   PreCopy → [Skipped_ByHook | Transferring → PostCopy → NodeCopied]

States:         Fetching → Buffering → VerifyingDigest → VerifyingSize
               → Pushing → Transferred

During reconsolidating the classes, I used following prompts.

Consolidate oras-py copy-engine dependency
Reorganize into a strict, acyclic top-to-bottom layering: [provider (entry) → copy (algorithm) → content → target interface (from/to) → implementations (memory/registry/layout)]. 
Motivation: eliminate the existing [content ⟷ copy] circular import, move the copy algorithm into [copy/], and gather the three target implementations behind one interface in a new [oras/content/] package.

Would have Copilot to review the work before proceeding to the next iteration including address the comments above.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.