Add container copy/cp command for host-container file transfer

[simone-panico]

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Adds the container copy (aliased as cp) command to copy files between a running container and the local filesystem.

I saw #1023 and the feedback from @dcantah — the previous attempt relied on tar being installed inside the container.
This implementation takes the recommended approach:
file transfers go through the guest agent via the existing copyIn/copyOut methods on the core Containerization, with no dependency on container tooling.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[JaewonHur]

Related to #232

[JaewonHur]

⏺ CopyOut/CopyIn Scenarios: Current vs Docker Behavior

Scenario 1: copy(in/out) src dst (src must exist)

• File, Directory → Non-existing:

  • Docker (desired): save as dst
  • Current: File is saved as dst, Directory is not supported.

• File → File

  • Docker (desired): replace dst
  • Current: replace dst

• Directory → File

  • Docker (desired): error
  • Current: Directory is not supported

• File, Directory → Directory

  • Docker (desired): save under dst
  • Current: File returns error, Directory is not supported.

Scenario 2: copy(in/out) src dst/ (dst must be a directory if existing)

• File → Non-existing:

  • Docker (desired): error no such directory
  • Current: dst is created and src file is saved under dst.

• Directory → Non-existing:

  • Docker (desired): save as dst
  • Current: Directory is not supported.

• File, Directory → Directory

  • Docker (desired): save under dst
  • Current: File is saved under dst. Directory is not supported.

Scenario 3: copy(in/out) src/ dst (src must be an existing directory)

• Directory → Non-existing:

  • Docker (desired): save as dst
  • Current: Directory is not supported.

• Directory → File

  • Docker (desired): error
  • Current: Directory is not supported.

• Directory → Directory

  • Docker (desired): save under dst
  • Current: Directory is not supported.

Scenario 4: copy(in/out) src/ dst/ (src must be an existing directory & dst must be a directory if existing)

• Directory → Non-existing:

  • Docker (desired): save as dst
  • Current: Directory is not supported.

• Directory → Directory

  • Docker (desired): save under dst
  • Current: Directory is not supported.

[JaewonHur]

Hi @simone-panico
Above I summarized the desired behavior and current behavior.

The bold italic phrases are what is different from desired behavior (i.e., docker) now, and can be addressed.

Below is general comments.

1. Copying directory is not supported yet, we need to handle it later.
2. If dst/, it would be better to check in command line if dst is directory if it exists.

[JaewonHur]

Hi @simone-panico
Could you try this PR with apple/containerization#474, which supports copying directories.

You may need to build this project with local containerization (BUILDING.md), and wire up SandboxService.

[dcantah]

@JaewonHur the change above does not allow copying directories. It adds a tar reader/writer that will be used to support this, but it's not in the change itself.

[simone-panico]

If you’re interested, I could try to create a Directory Copier within containerization. It would also be a great challenge for me

[JaewonHur]

if it starts creating and writing files and then fails we would need to first delete everything and then recreate.

Could you describe more how it fails in the middle of writing files? I thought vminitd's copy would fail before starting any persistent change if initial conditions don't meet.

Another option could be by making a Stat RPC that gives back two Bools (exists and is_directory). Like that we could handle the logic in Container but get the relevant Information we need. Something like the Unix stat syscall.

Yes, we might modify current copy RPC or add an RPC for stat.

[simone-panico]

container cp foo guest:/bar/, first try copy(foo, guest:/bar/foo), and if it fails, then try copy(foo, guest:/bar) if foo is directory.

I initially thought the first copy(foo, guest:/bar/foo) would create parent directories as a side effect, making rollback necessary. But actually we can do copy(foo, guest:/bar/foo, createParents: false) on the first call, so it fails.
My Bad ;)

Yes, we might modify current copy RPC or add an RPC for stat.

Sure, just let me know what the team decides and I'll implement it :)

[JaewonHur]

Thanks a lot! :)
Will be in touch soon!

[JaewonHur]

Hi @simone-panico we'd like to keep the API change in vminitd as small as possible.

Would it be able to achieve it without changing vminitd?

[simone-panico]

Hi @simone-panico we'd like to keep the API change in vminitd as small as possible.

Would it be able to achieve it without changing vminitd?

Hey, yeah sure, I'll try it :)

[JaewonHur]

Hi @simone-panico

Following is my pseudo code.

Given cp src dst

1. If dst is directory

• copy src to dst/src

2. If dst is file

• src must be file, and dst is replaced with src.

3. If dst is not existing

• If dst has trailing /, and src is file, error
• else, just create dst and write src

Based on it, I thought how the logic would be if we can get an error root cause from the failed copy (going retry approach).

Given cp src dst, we perform the following logic (copy in the following I mean vminitd's copy)

if copy src to dst/src succeeds,
    done
else
    if the error indicates dst = file
        if src = file
            copy src to dst
        else
            error
    else if the error indicates dst = non-existing
        if src = dir
            copy src to dst
        else
            if dst ends with "/"
                error
            else
                copy src to dst
    else
        error

It's so ugly lol. But, without something like stat RPC, I cannot come out with something better.
(actually we are using that error message instead of stat)

Oh but this logic only works for copyIn scenario as we need to stat src.
For copyOut

If dst = directory
    copy src to dst/src
else
    if dst ends with "/"
        copy src to dst
        then, check if src is file, and yes, then remove and error
    else
        copy src to dst (and let it error if those type are different)

LOL it's even more ugly.

How do you think?

[simone-panico]

Hey @JaewonHur, Thank you for the Pseudo.
Yes, it's not the prettiest . I'd still highly recommand adding a Stat RPC.
That said, I went ahead and implemented it using the exception handling approach (like your Pseudo) and all the Tests are passing :)

I also found a bug that I had to fix in 590#containerization, so that PR needs to be merged first.

If you and your team change your mind about the Stat RPC, I'd be happy to do that!

[JaewonHur]

@simone-panico Thanks for all your hard work :)

Give me a few more time til we decide the best direction!

[simone-panico]

Hey @JaewonHur, I hope you're doing great.
Do you have any updates?

[JaewonHur]

Hi @simone-panico sorry for delay 🥲
I made a PR for supporting stat in vminitd (apple/containerization#614), but waiting there for adding test.

Once I merge that PR, you can continue working on container copy, and I bet this will be the final!
Sorry again for taking your time.

Let me ping you as soon as I merge that PR!

[JaewonHur]

Hi @simone-panico!

I just merged apple/containerization#614, which adds stat RPC to vminitd.

It takes absolute path (outside container environment) and returns stat result of the file system object under that path.
So, now you should be able to stat whether the copy source or destination is file or directory.

This shall help making the logics clean.
Thank you for your patience, and let's finish it finally!

[simone-panico]

Hey @JaewonHur
No worries, and thank you for the Stat RPC. I've started looking into it but haven't finished yet. I'll get to it as soon as I can 😄

[JaewonHur]

@simone-panico Thank you!
You might want to change copyIn and copyOut methods of LinuxContianer in containerization to use stat RPC.

[simone-panico]

Hey @JaewonHur, sorry this took so long. I had to finish my finals.
I’ve now implemented the Stat RPC in apple/containerization#727 and updated this PR to use it for copyIn.
Every copy test is passing now: 70/70. 😄