Overlay size, as alternative to --storage-opt size=30G

[lukasmrtvy]

Description

Would be superb to support limiting disk space similar to the Docker's --storage-opt size=30G
Thanks

Is this feature related to a specific bug?

No response

Do you have a specific solution in mind?

No response

[manojks1999]

/assign

[ayushr2]

IIUC Docker's --storage-opt should be working correctly with gVisor too. Are you experiencing a bug with that? Or is this a feature request to "limit disk space" akin to that option?

Limiting disk space depends on the underlying filesystem driver. I think higher-level container runtimes like Docker are the right places to implement this feature.

You can easily limit disk space on tmpfs BTW using the size mount option; see mount(5). gVisor supports this mount option.

[lukasmrtvy]

@ayushr2 Hi, thanks. The use case would be with Docker + containerd ( https://docs.docker.com/desktop/features/containerd/ ). I have tried a storage-opt size= arg, but it's not working. Maybe this has to be supported directly in containerd snapshotter?

[ayushr2]

I have tried a storage-opt size= arg, but it's not working.

Is it only not working with gVisor? Or is this not working without gVisor as well?

Maybe this has to be supported directly in containerd snapshotter?

I believe so... the component configuring the host container filesystem has to set disk limits using filesystem driver specific primitives.

[PriyanshGodhani22]

Can I work on this issue!?

[stepancheg]

I did a simple experiment. I opened this code:

gvisor/runsc/boot/vfs.go

Lines 583 to 587 in f45c79e

	func (c *containerMounter) configureOverlay(ctx context.Context, conf *config.Config, creds *auth.Credentials, lowerOpts *vfs.MountOptions, lowerFSName string, filestoreFD *fd.FD, mountConf GoferMountConf, dst string) (*vfs.MountOptions, func(), error) {
	// First copy options from lower layer to upper layer and overlay. Clear
	// filesystem specific options.
	upperOpts := *lowerOpts
	upperOpts.GetFilesystemOptions = vfs.GetFilesystemOptions{InternalMount: true}

And added

	upperOpts.GetFilesystemOptions = vfs.GetFilesystemOptions{
		InternalMount: true,
		Data: "size=1m"}

And overlay size limit magically started to work.

So this task seems to be about propagating this parameter from runsc invocation down to this line of code.

AFAIU config.json does not have an option for limiting storage size, so it can be configured with command line option like:

runsc --overlay-size=1m

Please confirm my understanding is correct, and I'll try to create a PR.

[ayushr2]

@stepancheg could you clarify your use case? I assume you want to set a size limit on the container's rootfs.

Are you using the default --overlay2 flag (which is root:self)?

• If yes, then are you using Docker?
  • If yes, then the recommended way of doing that is just using Docker's --storage-opt flag. When using that flag, the storage limit should be applied to gVisor container's rootfs as well. However, note that --storage-opt is not supported with "Storage Driver: overlay2" and "Backing Filesystem: extfs" (common case). You will need to reformat /var/lib/docker to XFS (with pquota option) or btrfs or zfs. This is not a deficiency in gVisor, but in Docker itself; see Docker Run Error with –storage-opt and Overlay2 on XFS with pquota moby/moby#46823.
  • If not, then how are you running the container? You need to set up the rootfs on the host with the size limit. Usually the rootfs is an overlay. So you would need to set the size limit on the host tmpfs (upper layer).
• If no, are you using --overlay2=*:memory flag (i.e. not using a file-backed overlay upper layer). In this case the upper layer is backed by application memory. I think could achieve the same effect by setting memory limits on the container.

[stepancheg]

could you clarify your use case?

Run untrusted code and make sure it does not break the outer machine by wasting all disk space.

Also rely less on Linux kernel, which usually works, but sometimes creates unexpected problems under heavy load.

I assume you want to set a size limit on the container's rootfs.

Yes.

Are you using the default --overlay2 flag

Currently, yes.

are you using Docker
If not, then how are you running the container?

We invoke runsc directly by generating config.json and passing flags.

Usually the rootfs is an overlay.

Currently we rely on runsc overlay, no outer linux kernel overlay.

Technically we can reformat disks and use xfs or btrfs disk quota, or use tmpfs as you suggested, but:

• xfs and btrfs requires complex machine setup (and maintenance)
• tmpfs use precious memory
• also mounts usually work, but I've seen some strange behavior when kernel stuck at unmounting for example. We create a lot of short living containers, and these are painful to deal with compared to just being able to kill runsc if something times out

If no, are you using --overlay2=*:memory flag

This approach has two drawbacks:

• we spend expensive memory instead of cheap disk
• running out of "disk space" will result in misleading OOM kill instead of helpful error "No space left on device"

[ayushr2]

OK, lets proceed with review on #11723. We need to figure out a way to plumb the size option via the gofer mount config in Container.initGoferConfs() iff the overlay is being configured via the --overlay2 flag. Otherwise (in case of no overlay OR mount annotations), the size option should remain unset.