Skip to content

Commit fb8db88

Browse files
jiangliujiangliu
authored
Merge pull request #1181 from jiangliu/nydus-image-doc
nydus-image: update documentation docs/nydus-image.md
2 parents 6594edb + 646f320 commit fb8db88

File tree

1 file changed

+179
-54
lines changed

1 file changed

+179
-54
lines changed

‎docs/nydus-image.md

+179-54
Original file line numberDiff line numberDiff line change
@@ -1,65 +1,187 @@
1-
# Nydus Image Builder
1+
# Toolset for Working with RAFS Filesystems and Nydus Container Images
22

3-
`nydus-image` tool converts a directory tree (usually an image layer) into two parts: `bootstrap` and `blob`:
3+
The `nydus-image` toolset provides tools to build, check, inspect and export RAFS filesystems and Nydus container images.
44

5-
- `bootstrap` is a file presenting filesystem metadata information of the directory;
6-
- `blob` stores all files data in the directory;
5+
Logically, a RAFS filesystem consists of two parts:
6+
- a bootstrap/meta blob, containing filesystem metadata, such directory name, file attributes etc.
7+
- one or more data blobs, containing file contents.
78

8-
### Get binary from release page
9+
Physically, RAFS bootstrap/meta blobs can be stored as separate files, or inlined into data blob files.
10+
Therefore, a RAFS file system may consist of the following parts:
11+
- a blob file containing both RAFS metadata and data
12+
- a blob file containing RAFS metadata, and one or more blob files for RAFS data
13+
- a blob file containing RAFS metadata, and one or more blob files for RAFS data, and associated targz files for RAFS ZRAN mode.
914

10-
Get `nydus-image` binary from [release](https://github.com/dragonflyoss/image-service/releases/latest) page.
15+
## Installation
1116

12-
## Build Nydus Image From Directory Source
17+
Get latest `nydus-image` binary from [release](https://github.com/dragonflyoss/image-service/releases/latest) page.
1318

19+
## Nydus Image Builder
20+
21+
The `nydus-image create` subcommand creates a RAFS filesystem or a layer of Nydus image from a tar file or from a directory.
22+
23+
RAFS filesystem/Nydus image has three modes: Native, Zran and Tarfs. Each mode has different features and may be used for different scenarios.
24+
25+
| Mode | Blobs in Registry | Local Cache File | Runtime Integrity | Lazy-loading | Chunk Dedup | Encryption | OCIv1 Compatible |
26+
|:------:|:-----------------------:|:-----------------------------:|:-----------------:|:------------:|:-----------:|:----------:|:----------------:|
27+
| Tarfs | tar.gz / tar.zst | nydus.meta & tar | Optional | No | No | No | Yes |
28+
| Zran | tar.gz & nydus.meta | nydus.meta & nydus.data.cache | Yes | Yes | Yes | No | Yes |
29+
| Native | nydus.data & nydus.meta | nydus.meta & nydus.data.cache | Yes | Yes | Yes | Yes | No |
30+
31+
### Specify Data Blob Output Path
32+
33+
There are two ways to specify where to save the resulting data blob:
34+
35+
- Specify the file path via `--blob <BLOB_FILE>`. It could be a regular file into which the data blob contents are dumped. It can also be a fifo (named pipe) from which "nydusify" or other tools can receive the generated blob content.
36+
37+
- Specify a directory with `-D/--blob-dir BLOB_DIR`. `nydus-image` will use the sha256 digest of the resulting data blob as the filename, concatenated to the directory path. This is useful when you don't want to set a custom name or you are building a layered nydus image. Please create `BLOB_DIR` before executing the command.
38+
39+
### Build RAFS Filesystem in Native Mode from a Directory
1440
```shell
15-
nydus-image create \
16-
--bootstrap /path/to/bootstrap \
17-
--blob /path/to/blob \
41+
nydus-image create -t dir-rafs \
42+
-D /path/to/output/directory \
1843
/path/to/source/dir
44+
45+
[root@image-service]# nydus-image create -t dir-rafs -D images/ src
46+
[2023-03-29 16:34:28.092347 +08:00] INFO successfully built RAFS filesystem:
47+
meta blob path: images/f62a7e668c7f306655233367f8b6e4073d7fa94a6f57826069db3e745e2fd327
48+
data blob size: 0xe32b
49+
data blobs: ["e9d3d45f6ad9f647cc1a2e2f699a46f553ce87b1136026d53d474c6142f80763"]
50+
[root@image-service]# ls -l images/
51+
-rw-r--r-- 1 root root 58155 3月 29 16:34 e9d3d45f6ad9f647cc1a2e2f699a46f553ce87b1136026d53d474c6142f80763
52+
-rw-r--r-- 1 root root 20480 3月 29 16:34 f62a7e668c7f306655233367f8b6e4073d7fa94a6f57826069db3e745e2fd327
1953
```
2054

21-
## Output Blob
22-
23-
Nydus-image tool writes data portion into a file which is generally called `blob`. It has two options to control where `blob` is saved.
55+
### Build RAFS Filesystem in Native Mode with Inlined Metadata from a Directory
56+
```shell
57+
nydus-image create -t dir-rafs \
58+
--blob-inline-meta \
59+
-D /path/to/output/directory \
60+
/path/to/source/dir
61+
62+
[root@image-service]# nydus-image create -t dir-rafs --blob-inline-meta -D images/ src
63+
[2023-03-29 16:36:14.629372 +08:00] INFO successfully built RAFS filesystem:
64+
meta blob path: <none>
65+
data blob size: 0x1392b
66+
data blobs: ["903c62564da0cb18997a4d4c40f25d73c0ab9baef2177f9030d5e0c06ac26fa4"]
67+
[root@image-service]# ls -l images/
68+
-rw-r--r-- 1 root root 80171 3月 29 16:36 903c62564da0cb18997a4d4c40f25d73c0ab9baef2177f9030d5e0c06ac26fa4
69+
```
2470

25-
- With `--blob <BLOB_FILE>` option, nydus-image tool will write blob contents into the custom file path `BLOB_FILE`
71+
### Build RAFS Filesystem in Native Mode from a tar.gz File
72+
```shell
73+
nydus-image create -t targz-rafs \
74+
-D /path/to/output/directory \
75+
/path/to/source/targz.file
76+
77+
[root@image-service]# nydus-image create -t targz-rafs -D images/ src.tar.gz
78+
[2023-03-29 16:40:20.484997 +08:00] INFO successfully built RAFS filesystem:
79+
meta blob path: images/94bf66fc81425bfb72939c942ee1ead90e2e2ac9f09f08f369db15afde163b3b
80+
data blob size: 0xe328
81+
data blobs: ["d3bb8a2cdb6778cbdc31d97be88ef00217d29e4c119f41ef0a4d9f202088d813"]
82+
[root@image-service]# ls -l images/
83+
-rw-r--r-- 1 root root 20480 3月 29 16:40 94bf66fc81425bfb72939c942ee1ead90e2e2ac9f09f08f369db15afde163b3b
84+
-rw-r--r-- 1 root root 58152 3月 29 16:40 d3bb8a2cdb6778cbdc31d97be88ef00217d29e4c119f41ef0a4d9f202088d813
85+
```
2686

27-
- With `--blob-dir BLOB_DIR` provided to command, nydus-image tool creates the blob file named as its sha-256 digest. This is useful when you don't want to set a custom name or you are building a layered nydus image. Please create the `BLOB_DIR` before performing the command.
87+
### Build RAFS Filesystem in Zran Mode from a tar.gz File
88+
```shell
89+
nydus-image create -t targz-ref \
90+
-D /path/to/output/directory \
91+
/path/to/source/targz.file
92+
93+
[root@image-service]# sha256sum src.tar.gz
94+
13111d487b1958281514769eedea840d14e5f27f0d7c2c97b8a286d62645766b src.tar.gz
95+
[root@image-service]# cp src.tar.gz images/13111d487b1958281514769eedea840d14e5f27f0d7c2c97b8a286d62645766b
96+
[root@image-service]# file images/13111d487b1958281514769eedea840d14e5f27f0d7c2c97b8a286d62645766b
97+
images/13111d487b1958281514769eedea840d14e5f27f0d7c2c97b8a286d62645766b: gzip compressed data, last modified: Wed Mar 29 08:39:20 2023, from Unix, original size 245760
98+
[root@image-service]# nydus-image create -t targz-ref -D images/ images/13111d487b1958281514769eedea840d14e5f27f0d7c2c97b8a286d62645766b
99+
[2023-03-29 16:48:51.656612 +08:00] INFO successfully built RAFS filesystem:
100+
meta blob path: images/606e8f8fbce6496b676f09f6b5231d15c301424af5b54a0433b2e9071bbe857d
101+
data blob size: 0xb008
102+
data blobs: ["13111d487b1958281514769eedea840d14e5f27f0d7c2c97b8a286d62645766b"]
103+
[root@image-service]# ls -l images/
104+
-rw-r--r-- 1 root root 45064 3月 29 16:48 13111d487b1958281514769eedea840d14e5f27f0d7c2c97b8a286d62645766b
105+
-rw-r--r-- 1 root root 4343 3月 29 16:48 2ae4b87374bbb7be0f10300c20617bc7f40d96a8a12a43445f88d95dd326c7dd
106+
-rw-r--r-- 1 root root 20480 3月 29 16:48 606e8f8fbce6496b676f09f6b5231d15c301424af5b54a0433b2e9071bbe857d
107+
```
28108

29-
Generally, this is regular file which blob content will be dumped into. It can also be a fifo(named pipe) from which nydusify or other tool can receive blob content.
109+
### Build RAFS Filesystem in Tarfs Mode from a tar File
110+
```shell
111+
nydus-image create -t tar-tarfs \
112+
-D /path/to/output/directory \
113+
/path/to/source/tar.file
114+
115+
[root@image-service]# sha256sum src.tar
116+
0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a src.tar
117+
[root@image-service]# cp src.tar images/0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
118+
[root@image-service]# nydus-image create -t tar-tarfs -D images/ images/0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
119+
[2023-03-29 16:52:44.251252 +08:00] INFO successfully built RAFS filesystem:
120+
meta blob path: images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
121+
data blob size: 0x3c000
122+
data blobs: ["0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a"]
123+
[root@image-service]# ls -l images/
124+
-rw-r--r-- 1 root root 245760 3月 29 16:52 0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
125+
-rw-r--r-- 1 root root 20480 3月 29 16:52 90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
126+
```
30127

31-
## Layered Build Nydus Image
128+
### Layered Build Nydus Image
32129

33130
`nydus-image` tool supports to build Nydus image from multiple layers of image:
34131

35-
```shell
36-
# Build from lower layer
37-
nydus-image create \
38-
--bootstrap /path/to/bootstrap \
39-
--blob /path/to/blob \
132+
```shell
133+
# Build from lower layer
134+
nydus-image create \
135+
-D /path/to/output/dir \
40136
/path/to/lower/dir
41-
# Build from upper layer based on lower layer
42-
nydus-image create \
137+
# Build from upper layer based on lower layer
138+
nydus-image create \
43139
--parent-bootstrap /path/to/parent-bootstrap \
44-
--bootstrap /path/to/bootstrap \
45-
--blob /path/to/blob \
140+
-D /path/to/output/dir \
46141
/path/to/upper/dir
47142
```
48143

49-
## Build Nydus Image With Chunk-Dict
144+
### Build Nydus Image With Chunk-Dict
50145
`nydus-image` tool supports to build Nydus image with chunk-dict for chunk deduplication:
51146
1. reference chunks which are same as chunks in chunk-dict to blobs in chunk-dict
52147
2. new dumped blob would be smaller than without using chunk-dict
53148
3. save space of remote storage because of chunk-deduplication between images (e.g. oss, registry)
54149
```shell
55150
# Build with bootstrap type chunk-dict
56151
nydus-image create \
57-
--bootstrap /path/to/bootstrap \
58152
--chunk-dict bootstrap=/path/to/dict.boot \
59-
--blob /path/to/blob \
153+
-D /path/to/output/dir \
60154
/path/to/lower/dir
61155
```
62156

157+
## Merge Multiple RAFS Filesystems into One
158+
159+
`nydus-image` tool supports to build Nydus image from multiple layers of image:
160+
The `nydus-image merge` subcommand supports merging multiple RAFS filesystems into one.
161+
It applies the overlay rules defined the OCI Image Spec or the overlayfs, to avoid using `overlayfs` at runtime.
162+
163+
```shell
164+
nydus-image merge \
165+
-D /path/to/output/dir \
166+
/path/to/bootstrap1 /path/to/bootstrap2
167+
168+
[root@image-service]# nydus-image create --blob-inline-meta -D images/ src
169+
[2023-03-29 17:02:06.231478 +08:00] INFO successfully built RAFS filesystem:
170+
meta blob path: <none>
171+
data blob size: 0x1392b
172+
data blobs: ["903c62564da0cb18997a4d4c40f25d73c0ab9baef2177f9030d5e0c06ac26fa4"]
173+
[root@image-service]# nydus-image create --blob-inline-meta -D images/ blobfs/
174+
[2023-03-29 17:02:08.980743 +08:00] INFO successfully built RAFS filesystem:
175+
meta blob path: <none>
176+
data blob size: 0x86ba
177+
data blobs: ["9e50ae5ac02b2ef6ffb86075720e49d95d8240eed4717dd8ac9c68cadba00762"]
178+
[root@image-service]# nydus-image merge -D images/ images/903c62564da0cb18997a4d4c40f25d73c0ab9baef2177f9030d5e0c06ac26fa4 images/9e50ae5ac02b2ef6ffb86075720e49d95d8240eed4717dd8ac9c68cadba00762
179+
[root@image-service]# ls -l images/
180+
-rw-r--r-- 1 root root 80171 3月 29 17:02 903c62564da0cb18997a4d4c40f25d73c0ab9baef2177f9030d5e0c06ac26fa4
181+
-rw-r--r-- 1 root root 34490 3月 29 17:02 9e50ae5ac02b2ef6ffb86075720e49d95d8240eed4717dd8ac9c68cadba00762
182+
-rw-r--r-- 1 root root 20480 3月 29 17:02 df01f389850b79cd5a6ca6db98495bb457aa0821b0558351c55537551322fb96
183+
```
184+
63185
## Compact Nydus Image
64186
`nydus-image` tool supports to compact Nydus image for
65187
1. reduce number of blobs
@@ -105,35 +227,38 @@ nydus-image create \
105227
/path/to/lower/dir
106228
```
107229

108-
## Build Nydus Image From Stargz Index
230+
## Export RAFS Filesystem into Other Formats
109231

110-
### Convert image layer to stargz format
232+
### Export RAFS Filesystem as Raw Block Device Image
111233

112-
```shell
113-
tar --xattrs --selinux -czvf ./layer.tar.gz <layer-directory>
114-
stargzify file:layer.tar.gz file:layer.stargz
115-
tar -xzvf layer.stargz stargz.index.json
116-
```
117-
118-
### Stargz build
119-
120-
```shell
121-
nydus-image create \
122-
--source-type stargz_index \
123-
--bootstrap /path/to/parent-bootstrap \
124-
--blob-id <image-lower-layer-id> \
125-
/path/to/stargz.index.lower.json
126-
```
127-
128-
### Stargz layered build:
234+
A RAFS filesystem can be exported as a raw block device image, so it can be exposed as block device through loop device, NBD and virtio-blk etc.
235+
If the `--verity` option is given, it will also generate verification data to enable `dm-verity`.
129236

130237
```shell
131-
nydus-image create \
132-
--source-type stargz_index \
133-
--parent-bootstrap /path/to/parent-bootstrap \
134-
--bootstrap /path/to/bootstrap \
135-
--blob-id <image-upper-layer-id> \
136-
/path/to/stargz.index.upper.json
238+
nydus-image export --block --verity \
239+
-D /path/to/output/dir \
240+
-B /path/to/bootstrap
241+
242+
[root@image-service]# nydus-image create -t tar-tarfs -D images/ src.tar
243+
[2023-03-29 17:20:02.500167 +08:00] INFO successfully built RAFS filesystem:
244+
meta blob path: images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
245+
data blob size: 0x3c000
246+
data blobs: ["0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a"]
247+
[root@image-service]# cp src.tar images/0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
248+
[root@image-service]# nydus-image export --block --verity -D images/ -B images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
249+
[2023-03-29 17:20:47.676914 +08:00] INFO RAFS features: COMPRESSION_NONE | HASH_SHA256 | EXPLICIT_UID_GID | TARTFS_MODE
250+
dm-verity options: --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=512 --hash-block-size=4096 --data-blocks 4576 --hash-offset 2342912 6b5743e7da406a33ab3a8bb03b65e67d1c1951b2d7ebc5026e0de3fb44a7cc20
251+
[root@image-service]# ls -l images/
252+
-rw-r--r-- 1 root root 245760 3月 29 17:20 0e2dbe8b6e0f55f42c75034ed9dfc582ad0a94098cfc248c968522e7ef02e00a
253+
-rw-r--r-- 1 root root 20480 3月 29 17:20 90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47
254+
-rw-r--r-- 1 root root 2494464 3月 29 17:20 90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
255+
[root@image-service]# losetup /dev/loop1 images/90f0e6e7e0ff822d4acddf30c36ac77fe06f549fe58f89a818fa824b19f70d47.disk
256+
[root@image-service]# veritysetup open --no-superblock --format=1 -s "" --hash=sha256 --data-block-size=512 --hash-block-size=4096 --data-blocks 4576 --hash-offset 2342912 /dev/loop1 verity /dev/loop1 6b5743e7da406a33ab3a8bb03b65e67d1c1951b2d7ebc5026e0de3fb44a7cc20
257+
[root@image-service]# lsblk
258+
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
259+
loop1 7:1 0 2.4M 0 loop
260+
└─verity 252:0 0 2.2M 1 crypt /root/image-service/mnt
261+
[root@image-service]# mount -t erofs -r /dev/dm-0 mnt/
137262
```
138263

139264
**Note**: the argument value of image layer id specified in nydus-image CLI should omit `sha256:` prefix.

0 commit comments

Comments
 (0)