Spurious L0 EMBEDDED record at file offset 0xffffffffffff0000 #18239
Description
System information
| Type | Version/Name |
|---|---|
| Distribution Name | FreeBSD |
| Distribution Version | FreeBSD 14.3 |
| Kernel Version | customized 14.3-RELEASE |
| Architecture | amd64 |
| OpenZFS Version | zfs-2.2.7-FreeBSD_ge269af1b3 |
Describe the problem you're observing
I have a 2.5MB file on a ZFS file system. It is perfectly readable by ordinary tools like md5 and cp. However, if I attempt to zfs send -e the dataset and zfs recv it somewhere else, it will fail with the error cannot receive new filesystem stream: incompatible embedded data stream feature with encrypted receive.. Without -e, the error is slightly different: cannot receive resume stream: destination ... contains partially-complete state from "zfs receive -s".
zdb -ddddd shows the reason why: the file has an extra embedded record at offset 0xffffffffffff0000. It definitely shouldn't be there.
Details
Dataset XXXXX [ZPL], ID 3642930, cr_txg 2873887, 8.47T, 5909544 objects, rootbp DVA[0]=<1:28d7df440000:3000> DVA[1]=<3:28ec50ba1000:3000> [L0 DMU objset] fletcher4 uncompressed unencrypted LE contiguous unique double size=1000L/1000P birth=4156787L/4156787P fill=5909544 cksum=000000103576acdb:00002b778d2e8982:003f8aba33eff679:430b3567a64ad9db
Object lvl iblk dblk dsize dnsize lsize %full type
396510 6 128K 64K 1022K 512 0 inf ZFS plain file
192 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED
dnode maxblkid: 281474976710655
path XXXXX
uid 501
gid 20
atime Tue Dec 9 23:45:41 2025
mtime Sat Jun 27 01:23:29 2015
ctime Tue Dec 9 23:45:41 2025
crtime Tue Dec 9 23:45:41 2025
gen 3851750
mode 100777
size 2581026
parent 30643
links 1
pflags 840800000104
Indirect blocks:
0 L5 6:379c6cb85000:3000 20000L/1000P F=41 B=2880454/2880454 cksum=00000089d1763f47:0001f3c5221100e8:038d2d5233e00659:5250084caa44e9d1
0 L4 6:379c6cb7f000:3000 20000L/1000P F=40 B=2880454/2880454 cksum=00000084aaef6147:0001e5f1e2b19247:037c72e002b3ef33:47b0a666e23bbca3
0 L3 3:4dafe7879000:3000 20000L/1000P F=40 B=2880454/2880454 cksum=00000084cf3ad8de:0001e67cf598016c:037d7d741bd2ecfb:4905757fad87603b
0 L2 3:4dafe7834000:3000 20000L/1000P F=40 B=2880454/2880454 cksum=0000008608031e2e:0001eb5cdba93661:038736be85607802:55f447fcaf4e3def
0 L1 1:3e9ca3407000:3000 20000L/1000P F=40 B=2880454/2880454 cksum=0000011ec4b19d49:0002f19e8bec9a05:0421a6e4283c6420:1b85ddefedd56372
0 L0 4:3bfff3558000:9000 10000L/5000P F=1 B=2880454/2880454 cksum=000008c812205323:00631ea489f62438:9d8d2efb07e19c44:c681f1ac1b5232f7
10000 L0 4:3bfff3561000:9000 10000L/5000P F=1 B=2880454/2880454 cksum=0000083f497c4429:00600aef66285324:8f0fdb8f346c1c60:1288dfc22873795c
20000 L0 4:3bfff35a6000:9000 10000L/5000P F=1 B=2880454/2880454 cksum=0000092da020376e:0061147e67294b3e:90144eb20a829096:cd2d1cc236baf4f4
30000 L0 4:3bfff356a000:9000 10000L/5000P F=1 B=2880454/2880454 cksum=0000097000647733:005f4e47541370b2:6b75387504622a1c:ca0e39d68fedcc55
40000 L0 4:3bfff359d000:9000 10000L/5000P F=1 B=2880454/2880454 cksum=0000098d2c98527c:0061ea2e038acdab:875c7a0b44366489:d530ee2ea2a5b4c2
50000 L0 4:3bfefecbf000:c000 10000L/6000P F=1 B=2880454/2880454 cksum=00000a1ec2e612c6:008cfc4bfa7484b9:836e1ef6b4a37a13:1f677c214cfa5537
60000 L0 4:3bfff3573000:15000 10000L/b000P F=1 B=2880454/2880454 cksum=000014c2db6a487e:01d48b8e7cb5af07:cfccdbdd58433671:98eb8104812ac726
70000 L0 4:3bfff3588000:15000 10000L/b000P F=1 B=2880454/2880454 cksum=000013abccfc5a43:01c3a158b5b636af:86315f232f426886:1cf14e8a5a3bcf2a
80000 L0 4:3bfefecb3000:c000 10000L/6000P F=1 B=2880454/2880454 cksum=00000ad191bccd91:008cb4330d000e43:68fcc53ff590b049:f8bfde2158699958
90000 L0 4:3bfefeccb000:c000 10000L/6000P F=1 B=2880454/2880454 cksum=00000a9da22d6828:008b75274176a28e:59bb1113772ba9cd:c41dd8e739dd8126
a0000 L0 4:3bfefecd7000:c000 10000L/6000P F=1 B=2880454/2880454 cksum=00000a19b552c8d1:008a6185e1ce8364:60bf5c4c0d338132:4c0e3ff96fc92790
b0000 L0 4:3bfff35af000:f000 10000L/7000P F=1 B=2880454/2880454 cksum=00000dbd22aae8e9:00c94bf7e1719fbc:8e7d0a97ae5ba7b2:ba6153b39a268468
c0000 L0 4:3bfc1788f000:18000 10000L/e000P F=1 B=2880454/2880454 cksum=0000184b650cf995:02d719d764539284:46c7c82ac640c00f:019eb9b29bafe09f
d0000 L0 4:3bfc178a7000:18000 10000L/e000P F=1 B=2880454/2880454 cksum=0000189a1aa954ec:02e179d3bf654164:0d5aed3e127d035e:d48169c808fc3d98
e0000 L0 4:3bfc1793a000:18000 10000L/e000P F=1 B=2880454/2880454 cksum=000018821fc6dd6d:02df43f7edd8e67a:f15e1b2334e83c8e:a33941cec192b118
f0000 L0 4:3bfc17a84000:18000 10000L/e000P F=1 B=2880454/2880454 cksum=0000189a1b154726:02e1630dd13e58e8:098642ad3ab75aec:c63ec1c662b598ab
100000 L0 4:3bfc17a54000:18000 10000L/e000P F=1 B=2880454/2880454 cksum=00001847a785e7a3:02d8ed96659e8527:794e987fa5bfc296:17536cb3eea13ecf
110000 L0 4:3bfc17a6c000:18000 10000L/d000P F=1 B=2880454/2880454 cksum=0000171027e087a6:027a49788458ef44:0c767deb5fb53256:94a483c7230cf0c4
120000 L0 4:3bfff35be000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=000001c1c40a7f4f:0003f7ec151a7f23:0552211c68f9c5a9:491e3455c45d8b61
130000 L0 4:3bfff35c1000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=0000001274d60810:0000483fc1c9be8c:008d940a65a62534:b92c577773172da3
140000 L0 EMBEDDED et=0 10000L/17P B=2880454
150000 L0 EMBEDDED et=0 10000L/40P B=2880454
160000 L0 4:3bfff35c4000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=0000001dd01b452e:0000737538be6f57:00dfe3f89f3978f0:21da6a21dd239621
170000 L0 4:3bfff35c7000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=0000007226831f92:00018eed6e443613:02bdb9afffe4c4ec:3c54071a2be0c2ae
180000 L0 4:3bfff35ca000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=0000006597002eea:00016d111b73092b:029356b85e775c57:1df5a50c209efaf8
190000 L0 4:3bfff35cd000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=000000a164e8c3f0:000224cc0782ad7e:03afd57b4b6379c0:4632626648e0f034
1a0000 L0 4:3bfff35d3000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=0000001ff943c08c:00007b0decaa4b49:00ed27836a9c1214:3129ae602de9a689
1b0000 L0 4:3bfff35d0000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=00000104b915203d:00030f7954462cef:04c6d378f23fb0e7:247a712c60f86d88
1c0000 L0 4:3bfff35d6000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=00000129b842eafd:00033aeef18d622f:04c0f8eaf76e48ae:e6b8b6a0e95d25c7
1d0000 L0 4:3bfff35d9000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=0000003be39b49b1:0000e12e867750b0:01a8476fef4536ef:161f51b518afdc88
1e0000 L0 4:3bfff35df000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=000000d030703c64:00028d4f5550fece:0419c4aa4959dce6:7e2e080b2b210902
1f0000 L0 4:3bfff35dc000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=0000017a9ecd1c91:0003b8d6e9fb68e0:05345ea2bb833303:3ddaef4816a2add9
200000 L0 4:3bfff35e2000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=000001053bee924e:00030279cbfa5d8e:049fd7dc0be8d3f6:e9e2c4b37c058c94
210000 L0 1:3e9ca2ff9000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=000000f5a929538e:0002f089b20b7b7a:04a745232f34e0b6:0f1a55e8c122ee3f
220000 L0 1:3e9ca2ffc000:3000 10000L/1000P F=1 B=2880454/2880454 cksum=000000bfcde4ee84:0002744837a2659b:04169fcec5f496b7:9d2e6f988a1265d7
230000 L0 1:3e9ca3c32000:6000 10000L/2000P F=1 B=2880454/2880454 cksum=0000020131a48a75:000bff2dec017be1:2556ff89489ce698:407be8da89ac03f1
240000 L0 1:3e9ca3c38000:6000 10000L/2000P F=1 B=2880454/2880454 cksum=0000027f080c1a7a:000d838f549a7352:27540ad95c24bd52:f11fdbb52410ca2b
250000 L0 1:3e9ca33fb000:9000 10000L/4000P F=1 B=2880454/2880454 cksum=000007c57b92ef24:00402342f73638ab:553361b156d1e9ac:7baf4a9892d2383a
260000 L0 1:3e9ff932a000:c000 10000L/6000P F=1 B=2880454/2880454 cksum=00000a42864c3478:008b2e9ec63c4127:66f1079df9d2e7ea:8e6245f05c640e98
270000 L0 1:3e9ca3c3e000:6000 10000L/2000P F=1 B=2880454/2880454 cksum=000003e402c21a5a:001076b7accae731:2c263d3a627f5583:ac064926737093b3
ff00000000000000 L4 6:379c6cb82000:3000 20000L/1000P F=1 B=2880454/2880454 cksum=0000008a7c045fed:0002038ee86b9af4:03c274f83446124f:b144d896a7d280ce
ffffc00000000000 L3 3:4dafe787c000:3000 20000L/1000P F=1 B=2880454/2880454 cksum=00000086f6f668d4:0001f7702799b884:03ad9070a51712c4:993ae9f03b173280
fffffff000000000 L2 6:4db99776f000:3000 20000L/1000P F=1 B=2880454/2880454 cksum=00000088def4e1d5:0001fdf6f3974cf5:03b8bff017f41a83:a60666c1a2546041
fffffffffc000000 L1 3:4dafe7741000:3000 20000L/1000P F=1 B=2880454/2880454 cksum=00000086ce979e6e:0001f6fbdab2544b:03acf585b82115d5:98c58a598e4b49a1
ffffffffffff0000 L0 EMBEDDED et=0 200L/6dP B=2880454
segment [0000000000000000, 0000000000280000) size 2.50M
segment [ffffffffffff0000, ffffffffffff0000) size 0
< Last line repeats ad infinitum >
Describe how to reproduce the problem
I don't know. I've only seen this problem once.