DAOS-13563 vos: fix engine crash on iod_nr==0 fetch of flat-dkey objects

[hgichon]

Summary

dkey_fetch() crashes the engine on a CHECK_EXISTENCE / SET_TS_ONLY fetch (iod_nr == 0) of a flat-dkey object (array / flat KV on a pool with VOS_POOL_FEAT_FLAT_DKEY). vos_ioc_create() permits iod_nr == 0 for those flags, leaving ioc->ic_iods == NULL, but the flat path in dkey_fetch() dereferences ic_iods[0] without an iod_nr guard:

• SUBTR_EVT computation — ic_iods[0].iod_type → SIGSEGV (NULL + 0x18)
• KREC_BF_NO_AKEY branch — iod_set_cursor(ioc, 0) trips D_ASSERT(sgl_at < ic_iod_nr) → SIGABRT, and ic_iods[0] is then dereferenced NULL in fetch_value()

Regression introduced by ae105cb ("DAOS-13563 vos: Implement flat dkey for array and flat KV object types", #13175). Present on master and any branch carrying that commit.

Fix

Guard both flat-path ic_iods[0] accesses with ic_iod_nr > 0. The non-flat path is already guarded by its for (i = 0; i < ic_iod_nr; i++) loop; key existence is established earlier by key_tree_prepare() + key_ilog_check(), so the value fetch is correctly skipped when there are no iods.

How it was found

versitygw's native DAOS backend issues an S3 HEAD / conditional-PUT existence probe as a CHECK_EXISTENCE fetch (iod_nr == 0) against a flat DFS object. This crashed the target engine (SIGSEGV/SIGABRT), surfacing to the S3 client as dc_rw_cb ... DER_TIMEDOUT. Observed backtrace (release build):

vos_fetch_begin+0xa7e   -> SIGSEGV, "Failing for address: 0x18"   (SUBTR_EVT path)
vos_io.c:1540 iod_set_cursor() Assertion 'sgl_at < ioc->ic_iod_nr' failed -> SIGABRT (after guarding the first)
  <- obj_prep_fetch_sgls <- ds_obj_rw_handler

Test

Adds VOS283 to src/vos/tests/vts_io.c (io test group), reproducing the crash:

• Creates a flat-dkey array object (DAOS_OT_ARRAY_BYTE, uint64 dkey) and updates it, so the object/dkey exists. The earlier draft used a string dkey, which an array object rejects with -DER_INVAL in io_test_obj_update() before the fetch under test was reached — that was the "key format" issue, not a fault in the fix.
• Issues the iod_nr == 0 VOS_OF_FETCH_CHECK_EXISTENCE fetch — the crash trigger.
• Asserts the no-crash contract: rc == 0 || rc == -DER_NONEXIST. With no iods the flat-array dkey's subtree type is unknown to the probe, so a clean DER_NONEXIST is legitimate; reaching a clean rc rather than a SIGSEGV/SIGABRT is what the test verifies.
• A teardown restores the shared arg->oid so a failure can't perturb the rest of the io suite.

Verified on hardware (md-on-SSD build, PMEMOBJ_CONF=sds.at_create=0), same vos_tests binary with only libvos.so swapped:

• unfixed libvos.so → VOS283 ... Segmentation fault(11) (exit 1)
• fixed libvos.so → VOS283 OK, io group 21/21 PASS

Also reproduced the SIGSEGV on pure upstream master (the merge-base commit, no local changes) with the same test, confirming this is the upstream regression rather than a local artifact.

---

Signed-off-by: hgichon hgichon@gmail.com

[github-actions]

Ticket title is 'Flat-KV object '
Status is 'Resolved'
Labels: 'md_on_ssd2'
https://daosio.atlassian.net/browse/DAOS-13563

[hgichon]

@jolivier23 would you be able to review this? It's a small regression fix in the flat-dkey fetch path you introduced in ae105cb ("DAOS-13563 vos: Implement flat dkey for array and flat KV object types", #13175).

Context: dkey_fetch()'s ic_skip_akey_support branch dereferences ioc->ic_iods[0] (and the KREC_BF_NO_AKEY path calls iod_set_cursor(ioc, 0)) without an ic_iod_nr guard. A CHECK_EXISTENCE / SET_TS_ONLY fetch is allowed to pass iod_nr == 0 (see vos_ioc_create()), so on a flat-dkey object (array / flat KV) this NULL-derefs → SIGSEGV, and after guarding the first deref, iod_set_cursor()'s D_ASSERT(sgl_at < ic_iod_nr) → SIGABRT. The non-flat path is naturally guarded by its for (i = 0; i < ic_iod_nr; i++) loop, and existence is already established by key_tree_prepare() + key_ilog_check(), so the fix just guards both flat-path accesses with ic_iod_nr > 0.

Originally hit via versitygw's native DAOS backend (an S3 HEAD / conditional-PUT issues a CHECK_EXISTENCE fetch on a flat DFS object), which crashed the target engine.

One thing I'd appreciate your eye on: the VOS283 reproduction test is a draft — I based it on the existing vts_io.c helpers and the default VTS pool (which enables VOS_POOL_FEAT_FLAT_DKEY), but the flat-object update setup should be validated against however you prefer to exercise flat-dkey objects in the unit tests. Happy to rework it to match. Thanks!

[hgichon]

Update: I dropped the draft VOS283 unit test from this PR — it's now fix-only (src/vos/vos_io.c).

On actually running it in vos_tests, the draft didn't hold up: it created an DAOS_OT_ARRAY_BYTE object but generated string dkeys, so the setup io_test_obj_update() failed at btr_verify_key() ("invalid integer key, expected: 8, got: 7") long before reaching the iod_nr == 0 fetch — so it never exercised the crash, and (on the assert longjmp) it also left arg->oid unrestored, cascading into the following io_tests. Getting a correct flat-dkey reproduction in VTS needs the right object/key setup and a pool with VOS_POOL_FEAT_FLAT_DKEY, which is really your area.

The fix itself is verified independently: the two guarded derefs are confirmed in the compiled libvos_srv.so (disassembly), and the crash + the fix's effect were reproduced on a live cluster via versitygw's DAOS backend (S3 HEAD/existence probe on a flat DFS object → target-engine SIGSEGV/SIGABRT before the fix; clean after, with our projid-quota suite passing and no engine abort).

If you'd like a unit test in this PR, I'm happy to add one — could you point me at the preferred way to stand up a flat-dkey (array / flat-KV) object in vts_io.c (and confirm the VTS pool enables FLAT_DKEY)? Otherwise happy to leave the regression coverage to your test infra. Thanks!

[hgichon]

Re-added the VOS283 regression test (it was dropped earlier as a draft). The draft was failing in setup, not because of the fix: an array object (DAOS_OT_ARRAY_BYTE) requires a uint64 dkey, but the draft used a string dkey, so io_test_obj_update() returned -DER_INVAL before the fetch under test was ever reached — hence the "key format" trouble I mentioned.

Corrected test (vos_tests -i, io group):

• Create a flat-dkey array object (DAOS_OT_ARRAY_BYTE, uint64 dkey) and update it, so the object/dkey exists.
• Issue the iod_nr == 0 VOS_OF_FETCH_CHECK_EXISTENCE fetch — the crash trigger.
• Assert the no-crash contract: rc == 0 || rc == -DER_NONEXIST. With no iods the flat-array dkey's subtree type is unknown to the probe, so a clean DER_NONEXIST is legitimate here; the point is reaching a clean rc instead of a SIGSEGV/SIGABRT.
• A teardown restores the shared arg->oid so a failure can't perturb the rest of the io suite.

Verified on hardware (md-on-SSD build, PMEMOBJ_CONF=sds.at_create=0), same vos_tests binary with only libvos.so swapped:

• unfixed libvos.so → VOS283 ... Segmentation fault(11), process exit 1
• fixed libvos.so → VOS283 OK, io group 21/21 PASS

I also physically reproduced the SIGSEGV on pure upstream master (the merge-base commit, none of my local changes) with the same test, confirming this is an upstream regression from ae105cb rather than a local artifact.