test(abi): emit the cdylib under cargo test at all, then refuse a stale one

Two defects, both of which made this repo's only over-the-ABI coverage inert.

1. store-valkey-plugin declared crate-type = ["cdylib"] only. With cdylib-only, `cargo
   test` builds a "0 tests" harness for the package and NEVER produces the .dylib/.so, so
   tests/e2e.rs found nothing and silently skipped — green, with zero coverage of the
   durable valkey path. Verified: with cdylib-only, no deps/libbusbar_store_valkey_plugin.dylib
   appears even after touching src/lib.rs; with rlib added, cargo emits it on every build
   that recompiles the lib. The sibling sqlite/postgres/mysql plugin manifests already
   carry both crate-types for exactly this reason; this one had drifted.

2. plugin_path() looked in target/<profile>/, which only `cargo build` refreshes, and
   returned None -> "skip:" -> GREEN when absent. It now reads only deps/<name> (the
   output of the build graph that produced this test binary), panics instead of skipping,
   and asserts the artifact is no older than any workspace src/**/*.rs with a message that
   says STALE ARTIFACT rather than letting it read as a durability failure.

Also: the persistence test reused a fixed key id and relied on delete_key to clean up.
delete_key TOMBSTONES by contract, so get_key still answers afterwards — against a re-used
Valkey the test passed with the plugin's put_key stubbed to Ok(()). Proven: that stub
passed against a re-used instance and failed against a flushed one. A per-run key id makes
every read answerable only by this run's writes.

Author: MattJackson

store-valkey-plugin/Cargo.toml

@@ -9,7 +9,16 @@ description = "The Valkey store as a droppable busbar plugin — a cdylib export
 license = "Apache-2.0"
 
 [lib]
-crate-type = ["cdylib"]
+# cdylib for the C ABI delivery; `rlib` is REQUIRED for `cargo test` to emit the cdylib artifact AT
+# ALL. With `cdylib`-only, cargo builds a trivial "0 tests" harness for this package under `cargo
+# test` and never produces the .dylib/.so — so `tests/e2e.rs`, the ONLY over-the-ABI coverage of the
+# durable valkey path, found nothing and silently skipped, reporting GREEN with zero coverage. The
+# sibling store-postgres-plugin/store-mysql-plugin/store-sqlite-plugin manifests already carry both
+# crate-types for exactly this reason; this one had drifted. Verified, not assumed: with `cdylib`
+# only, `cargo test -p busbar-store-valkey-plugin` leaves no
+# `target/debug/deps/libbusbar_store_valkey_plugin.dylib` even after touching src/lib.rs; with
+# `rlib` added, cargo emits it on every build that recompiles the lib.
+crate-type = ["cdylib", "rlib"]
 
 [dependencies]
 # busbar-store-valkey is now a SAME-REPO sibling crate (this plugin brings 100% of what it needs — the

store-valkey-plugin/tests/e2e.rs

@@ -36,16 +36,106 @@ use std::time::{Duration, Instant};
 /// explicit signing key to mint virtual keys; busbar no longer auto-generates one.
 const TEST_SIGNING_KEY: &str = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
 
-/// Locate the built `busbar_store_valkey_plugin` cdylib in the target dir, derived from the test
-/// binary's own path (robust to a custom `CARGO_TARGET_DIR`). `None` if it hasn't been built —
-/// under `cargo test` (which builds the whole package including the cdylib target before running
-/// tests) it is always present, so this only guards against unusual invocations.
-fn plugin_path() -> Option<std::path::PathBuf> {
-    let exe = std::env::current_exe().ok()?; // .../target/<profile>/deps/e2e-<hash>
-    let profile_dir = exe.parent()?.parent()?; // .../target/<profile>
+/// Locate the cdylib THIS `cargo test` invocation just built — never a leftover artifact.
+///
+/// This looks ONLY in `target/<profile>/deps/`, never `target/<profile>/`, and that distinction is
+/// the whole point of this function.
+///
+/// `cargo` emits the lib target's cdylib into `deps/` as part of the very build graph that produces
+/// this test binary (this package's lib unit is compiled with BOTH declared crate-types — see
+/// `[lib] crate-type = ["cdylib", "rlib"]` in Cargo.toml), so `deps/libbusbar_store_valkey_plugin.dylib` is by construction up to
+/// date with the source tree under test. Cargo only *uplifts* a copy to `target/<profile>/` for
+/// `cargo build`, NEVER for `cargo test`. A lookup in `target/<profile>/` therefore reads an
+/// artifact that nothing in this test's dependency graph refreshes: whatever some earlier `cargo
+/// build` left there, from any commit — or nothing at all.
+///
+/// Both outcomes of that are lies about durability, and the second is the dangerous one:
+///   * NOTHING there  -> the old code `return`ed with a "skip:" line and reported GREEN. That is how
+///     `cargo test` can pass with ZERO over-the-ABI coverage of the durable store path.
+///   * STALE artifact -> a cdylib built before an ABI change answers every write `Ok(())` and every
+///     read empty, which is BYTE-FOR-BYTE the signature of the unrelayed-seam defect this file
+///     exists to catch (that defect was real: `DynStore`'s `impl Store` overrode 24 methods, none of
+///     them the task/call-log methods, so `put_task` took the accept-and-keep-nothing trait
+///     default). RED on a stale artifact is indistinguishable from RED on the real bug — and an
+///     artifact NEWER than a regression reports GREEN while the shipped ABI is broken. Proven, not
+///     theorised: with a regressed plugin in the tree and a good cdylib in `target/debug/`, the old
+///     lookup passed and this one fails.
+///
+/// Same hazard, and the same reasoning, as the engine's `crates/busbar/Cargo.toml` dev-dependency on
+/// `busbar-store-example-plugin`: keep the cdylib in the build graph so no test can judge a stale
+/// one. Here the plugin's lib IS this package, so that graph edge already exists — what was missing
+/// was reading the artifact that edge actually produces.
+///
+/// Panics rather than skipping: a missing cdylib under `cargo test` means the build graph changed
+/// shape, and the only honest report of that is a failure, not a silent pass.
+/// The newest mtime across every workspace crate's `src/` — "how fresh must a cdylib be to be the
+/// one this source tree describes".
+///
+/// Deliberately ONLY `src/**/*.rs` of each workspace member: editing a `tests/` file or a
+/// `[dev-dependencies]` line recompiles the test binary but NOT the lib, so including those would
+/// fail a perfectly current cdylib.
+fn newest_source_mtime() -> std::time::SystemTime {
+    fn walk(dir: &std::path::Path, newest: &mut std::time::SystemTime) {
+        let Ok(rd) = std::fs::read_dir(dir) else { return };
+        for e in rd.flatten() {
+            let p = e.path();
+            if p.is_dir() {
+                walk(&p, newest);
+            } else if p.extension().is_some_and(|x| x == "rs") {
+                if let Ok(m) = e.metadata().and_then(|m| m.modified()) {
+                    if m > *newest {
+                        *newest = m;
+                    }
+                }
+            }
+        }
+    }
+    let ws_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
+        .parent()
+        .expect("the plugin crate always sits under the workspace root");
+    let mut newest = std::time::SystemTime::UNIX_EPOCH;
+    for e in std::fs::read_dir(ws_root).into_iter().flatten().flatten() {
+        let src = e.path().join("src");
+        if src.is_dir() {
+            walk(&src, &mut newest);
+        }
+    }
+    newest
+}
+
+fn plugin_path() -> PathBuf {
+    let exe = std::env::current_exe().expect("current_exe"); // .../target/<profile>/deps/<test>-<hash>
+    let deps_dir = exe.parent().expect("the test binary always lives in deps/");
     let name = plugin_library_filename("busbar_store_valkey_plugin");
-    let candidate = profile_dir.join(&name);
-    candidate.exists().then_some(candidate)
+    let fresh = deps_dir.join(&name);
+    assert!(
+        fresh.exists(),
+        "the store-valkey-plugin cdylib is not at {}, where cargo emits it for the same build that produced \
+         this test binary. Refusing to fall back to target/<profile>/ (an artifact only `cargo \
+         build` refreshes) or to skip: judging a stale cdylib is exactly how an unrelayed plugin \
+         ABI reads as green.",
+        fresh.display()
+    );
+    // FRESHNESS, ASSERTED — not assumed. Under `cargo test` the artifact above is rebuilt by the
+    // same graph that built this binary (proven: delete it, re-run, cargo re-emits it). But this
+    // test binary can also be executed DIRECTLY out of `deps/`, where nothing rebuilds anything,
+    // and a stale cdylib there produces empty reads — indistinguishable from the unrelayed-ABI
+    // defect. So compare it against the sources and fail with a message that says STALE ARTIFACT,
+    // explicitly NOT a durability verdict.
+    let built = std::fs::metadata(&fresh)
+        .and_then(|m| m.modified())
+        .expect("cdylib mtime");
+    let newest_src = newest_source_mtime();
+    assert!(
+        built >= newest_src,
+        "STALE ARTIFACT — THIS IS NOT A DURABILITY FAILURE. {} predates this workspace's sources, \
+         so it cannot answer for the code in the tree; a pre-change cdylib returns empty for every \
+         read, which reads exactly like an unrelayed plugin ABI. Run `cargo build -p {}` (or just \
+         `cargo test`, which rebuilds it) and re-run.",
+        fresh.display(),
+        "busbar-store-valkey-plugin"
+    );
+    fresh
 }
 
 /// The live `VALKEY_URL`, mirroring `busbar-store-valkey`'s own `live_store()` gating discipline
@@ -116,34 +206,47 @@ fn ledger() -> UsageLedger {
 /// This is the proof that `store: valkey` operations over the ABI aren't silently no-ops.
 #[test]
 fn load_and_exercise_valkey_plugin_persists_to_real_valkey_across_reopen() {
-    let Some(path) = plugin_path() else {
-        eprintln!("skip: valkey plugin cdylib not built (run under `cargo test`)");
-        return;
-    };
+    let path = plugin_path();
     let Some(url) = valkey_url() else {
         return;
     };
     let cfg = serde_json::json!({ "url": url }).to_string();
 
     // Isolate from any prior run against a persistent (non-CI) Valkey instance.
+    //
+    // A FRESH ID PER RUN, not a fixed one plus a `delete_key`: `delete_key` TOMBSTONES the row (it
+    // is a soft delete by contract — see `Store::delete_key`), so `get_key` still answers with it
+    // afterwards. Against a re-used Valkey that made this test pass even when the plugin's
+    // `put_key` wrote NOTHING at all: every read below was satisfied by the previous run's row.
+    // Proven, not theorised — a `put_key` stubbed to `Ok(())` passed this test against a re-used
+    // instance and failed it against a flushed one. A per-run id makes every read here answerable
+    // only by THIS run's writes.
     let direct = ValkeyStore::connect(&url).expect("connect directly to seed/clean up");
-    let _ = Store::delete_key(&direct, "vk_e2e_dlopen");
+    let vk_id = format!(
+        "vk_e2e_dlopen_{}_{}",
+        std::process::id(),
+        std::time::SystemTime::now()
+            .duration_since(std::time::UNIX_EPOCH)
+            .unwrap()
+            .as_nanos()
+    );
+    let vk_id = vk_id.as_str();
 
-    let vk = key("vk_e2e_dlopen");
+    let vk = key(vk_id);
 
     {
         let store = load_store(&path, &cfg).expect("load valkey plugin against a real Valkey");
         store.put_key(&vk).expect("put_key over the ABI");
         store
-            .put_usage("vk_e2e_dlopen", 200, &ledger())
+            .put_usage(vk_id, 200, &ledger())
             .expect("put_usage over the ABI");
         assert_eq!(
             store
-                .get_key("vk_e2e_dlopen")
+                .get_key(vk_id)
                 .expect("get_key over the ABI")
                 .expect("present in the same session")
                 .id,
-            "vk_e2e_dlopen"
+            vk_id
         );
         // `store` (and the `RawPlugin` it wraps) drops here, running `busbar_close` and dropping
         // the plugin's own `ValkeyStore`/connection — the data must be durably in Valkey after
@@ -155,13 +258,13 @@ fn load_and_exercise_valkey_plugin_persists_to_real_valkey_across_reopen() {
     // on the first instance still being alive.
     let reopened = load_store(&path, &cfg).expect("re-load valkey plugin against the same URL");
     let got = reopened
-        .get_key("vk_e2e_dlopen")
+        .get_key(vk_id)
         .expect("get_key after reopen")
         .expect("the key must survive a full plugin close + reopen against the same Valkey");
     assert_eq!(got.group.as_deref(), Some("infra"));
     assert_eq!(got.labels.get("env").map(String::as_str), Some("e2e"));
     let usage = reopened
-        .get_usage("vk_e2e_dlopen", 200)
+        .get_usage(vk_id, 200)
         .expect("get_usage after reopen");
     assert_eq!(usage.requests, 5, "usage ledger must survive the reopen");
     let t = usage
@@ -175,22 +278,22 @@ fn load_and_exercise_valkey_plugin_persists_to_real_valkey_across_reopen() {
     // plugin's `put_key`/`put_usage` over the ABI were silent no-ops (or wrote somewhere other
     // than the configured Valkey), this independent reader would come back empty even though the
     // reopen-via-plugin check above passed.
-    let direct_key = Store::get_key(&direct, "vk_e2e_dlopen")
+    let direct_key = Store::get_key(&direct, vk_id)
         .expect("get_key via the direct connection")
         .expect("the key must be physically present in Valkey, bypassing the plugin");
     assert_eq!(direct_key.name, "e2e-dlopen-key");
     assert_eq!(
         direct_key.allowed_scopes,
         Some(vec![busbar_api::ScopeRef::pool("p")])
     );
-    let direct_usage = Store::get_usage(&direct, "vk_e2e_dlopen", 200)
+    let direct_usage = Store::get_usage(&direct, vk_id, 200)
         .expect("get_usage via the direct connection");
     assert_eq!(
         direct_usage.requests, 5,
         "usage must be physically present in Valkey, not just cached in-process by the plugin"
     );
 
-    let _ = Store::delete_key(&direct, "vk_e2e_dlopen");
+    let _ = Store::delete_key(&direct, vk_id);
 }
 
 /// END-TO-END FAILURE: an `open()` config that cannot produce a usable store — malformed JSON, a
@@ -199,10 +302,7 @@ fn load_and_exercise_valkey_plugin_persists_to_real_valkey_across_reopen() {
 /// case here fails before (or instead of) actually connecting.
 #[test]
 fn load_and_exercise_valkey_plugin_bad_config_fails_over_abi() {
-    let Some(path) = plugin_path() else {
-        eprintln!("skip: valkey plugin cdylib not built (run under `cargo test`)");
-        return;
-    };
+    let path = plugin_path();
 
     let err = load_store(&path, "{ not json")
         .err()
@@ -356,10 +456,7 @@ fn admin_test_valkey_url(base: &str) -> String {
 fn admin_api_installs_the_valkey_plugin_and_writes_land_in_real_valkey() {
     let Some(base_url) = valkey_url() else { return };
     let url = admin_test_valkey_url(&base_url);
-    let Some(so_path) = plugin_path() else {
-        eprintln!("skip: valkey plugin cdylib not built");
-        return;
-    };
+    let so_path = plugin_path();
 
     // Isolate from any prior run against a persistent (non-CI) Valkey instance.
     let direct = ValkeyStore::connect(&url).expect("connect directly to seed/clean up");