Ingest hangs permanently under sustained sequential bulk upsert load (RocksDB storage engine)

Update 2026-07-20: reproduced again on harper@5.1.22, with significant new diagnostic evidence (proof the write actually commits; a full thread dump showing the entire process idle, including the HTTP worker thread; zero pending libuv requests anywhere) that strongly confirms this is a lost native completion callback. Also confirmed unrelated to and unfixed by PR #1696, a real but different merged fix in this version range. See 2026-07-20-reproduction-5.1.22.md for the full write-up -- don't rely on this README alone for current status.

Summary

Under sustained, sequential (not concurrent) bulk upsert batches to a single table, a Harper process will — unpredictably, but reproducibly under load — get stuck on exactly one upsert operation: the HTTP/in-process call never returns a response and never errors. No exception is thrown, nothing is logged as an error, and the Harper process becomes otherwise idle (0% CPU, all threads parked) while continuing to answer other, unrelated requests normally. The only way out is killing and restarting the process; the stuck ingest job then wedges an internal ingest lock (application-level, held past its own stated TTL, since it's never released via the code path that only runs on success/failure).

Status: confirmed, and still present as of 5.1.22. Reproduced independently against a clean harper@5.1.18 install with no application code involved — see "Standalone reproduction" below. Also confirmed unchanged between 5.1.15 (original occurrences) and 5.1.18 (no relevant commits to resources/DatabaseTransaction.ts, no rocksdb-js version change). Reproduced again on 5.1.22 on 2026-07-20 with substantially stronger diagnostic evidence — see 2026-07-20-reproduction-5.1.22.md.

Environment

• Harper: 5.1.15 (original occurrences), 5.1.18 (first standalone repro, below), and 5.1.22 (second standalone repro — see 2026-07-20-reproduction-5.1.22.md; confirmed unchanged in this code path despite 5.1.22 including the unrelated PR #1696 fix)
• Storage engine: RocksDB (@harperfast/rocksdb-js / @harperfast/rocksdb-js-darwin-arm64 2.3.0, matches package.json's ^2.3.0 — verified no version drift)
• OS: macOS 26.5.1 (Darwin 25.5.0), arm64
• Node: v24.14.0
• THREADS_COUNT: 1 (both the original occurrence and the standalone repro)

Original occurrence (application context)

Found while running a large (~50-60M row) catalog ingest through a custom Harper component that calls the in-process operation() global with { operation: 'upsert', table, records } in batches of ~1000 records, applied sequentially (each batch awaited before the next begins — never concurrent). Three occurrences across one session, each on a fresh push of an already-successfully-packed bundle:

#	Table in flight at hang	Cumulative rows written this run	Time-to-hang
1	ProductVariantCustomAttribute (batch ~7.17M/40M into that table)	~46M	~58 min
2	ProductMaster → transitioning to ProductTranslation	~1M	~20 min (recovered mid-transition on its own before stalling again on retry)
3	ProductTranslation (batch 708k)	<1M	~20 min

Notably, occurrence 2 and 3 happened on retries of the same long-lived Harper process after occurrence 1 was cleared — each retry stalled sooner than the last, suggesting something cumulative across the process's lifetime (not purely a function of the current transaction's own size).

Confirmed diagnostic signature (all three occurrences)

• Application-level CatalogFeedRun status record stays running forever; no completedAt, no errorMessage.
• The calling code (inProcessWriter.ts) wraps every operation({ operation: 'upsert', ... }) call in a 300-second Promise.race timeout. This timeout never fires, even after the operation had been stuck for 14+ hours in occurrence 1. This rules out "just slow" — the call genuinely never settles (neither resolves nor rejects).
• ps/top on the Harper process: 0% CPU, state sleeping, not spinning.
• A native thread sample (sample <pid>) taken during the hang shows every thread idle: main thread parked in uv__io_poll/kevent (event loop has nothing to do), RocksDB's own background thread pool (6 BGThreadWrapper threads) parked on condvar wait with nothing queued. Nothing is executing, nothing is blocked mid-computation.
• Despite that, lightweight concurrent requests to the same process keep succeeding during the hang (status-poll search_by_hash queries against an unrelated table return normally) — so this is not a fully frozen event loop, and not a global deadlock. One specific in-flight write operation's completion signal is lost; everything else is fine.
• Harper's own log repeats "JavaScript execution has taken too long and is not allowing proper event queue cycling..." around the stall window, then goes silent (no further errors, no further table-load progress logged) while search_by_hash polls from the client's own status checks keep appearing and succeeding.
• RocksDB's own internal stats (database/*/LOG) show zero write-stall (Cumulative stall: 0.0 percent, rocksdb.stall.micros COUNT: 0) and its periodic background-compaction task keeps firing on schedule — so this is not RocksDB's own backpressure/stall mechanism; whatever is stuck is upstream of that.

Standalone reproduction (no application code, no RL-specific data)

Reproduced. Run against a clean harper@5.1.18 install, batches of 1000 synthetic records (new distinct keys each batch, ~200 bytes/record), sequential, no concurrency, no application code beyond the plain Operations API:

• Ran cleanly for 62,510,000 rows / 62,510 batches / ~57 minutes at a steady ~50ms/batch (~20k rows/sec), then batch 62,511 (the next one) never returned.
• At detection time, the process had been stuck for at least several minutes (see caveat below) with zero progress, zero errors, zero log output beyond what's described next.
• Diagnostics captured at the stall (repro-logs/stall-batch62510-*/) show the identical signature to all three original occurrences: top/ps report the process at 0% CPU, state sleeping; a sample <pid> 5 thread dump shows the main thread parked in uv_run/kevent (event loop idle, nothing queued) and RocksDB's background thread pool idle on condvars with no queued work. Nothing is computing; nothing is blocked mid-instruction. It is not busy — it has simply stopped being told to continue.
• The Harper process was deliberately left running (not killed) after the stall was detected, for live follow-up inspection.

Measurement caveat: the harness's own stall-timer was configured for 90s but the logged stalled_for_ms at detection was ~944,000ms (~15.7 min). This is very likely the harness's own Node process being suspended/throttled by macOS App Nap / background-process scheduling (a known OS behavior for long-running background CLI tools, unrelated to Harper) rather than a 15-minute delay in Harper's own state — but noting it plainly rather than asserting false precision. What's solid regardless of that caveat: the batch that started at 01:15:50 had still not returned by the time this was written, and the diagnostic signature (0% CPU, idle threads, no errors) is identical across all four occurrences (3 original + this one) regardless of the exact stall-detection latency.

To isolate this from any possibility of an application-level bug, a minimal, self-contained repro (repro.js, attached) does the following against a freshly-installed harper@5.1.18:

1. npm install harper@5.1.18 in an empty directory (published npm package, not a source build).
2. harper install + harper run with a dedicated ROOTPATH and an isolated $HOME (harper install tracks a single canonical install via $HOME/.harperdb/hdb_boot_properties.file, independent of --ROOTPATH — worth knowing if you try to run multiple harper instances on one machine for testing).
3. Creates one plain table (BulkHangRepro, id hash attribute, a handful of scalar columns).
4. Runs the plain HTTP Operations API (POST / {operation: 'upsert', database, table, records}), in batches of 1000 synthetic records with strictly new, distinct keys per batch (row-<sequential-index>) — matching the real ingest's pattern (new rows, not repeated updates to the same keys) — sequentially, one awaited batch after another, no concurrency.
5. Watches wall-clock time per batch; if one batch exceeds 90s (vs. the ~40-135ms/batch normal rate observed), it's flagged a stall and diagnostics are captured automatically (ps, top -pid, sample <pid> 5, lsof -p <pid>, and the tail of Harper's own log) into repro-logs/stall-batch<N>-<timestamp>/, and the Harper process is deliberately left running (not killed) for live follow-up inspection.

Run log: repro-logs/progress.log. Harper's own log: repro-logs/harper.log.

Result: reproduced on the first run, no tuning needed — see above. To reproduce again: STALL_MS=90000 TOTAL_BATCHES=100000 node repro.js from a clean checkout (delete hdb/, home/, and repro-logs/ first if re-running). Expect it to take somewhere in the range of 20 minutes to an hour of sustained load before stalling, based on both this run and the three original occurrences — this is not a fast/deterministic repro, but it is a reliable one.

Code-level observations (Harper source, resources/DatabaseTransaction.ts)

This is the transaction-commit path every upsert goes through (dataLayer/insert.ts → harperBridge.upsertRecords → resources/transaction.ts:transaction() → DatabaseTransaction.commit()).

• commit() already has real, recently-hardened retry handling for RocksDB ERR_BUSY / ERR_TRY_AGAIN rejections (git history: #1223 "retry ERR_TRY_AGAIN commits instead of dropping them silently", #1252 "never drop source-applied writes on conflict", referencing a prior incident #308 where such a rejection "propagated out of the unawaited onCommit() handler as an unhandled rejection and the write was silently dropped — records lost mid-copy"). For non-sourceApply transactions (which this ingest path is — it's a plain in-process operation() call, not replication/source-apply) this retry is capped at 40 attempts with a ≤1s backoff each, so a genuinely-retried conflict would surface a loud ServerError within well under a minute. That doesn't match what we observe — no error ever appears, for 14+ hours in one case.
• The most recent commit touching this method (4ba9fd9c5, "Reconcile commit() type for rocksdb-js 2.2.0 coordinated-retry sentinel", 2026-06-18) is a type-only cast around a new rocksdb-js 2.2+ feature: commit() can now resolve (not reject) to a RETRY_NOW sentinel when coordinatedRetry: true is passed to the transaction and an IsBusy conflict occurs. The commit message states Harper doesn't pass coordinatedRetry, so this "never" applies — and reading rocksdb-js's own type docs confirms that's an opt-in, default-false option, so on paper this shouldn't be reachable here. Flagging it anyway because: (a) it's the single most recent change in exactly this method, (b) its own commit message says "Full RETRY_NOW handling is a separate change" (i.e., acknowledged as incomplete/deferred), and (c) the rocksdb-js type docs describe the underlying mechanism as the native layer "parking on a VT slot before resolving" — i.e., a native-side wait state contingent on a different transaction's commit/release, which is a plausible shape for exactly the kind of "everything looks idle but one promise never settles" hang observed here if that parking is ever entered unexpectedly (e.g. via a concurrent internal/background transaction that does touch this path, or a native-side accounting bug that isn't strictly gated on the JS-level coordinatedRetry flag actually being requested by the caller that gets stuck).
• There is no timeout/watchdog anywhere in commit()'s success path for the underlying native transaction.commit() promise itself. checkOverloaded() exists and can reject a new transaction if an existing outstandingCommit has been pending too long (STORAGE_MAXTRANSACTIONQUEUETIME, default 45s) — but it protects concurrent, later callers from queueing behind a slow commit; it does nothing for the stuck commit itself, and in a purely sequential single-writer ingest (this scenario, and the standalone repro), there is no later caller to trigger it in the first place. If the native commit() promise is ever truly lost, nothing in this file can recover it.

Suggested areas to investigate (Harper side)

1. Whether transaction.commit()'s returned promise can, under any internal/background transaction interaction, ever fail to settle at all (not reject, not resolve) — independent of whether the caller requested coordinatedRetry. The "VT slot parking" behavior described in rocksdb-js's own docs for the coordinated-retry path is the closest known mechanism in this codebase for a promise that's contingent on another transaction's state.
2. Whether any Harper-internal write path (replication apply, index/analytics maintenance, compaction-adjacent bookkeeping) passes coordinatedRetry: true and could leave a shared native resource (a "VT slot") in a state that a completely unrelated, non-coordinated transaction then blocks on.
3. Whether rocksdb-js's native N-API completion callback (whatever fires the JS promise commit() returns) can be dropped/lost under sustained load — e.g. a uv_async_t handle or napi_threadsafe_function reused/closed prematurely, or a completion queued against a thread-pool slot that's already been reclaimed. Strongly confirmed by direct evidence as of the 2026-07-20 5.1.22 reproduction — see 2026-07-20-reproduction-5.1.22.md: the stuck write is proven to have committed in full, and a live full-process thread dump plus a libuv active-handle check show every thread (including the one that should generate the HTTP response) completely idle with zero pending requests anywhere. This was a plausible theory before; it now has hard evidence behind it and should be the primary lead.
4. A defensive fix on the Harper (JS) side regardless of native root cause: wrap the transaction.commit() call itself in a bounded timeout that throws a distinguishable error (not silently retried as ERR_BUSY/ERR_TRY_AGAIN, since we don't know it's actually a conflict) if it hasn't settled after some generous duration — converting a permanent silent hang into a bounded, loud failure the caller can act on. Caveat we'd flag to the Harper team: this is a mitigation, not a fix, and isn't safe to ship blind — if the original native commit later actually completes after the timeout-triggered retry/abort path has already run, that's a duplicate-write/consistency hazard, exactly the class of bug this same file's sourceApply "never drop a write on a transient conflict" comment is already protecting against for a different case.

Attachments

• repro.js — standalone repro harness (self-contained, no dependency on any private app code)
• repro-logs/progress.log — batch-by-batch timing log from the run
• repro-logs/harper.log — Harper's own stdout/stderr for the repro instance
• repro-logs/stall-*/ — diagnostics captured automatically at the moment of a detected stall, if any
• 2026-07-20-reproduction-5.1.22.md — second reproduction, against harper@5.1.22, with significantly stronger diagnostics (proof of commit success, full live thread dump, libuv active-handle check). Read this for the current state of the investigation.