[OPIK-7707] [BE] refactor: make dataset version count updates atomic

[JetoPistola]

Details

Replaces the read-modify-write on the dataset version counters with a single incrementing SQL statement, and drops the post-insert re-read of the row that was just written. Both were deferred from OPIK-7705 as follow-ups.

The insert path previously did findById → add in Java → updateCounts, then called getVersionById to re-read the same row: three MySQL round-trips per batch, all inside the per-dataset lock that serialises an upload, so a 100-batch upload paid them 100 times in sequence.

• incrementCounts applies the deltas in the database (SET items_total = COALESCE(items_total, 0) + :delta), so the counter arithmetic no longer depends on withDatasetVersionLock for mutual exclusion. The COALESCE matters: the columns are INT DEFAULT 0 (nullable) and map to boxed Integer, so a NULL would survive a bare increment and later unbox to an NPE — the absolute update this replaces happened to repair a NULL by overwriting it.
• Both delete call sites are converted too. They already passed a plain delta and only needed currentVersion for logging, so insert and delete now write counts the same way — this closes the FR asking that the delete path not be left disagreeing with insert.
• The delete path stays idempotent when the version row is gone. A concurrent DatasetService.delete can remove dataset_versions without holding withDatasetVersionLock; the previous absolute update discarded its result, so that race was a silent 204. It logs and continues rather than turning into a 404. The insert path does still throw — it threw from findById before, so that 404 is existing behaviour.
• updateCounts had no callers left after both paths moved over, so it is removed.
• The returned DatasetVersion is no longer re-read. The resource .block()s and returns 204, and saveBatch maps it to items.size(), so no caller reads its fields; the Mono still emits so saveBatch's map is reached.

Counter values are unchanged. This is a performance and correctness-hardening change with no intended behavioural surface.

Change checklist

• User facing
• Documentation update

Issues

• Resolves #
• OPIK-7707

AI-WATERMARK

AI-WATERMARK: yes

• Tools: Claude Code
• Model(s): Claude Opus 5
• Scope: Traced the call graph, wrote the implementation and the two new concurrency tests, ran the suites and the baseline comparison below.
• Human verification: Pending author review.

Testing

mvn test -Dtest='DatasetVersionResourceTest*'   # 123/123 pass
mvn spotless:apply                              # clean

Scenarios validated:

• Counters unchanged — the whole DatasetVersionResourceTest suite (123 tests) is green, including InsertClassificationCounts (new-vs-update classification), BatchVersioningDeleteTests, DeleteItemsWithVersioning, and MutateLatestVersion. This is the AC that counter values match the previous implementation.
• Concurrency invariant — two new tests in ConcurrentUploads, reusing that class's existing barrier harness so the HTTP calls genuinely overlap: 8 concurrent inserts into one version, and 8 concurrent deletes from one version, each asserting items_total equals the rows actually stored.
• Round-trip reduction — verified by reading the path: the count update is now one statement, and no getVersionById follows it.
• Post-merge — origin/main merged in (no conflicts) and the full 123 re-run green on the merged tree, after the review fixes.

Not verified, stated plainly:

• The two new tests also pass without this change. The per-dataset lock does serialise these writes today, so they pin the invariant rather than demonstrating a fixed race. The AC asked for correctness demonstrated "with the lock removed in a test"; that is not what these do, and doing it properly needs a DAO-level test that bypasses the service lock (no such test class exists for this DAO yet). Flagging rather than claiming the AC is met.
• DatasetsResourceTest$FindDatasets has 30 pre-existing failures on this branch. Confirmed unrelated: the identical 30 fail on a clean tree with all of this PR's changes stashed. Separately, a ClickHouse testcontainer failed to boot on one run — flaky infra, passed on retry.

Documentation

No documentation impact — internal refactor with no API or behavioural change.

[github-actions]

⏱️ pre-commit per-hook timing

Hook	Description	Result	Duration
☕ spotless — java backend	Format Java code	✅	7.42s
Total (1 ran)			7.42s

⏭️ 42 skipped (no matching files changed)

Hook	Description	Result
🐍 trim trailing whitespace — python sdk	Strip trailing whitespace	⏭️
🐍 fix end of files — python sdk	Ensure files end in a newline	⏭️
🐍 ruff — python sdk	Lint + autofix Python (ruff)	⏭️
🐍 ruff-format — python sdk	Format Python code (ruff)	⏭️
🐍 mypy — python sdk	Static type check	⏭️
🤖 trim trailing whitespace — optimizer	Strip trailing whitespace	⏭️
🤖 fix end of files — optimizer	Ensure files end in a newline	⏭️
🤖 check yaml — optimizer	Validate YAML syntax	⏭️
🤖 check json — optimizer	Validate JSON syntax	⏭️
🤖 check toml — optimizer	Validate TOML syntax	⏭️
🤖 check for added large files — optimizer	Block large files (>1MB)	⏭️
🔐 detect private key — optimizer	Block committed private keys	⏭️
🤖 check for merge conflicts — optimizer	Block merge-conflict markers	⏭️
🤖 check for case conflicts — optimizer	Block case-only name clashes	⏭️
🤖 pyupgrade — optimizer	Modernize Python syntax	⏭️
🤖 ruff — optimizer	Lint + autofix Python (ruff)	⏭️
🤖 ruff-format — optimizer	Format Python code (ruff)	⏭️
🤖 mypy — optimizer	Static type check	⏭️
📓 nbstripout — optimizer notebooks	Strip notebook output	⏭️
📝 markdownlint — optimizer	Lint Markdown	⏭️
🔤 codespell — optimizer	Fix common misspellings	⏭️
📊 radon cc — optimizer	Cyclomatic-complexity gate	⏭️
📊 radon raw — optimizer	Raw size metrics gate	⏭️
📊 xenon — optimizer	Fail on complexity thresholds	⏭️
📊 lizard — optimizer	Cyclomatic-complexity gate	⏭️
🧹 vulture — optimizer	Find dead code	⏭️
🛡️ trim trailing whitespace — guardrails	Strip trailing whitespace	⏭️
🛡️ fix end of files — guardrails	Ensure files end in a newline	⏭️
🛡️ ruff — guardrails	Lint + autofix Python (ruff)	⏭️
🛡️ ruff-format — guardrails	Format Python code (ruff)	⏭️
🛡️ mypy — guardrails	Static type check	⏭️
⚓ helm-docs	Regenerate Helm chart README	⏭️
block non-public FE plugins	Block non-public FE plugins	⏭️
🧪 pre-commit wrapper smoke tests	Self-test the wrapper scripts	⏭️
🧪 rebaseline script tests	Self-test the changelog re-baseline script	⏭️
🌐 eslint — frontend	Lint + autofix JS/TS	⏭️
🌐 typecheck — frontend	Whole-project tsc type check	⏭️
📘 eslint — typescript sdk	Lint + autofix JS/TS	⏭️
📘 typecheck — typescript sdk	Whole-project tsc type check	⏭️
⚙️ actionlint — github workflows	Lint GitHub Actions workflows	⏭️
🐳 hadolint — dockerfiles	Lint Dockerfiles	⏭️
🌈 zizmor — github workflows security	Security-scan GitHub Actions workflows	⏭️

[CometActions]

No test needed here.

The counter arithmetic is unchanged — read-modify-write of (total, added, modified, deleted) becomes the same deltas applied in one UPDATE — so the only behaviour that differs is under concurrent writers, and that isn't something an e2e run can reproduce deterministically. Our spec e2e/tests/datasets/dataset-version-counters.spec.ts (@cap:datasets.version-history-view, green 18/18) already asserts items_total/items_added/items_modified after a multi-batch insert, both sequential and with num_threads=8, cross-checks the stored total against the item ids actually in the dataset, and asserts the Version history tab renders it — so a swapped or sign-flipped delta in incrementCounts fails an existing test. The two integration tests you added in DatasetVersionResourceTest cover the race at the layer where it is actually reproducible. Also checked the stub DatasetVersion.builder().id(versionId).build(): both callers do discard it (DatasetsResource returns 204, saveBatch maps to items.size()), so nothing reads the fields you stopped populating.

Run

Advisory, from the QA test radar. Nothing here blocks this PR, and anything it proposes is a draft for review.

[baz-reviewer]

Large deletions corrupt version counters

The Long from countItemsMatchingFilters is narrowed with deletedCount.intValue() at both deletion call sites, and because removeItemsFromVersionByFilters can delete every row for a dataset_id with null/empty filters, counts above Integer.MAX_VALUE wrap so -deletedCount can increase items_total while applying a negative items_deleted delta. Should we reject or bound counts before narrowing, or preserve a wide representation through the storage/API contract?

Want Baz to fix this for you? Activate Fixer

Other fix methods

Prompt for AI Agents

Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/main/java/com/comet/opik/domain/DatasetItemService.java` around
lines 1777-1777, fix the deletion-count handling in `removeItemsFromVersionByFilters`
and its callers: `countItemsMatchingFilters` returns a `Long`, but converting it with
`intValue()` can overflow before `incrementCounts` applies the negative delta. Reject or
explicitly bound counts before narrowing, or preserve a wide numeric type consistently
through the service and persistence/API contract, ensuring oversized deletions cannot
increase `items_total` or apply negative `items_deleted` values.