Reuse CRC when appending index components

[pkolaczk]

Fixes #10873

[eolivelli]

Do we have unit tests that cover this change?

[pkolaczk]

Yes, there is.
NativeIndexDDLTest.verifyRebuildCorruptedFiles.
And it is failing. Huh, checking that.

[pkolaczk]

Ok, apparently our usage is a bit more complex.
I made the implementation safer and less sensitive to usage patterns.

[pkolaczk]

@eolivelli can you have another look? I hardened it. It's not perfect but I can only go so far in Java without affine types ;)

[eolivelli]

I think that the current approach works, but I would feel more confident if we add some unit tests about concurrent access.

[pkolaczk]

I think that the current approach works, but I would feel more confident if we add some unit tests about concurrent access.

There are plenty of scenarios tested in NativeIndexDDLTest, which also test for things like e.g. interrupting index build in the middle or parallel index builds.
Also, not sure what kind of concurrent testing I should perform on a class that's not thread safe anyways.
You must not run two writers on the same file.

[eolivelli]

a class that's not thread safe anyways

The class is not thread safe, but we are adding some shared state, so it is not a matter of testing a single instance, but multiple instances accessing the same entries in the shared state

[pkolaczk]

but we are adding some shared state

No, the added state is still split by the file name; it is not shared. The only thing that's shared is the cache instance, but that one is thread safe, and I guess it's well tested and does the right thing. Should I test library code?

Earlier:
A. 2 writer instances operating on different files concurrently: ok
B. 2 writer instances operating the same file concurrently: incorrect (this is what I meant by saying it's not thread safe)
C. 2 threads calling into 1 writer instance: incorrect

Now:
A. 2 writer instances operating on different files concurrently: ok (because each gets their own FileChecksum by how Cache works)
B. 2 writer instances operating on the same file concurrently: incorrect
C. 2 threads calling into 1 instance: incorrect

What particular scenario do you have in mind?
The concurrent index build test + other NativeDDLIndexTests should already cover scenario A.
Do you want me to test scenarios B and C, so I can show they are not "more incorrect" than before?

[jasonstack]

LGTM, can you create corresponding CNDB pr and make sure the multi-segment index tests in SingleCompactorTest passes?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
81.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[pkolaczk]

I added warnings and I created a PR on the cndb side to test SingleCompactorTest: https://github.com/riptano/cndb/pull/11145

@jasonstack is it good now? Can I merge?

[pkolaczk]

Zhao posted his answer on the cndb PR, so crosslinking it here:
https://github.com/riptano/cndb/pull/11145#issuecomment-2393747297

LeaderTableTest and CompactionTasksOrganizerTest are not related. The CI looks good. Let's rebase and megre

Merging then. Thank you @jasonstack !

[eolivelli]

+1