Verify values in secondary database against expected state #13281

archang19 · 2025-01-08T18:44:59Z

Summary

TLDR: This PR enables secondary DB verification inside the "simple" crash tests (NonBatchedOpsStressTest). Essentially, we want to be able to verify that the secondary is a valid "prefix" of the primary. This PR allows us to do this by piggybacking on the existing verification of the primary through Get() requests.

I originally proposed replaying the trace file to recreate the ExpectedState as of a specific sequence number. This could be used to run verifications against the secondary database. I did some experimenting in #13266 and got a "mostly working" implementation of this approach. I could sometimes get through entire key space verifications but eventually one of the keys would fail verification. I have not figured out the root cause yet, but I assume that something caused the sequence number to trace record alignment to break.

The approach in this PR is considerably simpler. We can just check that the secondary database's value is in the correct "range," which we already have functionality for checking that. Compared to the approach in #13266, this approach is much, much simpler since we do not have to go through the whole headache of replaying the trace and creating an entire new ExpectedState. (Look at #13266 to see how much of a mess that creates.) I think this approach is better than my original approach in almost most aspects: it's faster, uses less space, and has less room for implementation errors.

Other nice aspects of this approach:

We don't need to block the primary. (Another approach you could imagine would be to block writes to the primary, have the secondary catch up, do the whole verification, and then re-enable writes to the primary.)
We don't need to block the secondary or do any special coordination (locks, sync points, etc). (If we insist on one "golden" expected value to be read from the secondary, then we need to make sure that another thread does not call TryCatchUpWithPrimary while we are trying to perform a Get())
More "realistic" usage of the secondary. For instance, writes to the primary and secondary would continue on in production while we try to read from the secondary.

The main drawback of course is that we verify against a range of expected values, rather than one particular expected value. However, I think this is acceptable and "good enough" especially with all of other the aforementioned benefits.

Historical context: There is some very old code that attempted to verify secondaries, but is not enabled. This code has not been touched or executed in an extremely long time, and the crash tests started failing when I tried enabling it, most likely because the code is not compatible with certain other crash test options. This code is for the "continuous verification" and involves long iterator scans over the secondary database. Some of the code involved the cross CF consistency test type. I don't think the old checks are what we really want for our purposes of verifying the secondary functionality. Since I don't think we will get much value out of this old "continuous verification" code, I integrated my secondary verification with the "regular" database verification. This also makes the rollout simpler on my end, since I can control whether my secondary verifications are enabled through one test_secondary configuration. To make sure the old code does not execute for our recurring crash test runs, I had to enforce that continuous_verification_interval is 0 whenever test_secondary is set.

Monitoring: I will want to monitor the Sandcastle "simple" runs for failures where test_secondary is set. All of my error messages are prefixed with "Secondary" so it should be easy to tell if this PR causes any crash test issues.

Future work:

Extend this to followers. I think the same verification method should work, so most of the code from this PR should be reusable
Add additional checks to make sure the sequence number of the follower/secondary is actually increasing. For instance, if the primary's sequence number has advanced, and in that period the secondary has not (even after calling TryCatchUpWithPrimary), then we know there is a problem
Potentially checking things other than Get() for the secondary (i.e. iterators). I think the focus here should be testing replication-specific logic, and since we will already have separate unit tests, we do not need to repeat all of tests against both the primary and the secondary.

Test Plan

The primary crash test commands I ran were:

python3 tools/db_crashtest.py --simple blackbox --test_secondary=1
python3 tools/db_crashtest.py --simple whitebox --test_secondary=1

As a sanity check, I added an assert(false) right after my secondary verification code to make sure that my code was actually being run.

facebook-github-bot · 2025-01-08T22:16:40Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-01-08T22:25:01Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-08T22:28:42Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-01-08T23:13:09Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-08T23:14:12Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-01-08T23:57:18Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-08T23:59:01Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-09T00:00:23Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-01-09T17:43:53Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-09T17:44:35Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-01-09T18:03:42Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-09T18:08:06Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

archang19 · 2025-01-09T19:04:06Z

db_stress_tool/db_stress_test_base.h

@@ -412,9 +412,8 @@ class StressTest {
  std::atomic<bool> db_preload_finished_;
  std::shared_ptr<SstQueryFilterConfigsManager::Factory> sqfc_factory_;

-  // Fields used for continuous verification from another thread


I guess the original others intended for cmp_db_ to potentially be used for other purposes, but right now the only usages are for opening secondary databases. So I think we can improve the naming

archang19 · 2025-01-09T19:04:35Z

db_stress_tool/no_batched_ops_stress.cc

+            s = secondary_db_->Get(options, column_families_[cf], key,
+                                   &from_db);
+
+            assert(!pre_read_expected_values.empty() &&


This was to get our internal code linter to stop complaining about the vector index access

archang19 · 2025-01-09T19:06:39Z

db_stress_tool/no_batched_ops_stress.cc

@@ -2810,6 +2863,84 @@ class NonBatchedOpsStressTest : public StressTest {
    return true;
  }

+  // Compared to VerifyOrSyncValue, VerifyValueRange takes in a


I thought about adding this functionality into VerifyOrSyncValue but that would make the method signature and the implementation even more complicated

…FLAGS_continuous_verification_interval

facebook-github-bot · 2025-01-23T00:04:13Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-23T00:04:42Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

anand1976 · 2025-01-23T19:38:07Z

db_stress_tool/no_batched_ops_stress.cc

+    // secondary_db_, the crash test fails during this iterator scan. The stack
+    // trace mentions BlobReader/BlobSource but it may not necessarily be
+    // related to BlobDB
+    if (!secondary_db_ || !FLAGS_continuous_verification_interval) {


Does this mean verify_db_one_in > 0 is ignored unless test_secondary is set?

Okay let me update the comment since now that I read it, it is confusing. By "currently this method gets called," I really should have said "currently (before this PR which makes a change), this is what was happening, and we do not want this to occur.

We do not want verify_db_one_in > 0 to matter if FLAGS_continuous_verification_interval is 0

@anand1976 I updated teh comment. Hopefully it is clearer now. Basically:

ContinuouslyVerifyDb doesn't actually work when there is a secondary. It was written so long ago, and the crash tests will fail if you try to run it.

We want a way to make sure ContinuouslyVerifyDb does not run.

The way I have set this up is to override FLAGS_continuous_verification_interval to 0 whenever I am testing secondaries (inside db_crashtest.py).

This way someone in the future can still experiment with ContinuouslyVerifyDb, but we won't be messing up all our automated crash tests

facebook-github-bot · 2025-01-23T20:05:39Z

@archang19 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2025-01-23T20:07:52Z

@archang19 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

anand1976

LGTM! This is a great start. I think eventually we should have a more rigorous verification that relies on the trace replay to verify point in time consistency. We might need to fix the KV tracing so we can replay upto a specific sequence number.

facebook-github-bot · 2025-01-24T23:09:08Z

@archang19 merged this pull request in a8bd6a3.

Summary: #13281 added support to the crash tests for secondary DB verification. I looked at our recurring crash tests to see what impact #13281 had. The actual secondary verification looks okay to me (no `assert` failures), but I noticed memory leaks were detected. The problematic areas were tracked down to the call to `DB::OpenAsSecondary` from `rocksdb::StressTest::Open`. Pull Request resolved: #13337 Test Plan: Monitor recurring crash tests. It is likely hard to reproduce the ASAN failures locally if they are rare enough. ``` make -j100 db_stress COMPILE_WITH_ASAN=1 python3 tools/db_crashtest.py --simple blackbox --test_secondary=1 ``` Reviewed By: cbi42 Differential Revision: D68721624 Pulled By: archang19 fbshipit-source-id: 9c3044884c505c43c1819a3e98ce99b2d171f3ca

…led or manual_wal_flush is set (#13338) Summary: #13281 added support for verifying secondaries in the crash tests. We are trying to check that the values returned by the secondary in `Get` requests fall within an expected range of values. We do reads from the shared expected state before and after we read from the secondary. There are some rare verification failures where `VerifyValueRange` fails with `Unexpected value found outside of the value base range`. I have some ideas on what the root cause could be. The secondary can read the WAL, MANIFEST, and SST files, but in some scenarios some of these pieces may not be present. I noticed that the failures had `manual_wal_flush_one_in=1000`, which means that `options.manual_wal_flush` is set to `true`. With this setting, RocksDB has its own internal buffers that need to be manually flushed for the WAL to be persisted. Although the test failures I looked at did not disable the WAL, I realized that, when the WAL is disabled, we should flush the primary's memtables, since the secondary needs to be able to find SST files to fully catch up. Injected faults further complicate matters, so I have a check to skip secondary verification whenever the WAL or memtable flushes fail due to fault injection. Pull Request resolved: #13338 Test Plan: Locally: ``` python3 tools/db_crashtest.py --simple blackbox --test_secondary=1 python3 tools/db_crashtest.py --simple whitebox --test_secondary=1 python3 tools/db_crashtest.py --simple blackbox --test_secondary=1 --disable_wal=1 python3 tools/db_crashtest.py --simple blackbox --test_secondary=1 --disable_wal=0 --manual_wal_flush_one_in=1000 ``` I will monitor the recurring crash tests after this gets merged. Reviewed By: anand1976 Differential Revision: D68741287 Pulled By: archang19 fbshipit-source-id: 86f474c41a68b7b06f2ed80a851c6cb52a47ebe7

…13343) Summary: This is a continuation of #13338, which aims to address crash test failures caused by #13281. This PR attempts to address the TSAN failures. I searched for wherever we call `column_families_.clear()` and made sure that we also clear the secondary column families as well. I made a helper method since it is easy to forget to clear both sets of column families. Pull Request resolved: #13343 Test Plan: Monitor recurring crash test results. Reviewed By: cbi42 Differential Revision: D68790580 Pulled By: archang19 fbshipit-source-id: 96ed758a21545dd20181b8db71b81dd660546e18

Summary: #13281 added secondary database verification to the crash tests. I am seeing failures in the crash test that trace back to these two code sections: 1. https://github.com/facebook/rocksdb/blob/main/db_stress_tool/no_batched_ops_stress.cc#L2969-L2975 ```cpp VerificationAbort( shared, msg_prefix + "Non-OK status" + read_u64ts.str() + s.ToString(), cf, key, "", Slice(expected_value_data, expected_value_data_size)); ``` 2. https://github.com/facebook/rocksdb/blob/main/table/block_fetcher.cc#L327-L331 ```cpp io_status_ = IOStatus::Corruption( "truncated block read from " + file_->file_name() + " offset " + std::to_string(handle_.offset()) + ", expected " + std::to_string(block_size_with_trailer_) + " bytes, got " + std::to_string(slice_.size())); ``` The error messages look like ``` Secondary get verificationNon-OK statusCorruption: truncated block read from /dev/shm/rocksdb_test/rocksdb_crashtest_blackbox/011887.sst offset 11780096, expected 16274 bytes, got 0 ``` As you can see, the issue is not that the values of the secondary DB differ from what we expect. Rather, the `get` request itself is returning a non-OK status. I looked at the test configurations for the failed test runs, and I saw that both of them enabled fault injections (e.g. `read_fault_one_in`). Pull Request resolved: #13366 Test Plan: Before merging: `python3 tools/db_crashtest.py --simple blackbox --test_secondary=1` After merging: monitor for crash test failures Reviewed By: jaykorean Differential Revision: D69059138 Pulled By: archang19 fbshipit-source-id: a9c07d80381f52bdff220b0db3302748ebccd96c

facebook-github-bot added the CLA Signed label Jan 8, 2025

archang19 force-pushed the secondary-value-pre-post-range branch from fdffa05 to 8f637f9 Compare January 8, 2025 18:45

archang19 changed the title ~~[in progress] Update secondary db crash test verification~~ Verify values in secondary database against expected state Jan 8, 2025

archang19 force-pushed the secondary-value-pre-post-range branch 4 times, most recently from 7831d9a to 8638ad5 Compare January 8, 2025 22:11

archang19 marked this pull request as ready for review January 8, 2025 22:16

archang19 force-pushed the secondary-value-pre-post-range branch from 8638ad5 to 2a6c0e3 Compare January 8, 2025 22:24

archang19 force-pushed the secondary-value-pre-post-range branch from 2a6c0e3 to 8820235 Compare January 8, 2025 23:13

archang19 force-pushed the secondary-value-pre-post-range branch from ce0cd10 to f5a7877 Compare January 8, 2025 23:58

archang19 force-pushed the secondary-value-pre-post-range branch from f5a7877 to 8f176e9 Compare January 9, 2025 17:43

archang19 force-pushed the secondary-value-pre-post-range branch from 8f176e9 to 843761d Compare January 9, 2025 18:03

archang19 commented Jan 9, 2025

View reviewed changes

archang19 requested a review from anand1976 January 9, 2025 19:07

archang19 mentioned this pull request Jan 9, 2025

Add test_follower option to crash test #13282

Draft

archang19 added 8 commits January 22, 2025 16:03

use all threads to verify different portions

cce848a

Rename cmp_db and cmp_cfhs

14f143a

Update comments and abort error messaging

3cc59d5

update comment and error message

73da410

Add an option for test_secondary to the simple default params

e6076bc

Incorporate secondary testing as part of regular get() tests

eb08ed2

Do not run continuous verification interval unless we explicitly set …

0e9f00d

…FLAGS_continuous_verification_interval

Fix lint warning

dd5be01

archang19 force-pushed the secondary-value-pre-post-range branch from cc8b070 to dd5be01 Compare January 23, 2025 00:04

anand1976 reviewed Jan 23, 2025

View reviewed changes

Update comment

f1aa17a

anand1976 approved these changes Jan 24, 2025

View reviewed changes

facebook-github-bot closed this in a8bd6a3 Jan 24, 2025

facebook-github-bot added the Merged label Jan 24, 2025

This was referenced Jan 27, 2025

Call delete on secondary_db_ wherever it is done on db_ #13337

Closed

Force a primary flush before secondary verification when WAL is disabled or manual_wal_flush is set #13338

Closed

This was referenced Jan 28, 2025

Only run secondary db verification for blackbox crash tests #13341

Closed

Clean up secondary column families alongside primary column families #13343

Closed

This was referenced Jan 30, 2025

Use secondary_cfhs in secondary_db_->Get #13351

Closed

Skip secondary verification on injected read error #13366

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Verify values in secondary database against expected state #13281

Verify values in secondary database against expected state #13281

archang19 commented Jan 8, 2025 •

edited

Loading

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

archang19 Jan 9, 2025

archang19 Jan 9, 2025 •

edited

Loading

archang19 Jan 9, 2025

facebook-github-bot commented Jan 23, 2025

facebook-github-bot commented Jan 23, 2025

anand1976 Jan 23, 2025

archang19 Jan 23, 2025

archang19 Jan 23, 2025

facebook-github-bot commented Jan 23, 2025

facebook-github-bot commented Jan 23, 2025

anand1976 left a comment

facebook-github-bot commented Jan 24, 2025

Verify values in secondary database against expected state #13281

Verify values in secondary database against expected state #13281

Conversation

archang19 commented Jan 8, 2025 • edited Loading

Summary

Test Plan

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 8, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

facebook-github-bot commented Jan 9, 2025

archang19 Jan 9, 2025

Choose a reason for hiding this comment

archang19 Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

archang19 Jan 9, 2025

Choose a reason for hiding this comment

facebook-github-bot commented Jan 23, 2025

facebook-github-bot commented Jan 23, 2025

anand1976 Jan 23, 2025

Choose a reason for hiding this comment

archang19 Jan 23, 2025

Choose a reason for hiding this comment

archang19 Jan 23, 2025

Choose a reason for hiding this comment

facebook-github-bot commented Jan 23, 2025

facebook-github-bot commented Jan 23, 2025

anand1976 left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Jan 24, 2025

archang19 commented Jan 8, 2025 •

edited

Loading

archang19 Jan 9, 2025 •

edited

Loading