#6853 Adding store tests for data column pruning #7228

SunnysidedJ · 2025-03-31T15:02:30Z

Issue Addressed

#6853 Update store tests to cover data column pruning

Proposed Changes

Created a helper function check_data_column_existence which is a copy of check_blob_existence but checking data columns instead.
The helper function is then used to check whether data columns are also pruned when blobs are pruned if PeerDAS is enabled.

Additional Info

.

jimmygchen · 2025-04-01T06:35:58Z

beacon_node/beacon_chain/tests/store_tests.rs

@@ -3205,6 +3210,9 @@ async fn deneb_prune_blobs_no_finalization() {
        Some(deneb_fork_slot)
    );
    check_blob_existence(&harness, Slot::new(0), harness.head_slot(), true);


You might have already noticed, this test runs multiple times on CI for all forks.

The code that reads FORK_NAME from env var is here:

lighthouse/beacon_node/beacon_chain/src/test_utils.rs

Lines 193 to 206 in 6f31d44

/// Return a `ChainSpec` suitable for test usage.

///

/// If the `fork_from_env` feature is enabled, read the fork to use from the FORK_NAME environment

/// variable. Otherwise use the default spec.

pub fn test_spec<E: EthSpec>() -> ChainSpec {

let mut spec = if cfg!(feature = "fork_from_env") {

let fork_name = std::env::var(FORK_NAME_ENV_VAR).unwrap_or_else(|e| {

panic!(

"{} env var must be defined when using fork_from_env: {:?}",

FORK_NAME_ENV_VAR, e

)

});

let fork = ForkName::from_str(fork_name.as_str()).unwrap();

fork.make_genesis_spec(E::default_spec())

So this assertion would fail when FORK_NAME=fulu, because we don't store any blobs from since the Fulu fork. I think it might be cleaner to start a separate test that verifies data column pruning instead of having to add conditions on every assertion.

Sorry, I thought we still keep the blobs in Fulu. Changes have been made respectively. Is there a reason that we no longer keep blobs? Is it to save the storage?

Yes, that's one of the reasons - but also it's no longer useful to the CL from Fulu.

The network form of blobs on the beacon chain is DataColumnSidecar, we'll stop using BlobSidecar on p2p from Fulu slots.

Another reason is fullnodes will no longer have blobs, but just samples (DataColumns)

jimmygchen · 2025-04-01T06:45:37Z

beacon_node/beacon_chain/tests/store_tests.rs

@@ -3337,6 +3348,9 @@ async fn deneb_prune_blobs_margin_test(margin: u64) {
        Some(deneb_fork_slot)
    );
    check_blob_existence(&harness, Slot::new(1), harness.head_slot(), true);
+    if check_data_columns {
+        check_data_column_existence(&harness, Slot::new(1), harness.head_slot(), true);
+    }


Same as above, when FORK_NAME=fulu, check_blob_existence would fail already before hitting this line.

jimmygchen · 2025-04-01T06:52:04Z

Thanks for the PR @SunnysidedJ !

Please see my comments above and let me know if it's unclear or if you have any questions!

SunnysidedJ · 2025-04-02T19:42:38Z

Major changes:

Separated tests for data column pruning.
Blob pruning tests only run on Deneb while column ones only run on Fulu with PeerDAS enabled.
Found that fork_boundary test is not working (see inline comments).

SunnysidedJ · 2025-04-02T19:44:32Z

beacon_node/beacon_chain/tests/store_tests.rs


-    // Finalize to epoch 5.
+    // Finalize to epoch 5 (Deneb).
    harness


Returns a stack overflow here. Seems like using the spec with customization is not working properly. I think we need a separate issue for this. Any thoughts?

(The test fails at the unstable branch as well)

Yeah this has been a known issue for a while, so these tests are always run with --release. This is because the c-kzg library forces us to initialise a blob-sized array. I think either moving to rust-eth-kzg or perhaps updating to the latest c-kzg library would resolve this (I haven't checked)

stack overflow when running overflow_cache_test_state_cache in debug #6120

Non-release tests using KZG verification stack overflow on macOS M1 #4987

I see. I will try running them either on Windows or Linux machine for the completion of this task for now

This should work on macOS if you run with --release or you can run the debug test with this env var: RUST_MIN_STACK=4194304 (setting a higher minimum stack size)

SunnysidedJ · 2025-04-02T19:45:37Z

beacon_node/beacon_chain/tests/store_tests.rs

@@ -3282,6 +3290,56 @@ async fn deneb_prune_blobs_fork_boundary() {
    assert_eq!(store.get_blob_info().oldest_blob_slot, Some(pruned_slot));
    check_blob_existence(&harness, Slot::new(0), pruned_slot - 1, false);
    check_blob_existence(&harness, pruned_slot, harness.head_slot(), true);
+
+    // Finalize to epoch 11 (Fulu)


Added a logic to test that blobs get removed once we reach Fulu fork, but couldn't test it due to the stack overflow issue above

ah yeah, could you run it with --release, unfortunately we haven't been able to tackle that issue yet.

SunnysidedJ · 2025-04-02T19:46:08Z

beacon_node/beacon_chain/tests/store_tests.rs

+    let num_blocks = E::slots_per_epoch() * 7;
+
+    // Finalize to epoch 5.
+    harness


The same issue as above

SunnysidedJ · 2025-04-14T09:08:30Z

When I enable Fulu and try blob pruning, the blobs aren't pruned correctly. It'd be appreciated if you could have a look on the commit that reproduces the problem: SunnysidedJ@6217cf9

mergify · 2025-05-19T05:08:19Z

Some required checks have failed. Could you please take a look @SunnysidedJ? 🙏

mergify · 2025-05-19T05:08:20Z

Hi @SunnysidedJ, this pull request has been closed automatically due to 30 days of inactivity. If you’d like to continue working on it, feel free to reopen at any time.

…l-prune-store-tests

eserilev · 2025-05-22T20:04:15Z

I'm going to take a look at the fulu blob pruning test issues

eserilev · 2025-05-22T21:49:40Z

The test is now passing. The reason it was failing was two fold:

we werent advancing the slot clock before extending the chain
we tried to fork from deneb directly to fulu

I've added an intermediary step to fork to electra and run some checks there. Not sure if theres any additional checks we want to make in this test.

mergify · 2025-05-22T22:23:51Z

Some required checks have failed. Could you please take a look @SunnysidedJ? 🙏

eserilev · 2025-05-22T22:46:50Z

ah still need to fix one other test, working on that now

eserilev · 2025-05-23T16:09:14Z

a634758 we werent updating data column info after pruning.

mergify · 2025-05-23T17:23:21Z

All required checks have passed and there are no merge conflicts. This pull request may now be ready for another review.

SunnysidedJ

Sorry, I have been busy with other things and couldn't take much attention to this PR. Thank you for taking a deeper look and unblocking some of the issues I faced with the test cases. Please let me know if you have worked further since last commit and if this needs to be done with priority so that I can clearly understand the status. I will continue working on this whenever I'm available!

SunnysidedJ · 2025-05-26T05:43:28Z

beacon_node/beacon_chain/tests/store_tests.rs

    harness
        .extend_chain(
-            blocks_to_fulu as usize,
+            blocks_to_electra as usize,


Shouldn't this be 13 (7 + 6 epochs in total)?

yes the chain was extended to epoch 13, but the finalized epoch is at 11, I've added some comments here:
06431e4 to make things a bit clearer

SunnysidedJ · 2025-05-26T05:48:35Z

beacon_node/beacon_chain/tests/store_tests.rs

+            // Fulu fork epoch
+            let oldest_slot = fulu_fork_epoch.start_slot(E::slots_per_epoch());
+            // Blobs should not exist
+            check_blob_existence(&harness, oldest_slot, harness.head_slot(), false);


Need to also check that blobs don't exist from Slot 0

SunnysidedJ · 2025-05-26T05:49:07Z

beacon_node/beacon_chain/tests/store_tests.rs

+    check_blob_existence(&harness, pruned_slot, electra_first_slot - 1, true);
+    // Check that blobs exist from electra to the current head
+    check_blob_existence(&harness, electra_first_slot, harness.head_slot(), true);
+    // Finalize to epoch 15 (Fulu)


weve extended the chain to 17 but finalized epoch 15 (added some comments for clarity)

SunnysidedJ · 2025-05-26T05:49:58Z

beacon_node/beacon_chain/tests/store_tests.rs

+            // Blobs should not exist
+            check_blob_existence(&harness, oldest_slot, harness.head_slot(), false);
+            // Data columns should exist
+            check_data_column_existence(&harness, oldest_slot, harness.head_slot(), true);


Need to also check that columns don't exist from Slot 0

SunnysidedJ · 2025-05-26T06:20:00Z

beacon_node/beacon_chain/tests/store_tests.rs

+            let oldest_slot = data_availability_boundary.start_slot(E::slots_per_epoch());
+            assert_eq!(store.get_blob_info().oldest_blob_slot, Some(oldest_slot));
+            // Blobs should exist
+            check_blob_existence(&harness, oldest_slot, harness.head_slot(), true);


Need to also check that blobs don't exist from Slot 0

Pulled the changes and I'm looking into why Slot 40 exists when I call check_blob_existence(&harness, Slot::new(0), oldest_slot - 1, false)

mergify · 2025-05-27T20:52:34Z

All required checks have passed and there are no merge conflicts. This pull request may now be ready for another review.

sigp#6853 Adding store tests for data column pruning

9c84cb1

jimmygchen requested changes Apr 1, 2025

View reviewed changes

jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. das Data Availability Sampling labels Apr 1, 2025

make specific tests for col pruning

24548c3

SunnysidedJ commented Apr 2, 2025

View reviewed changes

SunnysidedJ requested a review from jimmygchen April 2, 2025 19:47

SunnysidedJ mentioned this pull request Apr 17, 2025

Fix for #7212 Call engine_getBlobs as soon as receiving a valid data_column_sidecar from gossip #7329

Open

mergify bot closed this May 19, 2025

mergify bot added the stale Stale PRs that have been inactive and is now outdated label May 19, 2025

Merge branch 'unstable' of https://github.com/sigp/lighthouse into co…

af42806

…l-prune-store-tests

eserilev reopened this May 22, 2025

fix test

d8f1661

eserilev added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. stale Stale PRs that have been inactive and is now outdated labels May 22, 2025

mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels May 22, 2025

Make sure to update data column info

a634758

Merge branch 'unstable' into col-prune-store-tests

8d8f037

mergify bot added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels May 23, 2025

SunnysidedJ commented May 26, 2025

View reviewed changes

eserilev added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels May 27, 2025

mergify bot added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels May 27, 2025

additional comments

06431e4

	/// Return a `ChainSpec` suitable for test usage.
	///
	/// If the `fork_from_env` feature is enabled, read the fork to use from the FORK_NAME environment
	/// variable. Otherwise use the default spec.
	pub fn test_spec<E: EthSpec>() -> ChainSpec {
	let mut spec = if cfg!(feature = "fork_from_env") {
	let fork_name = std::env::var(FORK_NAME_ENV_VAR).unwrap_or_else(\|e\| {
	panic!(
	"{} env var must be defined when using fork_from_env: {:?}",
	FORK_NAME_ENV_VAR, e
	)
	});
	let fork = ForkName::from_str(fork_name.as_str()).unwrap();
	fork.make_genesis_spec(E::default_spec())

#6853 Adding store tests for data column pruning #7228

Are you sure you want to change the base?

#6853 Adding store tests for data column pruning #7228

Uh oh!

Conversation

SunnysidedJ commented Mar 31, 2025

Issue Addressed

Proposed Changes

Additional Info

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmygchen commented Apr 1, 2025

Uh oh!

SunnysidedJ commented Apr 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SunnysidedJ Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SunnysidedJ commented Apr 14, 2025

Uh oh!

mergify bot commented May 19, 2025

Uh oh!

mergify bot commented May 19, 2025

Uh oh!

eserilev commented May 22, 2025

Uh oh!

eserilev commented May 22, 2025

Uh oh!

mergify bot commented May 22, 2025

Uh oh!

eserilev commented May 22, 2025

Uh oh!

eserilev commented May 23, 2025

Uh oh!

mergify bot commented May 23, 2025

Uh oh!

SunnysidedJ left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify bot commented May 27, 2025

Uh oh!

Uh oh!

SunnysidedJ Apr 4, 2025 •

edited

Loading