[CHIA-3854] WIP: test INTERNED_GENERATOR flag active (chia_rs #1377)#20800
Draft
richardkiss wants to merge 35 commits intoChia-Network:mainfrom
Draft
[CHIA-3854] WIP: test INTERNED_GENERATOR flag active (chia_rs #1377)#20800richardkiss wants to merge 35 commits intoChia-Network:mainfrom
richardkiss wants to merge 35 commits intoChia-Network:mainfrom
Conversation
danieljperry
changed the title
Test what breaks when INTERNED_GENERATOR is active at hard_fork2_height. Uses commit f2dbd518 from pure-storage-model-v2 branch. Adds Rust toolchain to CI for building from git source. Made-with: Cursor
compute_block_cost() was using len(bytes) * COST_PER_BYTE for serialization cost even when INTERNED_GENERATOR is active. This made the declared cost in block headers too low, causing BLOCK_COST_EXCEEDS_MAX during validation. Fix: when height >= HARD_FORK2_HEIGHT, use run_block_generator2 (with signature validation disabled) to compute the correct tree-based cost. Also update test_basics in test_cost_calculation.py to verify the cost breakdown for the new model instead of asserting byte-length-based cost. Made-with: Cursor
Two fixes for tests failing with HARD_FORK_3_0 consensus modes: 1. compute_block_cost: When run_block_generator2 fails on invalid blocks (e.g. wrong announcements), return MAX_BLOCK_COST_CLVM instead of falling through to the old byte-based cost. This ensures the block validator has enough headroom to find the real condition error instead of bailing with BLOCK_COST_EXCEEDS_MAX. 2. test_double_spend_with_higher_fee: Under INTERNED_GENERATOR, 4 spends sharing the same puzzle are cheaper as one bundle than as separate bundles (interning deduplicates the puzzle tree). This means sb1234_1 genuinely has higher fee-per-cost than the conflicting sb12+sb3, so the replacement correctly succeeds. Made-with: Cursor
Point chia_rs at PR Chia-Network#1377 (pure-storage-model-v2, commit 9774629f) which has INTERNED_GENERATOR defined but not wired in get_flags_for_height_and_constants. Add chia/consensus/flags.py with a wrapper that ORs in INTERNED_GENERATOR (0x0800_0000) when prev_tx_height >= HF2. All 16 call sites now import this wrapper aliased as get_flags_for_height_and_constants, so existing code is unchanged. This branch exists to identify what breaks when the flag is activated, without modifying chia_rs itself. Made-with: Cursor
Extract the height-dependent hash selection into a small pure function generator_root(program, height, constants) and use it in both block_creation.py and block_body_validation.py instead of duplicating the if/else. Tests now call the helper directly — no scaffolding needed. Drops 100 lines of dummy ProofOfSpace/RewardChainBlockUnfinished construction. The "differs_between_forks" test is removed: the two pre/post tests already cover everything and validating that std_hash != tree_hash is not this code's responsibility. Made-with: Cursor
…ck_tools; add HF2 tests blockchain.py used std_hash unconditionally in validate_unfinished_block_header, causing all valid post-HF2 unfinished blocks (which have tree_hash generator_root) to be rejected. wallet_block_tools.py had the same bug, making wallet tests construct blocks with wrong generator_root post-HF2. Add three new tests to TestBodyValidation asserting the correct hash format is used pre- and post-HF2, and that the old format is rejected after the fork. Made-with: Cursor
…lidation Test that validate_unfinished_block accepts a transaction block with tree_hash generator_root post-HF2, and rejects one with std_hash (old format). These go through validate_unfinished_block_header, the code path exercised when a farmer submits a new block to a full node. Made-with: Cursor
Made-with: Cursor
…ot() for hash commitment=1 previously hardcoded std_hash for the generator_root in the tampered TransactionsInfo, which produces the right error code (INVALID_TRANSACTIONS_INFO_HASH) pre-HF2 but for the wrong reason post-HF2. Use generator_root() so the test correctly represents a generator whose hash matches the replaced program under the current fork rules. Made-with: Cursor
Made-with: Cursor
richardkiss
force-pushed
the
chia-rs-1377-wired
branch
from
73c29fa to
96f4a7b
Compare
This was referenced Apr 22, 2026
block_creation.py used tree_hash() without importing it from chia_rs, causing NameError at runtime and mypy failures. Also fix test_blockchain to pass npc_result.conds (SpendBundleConditions) instead of the full NPCResult to validate_unfinished_block(). Made-with: Cursor
compute_block_cost was importing get_flags_for_height_and_constants directly from chia_rs (without INTERNED_GENERATOR), while validation uses the wrapper from flags.py that includes it. This mismatch caused test blocks to be generated with pre-interned cost but validated with interned cost, triggering BLOCK_COST_EXCEEDS_MAX in HF3 tests. Made-with: Cursor
Renamed to .disabled: build installers (deb, rpm, macOS, Windows), require-labels, test-install-scripts, benchmarks, docker triggers. These all fail due to git dep (no PyPI wheel) and are not useful signal for the HF2 maze integration work. Made-with: Cursor
The tests expected blocks[1].transactions_generator to be non-None, but guarantee_transaction_block only ensures a transaction block is created, not that it has a generator (transactions). Use include_transactions=True so blocks actually contain spend bundles and thus have generators. Made-with: Cursor
In HF3 consensus modes where HARD_FORK2_HEIGHT=0, INTERNED_GENERATOR uses tree-based serialization cost instead of flat byte cost. Update block_base_cost values in test_unknown_conditions_with_cost and test_softfork_condition to match the new cost model. Made-with: Cursor
…D_GENERATOR - test_overlong_generator_encoding: use generator_root() which respects HF2 tree_hash vs std_hash, instead of hardcoded std_hash - test_new_transaction_and_mempool: INTERNED_GENERATOR cost model lets 22 txns fit in mempool (was 20) in HF3 modes Made-with: Cursor
Cached test blocks have std_hash generator_root but validation now expects tree_hash post-HF2. These tests need the test block cache regenerated via: pytest -m build_test_chains Made-with: Cursor
richardkiss
force-pushed
the
chia-rs-1377-wired
branch
from
6e288e6 to
52bd896
Compare
validate_unfinished_block now asserts conds.validated_signature. Use run_block_generator2 with the aggregated signature instead of get_name_puzzle_conditions which uses DONT_VALIDATE_SIGNATURE. Made-with: Cursor
After HARD_FORK2_HEIGHT, blocks use serde_2026 interned format instead of backrefs for generator serialization. Wired at: - mempool.py: switches based on INTERNED_GENERATOR flag - block_tools.py: both farming and test paths check height - bundle_tools.py: adds simple_solution_generator_2026 helper Made-with: Cursor
Add create_block_generator_2026() to Mempool and MempoolManager. Snapshots mempool candidates into Block2026Builder (with dedup/FF), lets it optimize in a background thread for the timeout duration, then retrieves the best block instantly via best(). Gated on both block_creation=2 config AND height >= HARD_FORK2_HEIGHT. Falls back to version 1 pre-HF2. Made-with: Cursor
…uilder The pyproject.toml was pinned to Chia-Network/chia_rs@e9677319 which predates the Block2026Builder commit. This caused ImportError in CI since Block2026Builder only exists in the richardkiss fork at 60c1915f. Made-with: Cursor
- Format long import line in mempool.py (ruff format) - Sort imports alphabetically in test_blockchain.py (ruff isort) - Add type: ignore for Block2026Builder (stubs not yet regenerated) Made-with: Cursor
solution_generator_2026 and Block2026Builder produce serde_2026 format which is not standard CLVM serialization. Program.from_bytes() validates as CLVM and rejects it with "unexpected end of buffer". Use the raw bytes constructor Program(bytes) to bypass CLVM validation. Made-with: Cursor
Program() (raw constructor) raises TypeError. Program.from_bytes() raises ValueError for serde_2026 format. Both paths fail — this requires a chia_rs-side fix (e.g. from_bytes_auto) to support serde_2026 format in the Program type. Reverting to from_bytes() to keep the error consistent with the pre-existing known issue. Made-with: Cursor
Now that chia_rs exposes from_program_bytes() (raw bytes without CLVM validation), use it for serde_2026 code paths where solution_generator_2026 and Block2026Builder produce non-standard-CLVM serialized generators. Also bumps chia_rs to 4c91d48e which includes the new method and Block2026Builder type stubs. Made-with: Cursor
generator_root() calls tree_hash on the block generator bytes. For serde_2026 blocks, tree_hash (which only handles standard CLVM and backrefs) fails with "bad encoding". Switch to tree_hash_auto which auto-detects format using node_from_bytes_auto. Bumps chia_rs to bf0ff53a which adds tree_hash_auto. Made-with: Cursor
Post-HARD_FORK2 blocks use serde_2026 serialization which is not standard CLVM, so is_canonical_serialization() always rejects them. The canonical encoding check only applies to standard CLVM generators (between SOFT_FORK9 and HARD_FORK2). Made-with: Cursor
run_block_generator2 checks that generators start with [0xff, 0x01] (CLVM quote prefix) when SIMPLE_GENERATOR flag is set. serde_2026 blocks have a different prefix. Updated chia_rs bypasses this check when INTERNED_GENERATOR is also active. Made-with: Cursor
FullBlock.from_json_dict now accepts blocks with serde_2026 generators by using node_from_bytes_auto as a fallback in Program.from_json_dict. Made-with: Cursor
- tree_hash_auto matches tree_hash for standard CLVM programs - is_canonical_serialization still rejects overlong encoding - serde_2026 generators work with generator_root post-HF2 - serde_2026 is NOT considered canonical (is_canonical_serialization) Bumps chia_rs to bce770ae (includes Rust-side guard tests). Made-with: Cursor
- bump chia_rs to 1b430ce2 which fixes Program::parse() to auto-detect serde_2026 format (uses serialized_length_serde_2026 for delimiting) - fix create_block_generator: use from_program_bytes for serde_2026 generators instead of from_bytes (which validates classic CLVM) - fix canonical encoding check: detect serde_2026 by magic prefix instead of blanket-skipping the check post-HF2 (classic generators should still be validated) Made-with: Cursor
The post-HF2 generator_root tests called tree_hash() on raw bytes that may be in serde_2026 format. tree_hash() only handles classic CLVM serialization and raises 'bad encoding'. Use tree_hash_auto() which detects the format, matching what consensus generator_root() uses internally. Made-with: Cursor
Coverage Report for CI Build 25075071645Warning Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes. Coverage decreased (-0.04%) to 91.218%Details
Uncovered ChangesNo uncovered changes found. Coverage Regressions247 previously-covered lines in 31 files lost coverage.
Coverage Stats
💛 - Coveralls |
Adds direct unit-test coverage for the new 2026 block-builder path: - extends test_create_block_generator_custom_spend (both test_mempool.py and test_mempool_manager.py) to parametrize over old/new/2026 modes - new test_create_block_generator_2026_empty_mempool (covers the early-return when no candidates accumulated) - new test_create_block_generator_2026_zero_timeout (covers the no-sleep branch when timeout has already elapsed) - new test_create_block_generator_2026_skip_dedup (covers the SkipDedup handler by adding two items spending the same dedup-eligible coin with different solutions) - new test_create_block_generator_2026_wrong_header_hash (covers the manager wrapper return None path) - extends test_max_spends_per_block to also exercise the 2026 path, covering the > MAX_SPENDS_PER_BLOCK skip branch Also marks defensive paths # pragma: no cover (matching the equivalent treatment of create_block_generator2): - mempool.py: MAX_COIN_AMOUNT overflow break, generic Exception fallback, empty block_program return None - full_node_api.py: dispatch line for block_creation=2 + post-HF2 height (paths exercised by mempool tests directly) - block_tools.py: compute_block_cost() defensive Exception handler Made-with: Cursor
github-actions
Bot
added
the
merge_conflict
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces #20799 (closed when
bump-clvmrsmerged). Rebased ontomain.f2dbd518from PR Update initial-config.yaml #1377 (pure-storage-model-v2), which wiresINTERNED_GENERATORintoget_flags_for_height_and_constants()athard_fork2_height.block_tools.py:compute_block_cost()callsrun_block_generator2post-HF2 for correct tree-based costtest_cost_calculation.py: Updated assertions for interned cost modeltest_mempool.py: Account for interning changing fee-per-cost ratios (shared puzzles are cheaper as one bundle)Context
Known remaining failures
Made with Cursor