tests: improve unit test coverage and set threshold#296
Merged
Conversation
pdp2121
changed the title
Collaborator
Author
|
/ai-review |
![xrplf-ai-reviewer[bot]](https://avatars.githubusercontent.com/u/67929741?s=60&v=4){kind=small}
kuan121
previously approved these changes
May 6, 2026
ckeshava
reviewed
May 7, 2026
kuan121
previously approved these changes
May 12, 2026
kuan121
reviewed
May 12, 2026
Comment on lines
+60
to
+62
| --fail-under-lines 85 \ | ||
| --fail-under-regions 85 \ | ||
| --fail-under-functions 75 |
Collaborator
There was a problem hiding this comment.
Is there a way to fail the check when coverage for new or modified code introduced by a PR falls below a certain percentage?
I’m less concerned about the current coverage of existing code. If we want to maintain a high bar for test coverage, we should apply that standard to new code. Over time, that will naturally improve the overall coverage of the repo.
Collaborator
Author
There was a problem hiding this comment.
Yes, patch coverage is possible to do, but would require Codecov (similar to the requirements for removing integration test regex) since cargo-llvm-cov alone would not suffice. Probably worth a follow-up PR since we continue to find more needs to integrate
Collaborator
There was a problem hiding this comment.
Can you create a follow-up story to track this work so it doesn’t get lost?
I think enforcing patch coverage with a high threshold could be a better approach than setting a project-wide threshold, since it focuses on improving new and modified code without relying on broad exclusions. Over time, this should help gradually improve coverage in a repo with low overall coverage.
BTW, xrpl4j uses Codecov (see code). Example PR - XRPLF/xrpl4j#786
Collaborator
Author
There was a problem hiding this comment.
Created the issue: #307
Nice! that means the setup for Codecov in XRPLF has already been done, making our task easier without the need for admin privilege
Collaborator
Author
There was a problem hiding this comment.
Actually let me just add it directly into this PR since Codecov is enabled already in XRPLF so there's no actual blocker
4 tasks
kuan121
reviewed
May 14, 2026
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
Collaborator
Author
ckeshava
previously approved these changes
May 18, 2026
kuan121
reviewed
May 20, 2026
ckeshava
reviewed
May 20, 2026
Comment on lines
+66
to
+68
| --fail-under-lines 83 \ | ||
| --fail-under-regions 85 \ | ||
| --fail-under-functions 73 |
Collaborator
There was a problem hiding this comment.
nit: the fail threshold parameters are redundant with the Codecov checks, because the latter also performs coverage checks.
Collaborator
Author
There was a problem hiding this comment.
Only lines would be covered by codecov, not regions and functions. I would say let's just keep them for extra rigor
Collaborator
Collaborator
Author
Collaborator
|
@pdp2121 there seems to be a flaky test: https://github.com/XRPLF/xrpl-rust/actions/runs/26236700524/job/77211757936?pr=296#step:6:446. I suspect the error is due to the discrepancy in the LedgerEntry Response model. |
Collaborator
Author
Yea I noticed that. We need to either relax the current model (using |
kuan121
previously approved these changes
May 21, 2026
Collaborator
Author
|
@ckeshava I implemented a patch fix (using |
Collaborator
Author
kuan121
approved these changes
May 21, 2026
ckeshava
approved these changes
May 21, 2026
ckeshava
added a commit
that referenced
this pull request
Jun 1, 2026
## High Level Overview of Change Closes #297. The CI integration workflow now runs all five integration test binaries (previously only `integration_test`) under `cargo-llvm-cov` and enforces a 65% line-coverage gate scoped to integration-territory files. Six commits add tests that lift integration coverage from 44.52% → 70.30% to satisfy the gate. Beyond the integration gate, this PR also adds `is_success()` unit tests (`src/models/results/mod.rs`), splits the codecov patch gate into per-flag sections (unit/integration), fixes `get_latest_open_ledger_sequence` (it requested `ledger { ledger_index: "open" }`, which rippled rejects), and documents both coverage flows in `CONTRIBUTING.md`. (The unit thresholds — 83% lines / 85% regions / 73% functions — were set earlier in #296 and are unchanged here.) ### Context of Change The integration suite was running only `tests/integration_test.rs` (40 transaction tests, all routed through one `test_transaction` helper). Several public APIs sat at 0% integration coverage: the sync wrappers in `src/{account,ledger,transaction}/mod.rs`, `asynch::transaction::submit_and_wait`'s poll loop, the WebSocket request/response path, multisign, the sync faucet generator, and the `clients::json_rpc::JsonRpcClient` sync facade. xrpl-py gates its integration suite at 70%; this PR starts xrpl-rust at 65%, with room to tighten as coverage grows. ### Type of Change - [x] New feature (CI coverage gate) - [x] Tests - [x] Bug fix (`get_latest_open_ledger_sequence`) - [x] Documentation (CONTRIBUTING coverage guide) ## Before / After **Workflow.** - **Before:** `cargo test --test integration_test …` — ran a single test binary, no coverage measured. - **After:** coverage runs in three steps: 1. `cargo llvm-cov --no-report … --test integration_test --test cli_integration --test funding --test utils --test test_utils` — collects coverage across all five integration binaries. 2. `cargo llvm-cov report --lcov --ignore-filename-regex "$COVERAGE_IGNORE_REGEX"` — writes an lcov scoped to integration-territory files. 3. That lcov is uploaded to codecov, which enforces the 65% gate (`codecov.yml → project.integration`), and is also saved as a CI artifact. `COVERAGE_IGNORE_REGEX` is defined once (workflow `env`), so the scoped report and the gate always use the same file list. **Scoped coverage:** 44.52% lines → **70.30% lines**. Notable per-file moves: | File | Before | After | |---|---|---| | `account/mod.rs` (sync) | 0% | 100% | | `ledger/mod.rs` (sync) | 47.37% | 100% | | `transaction/mod.rs` (sync) | 0% | 100% | | `wallet/faucet_generation.rs` | 0% | 100% | | `asynch/ledger/mod.rs` | 62.50% | 94.64% | | `asynch/account/mod.rs` | 55.17% | 95.40% | | `asynch/clients/client.rs` | 0% | 91.67% | | `asynch/clients/websocket/websocket_base.rs` | 12.73% | 78.18% | | `asynch/transaction/submit_and_wait.rs` | 0% | 72.87% | ## Test Plan All tests run against `rippleci/xrpld:develop` (CI's standalone rippled image). New test files / functions: - `tests/transactions/submit_and_wait.rs::test_submit_and_wait_payment` — autofill, sign, submit, poll loop, with a background `ledger_accept` driver. - `tests/utils.rs` — three new WS tests: `server_info` round-trip + close, `account_info` against genesis, three sequential requests on one connection (exercises id-based message routing). - `tests/transactions/sync_wrappers.rs` — 10 tests covering every sync wrapper in `src/{account,ledger,transaction}/mod.rs`, plus `clients::json_rpc::JsonRpcClient` and `wallet::faucet_generation`. Each owns a `tokio::runtime::Runtime` + `EnterGuard` so `reqwest` can find the reactor while `embassy_futures::block_on` parks the thread. - `tests/transactions/multisign_payment.rs::test_multisign_payment` — register a 2-of-2 signer list, autofill with `signers_count=2`, each signer signs a copy, `multisign()` merges, `submit()` confirms `tesSUCCESS`. Verified locally by running the same `cargo llvm-cov` invocation CI uses, with the same exclusion regex; the scoped report clears the 65% gate. --------- Co-authored-by: Phu Pham <ppham@ripple.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
realonbebeto
pushed a commit
to realonbebeto/xrpl-rust
that referenced
this pull request
Jun 5, 2026
## High Level Overview of Change <!-- Please include a summary/list of the changes. If too broad, please consider splitting into multiple PRs. If a relevant Asana task, please link it here. --> Brings xrpl-rust unit-test coverage in line with 85% line gate by adding ~120 unit tests to previously-untested or low-coverage files, and reframing what "unit-test coverage" measures. **Tests added** - 29 request models — were all at 0%, now 84–100% - 3 pseudo-transactions (`enable_amendment`, `set_fee`, `unl_modify`) — 0% → 91–94% - `xchain_add_claim_attestation` (0% → 92%), `xchain_commit` (45% → 95%) - 3 ledger objects (`bridge`, `xchain_owned_claim_id`, `objects/mod.rs`) — 0% → 87–100% - `utils/str_conversion` and `utils/get_xchain_claim_id` — 0% → 98–100% - `models/results/mod.rs` dispatcher (17% → 58%), and low-coverage result types `account_tx/ledger_entry/tx/nftoken` — 0–34% → 98–100% - `txn_parser` internals (`nodes`, `parser`, `utils/mod`) — 51–62% → 97–100% - 6 lower-coverage transaction models (`check_cancel`, `escrow_cancel`, `set_regular_key`, `ticket_create`, `payment_channel_fund`, `check_create`) — 68–80% → 97% - `transaction/mod.rs` (`calculate_fee_per_transaction_type`) — 0% → 44% **CI changes** - Thresholds raised: lines 73 → 85, regions 75 → 85, functions 67 → 75 - Excluded integration-territory files from the unit-test coverage report — CLI, async network clients (`asynch/clients/**`), sync wrappers around network calls (`account/`, `ledger/`, `transaction/mod.rs`), and the faucet client. These are exercised by the existing tests/integration/ suite (behind the integration feature flag) and would more naturally fit a separate integration-coverage gate. ### Context of Change <!-- Please include the context of a change. If a bug fix, when was the bug introduced? What was the behavior? If a new feature, why was this architecture chosen? What were the alternatives? If a refactor, how is this better than the previous implementation? If there is a design document for this feature, please link it here. --> [xrpl-py](https://github.com/XRPLF/xrpl-py) is currently at 85% unit-test and 70% integration test coverage. ### Type of Change <!-- Please check relevant options, delete irrelevant ones. --> - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Refactor (non-breaking change that only restructures code) - [x] Tests (You added tests for code that already exists, or your new feature included in this PR) - [ ] Documentation Updates - [ ] Release ## Test Plan <!-- Please describe the tests that you ran to verify your changes and provide instructions so that others can reproduce. --> All CI tests pass. <!-- ## Future Tasks For future tasks related to PR. --> Add integration test coverage threshold
realonbebeto
pushed a commit
to realonbebeto/xrpl-rust
that referenced
this pull request
Jun 5, 2026
## High Level Overview of Change Closes XRPLF#297. The CI integration workflow now runs all five integration test binaries (previously only `integration_test`) under `cargo-llvm-cov` and enforces a 65% line-coverage gate scoped to integration-territory files. Six commits add tests that lift integration coverage from 44.52% → 70.30% to satisfy the gate. Beyond the integration gate, this PR also adds `is_success()` unit tests (`src/models/results/mod.rs`), splits the codecov patch gate into per-flag sections (unit/integration), fixes `get_latest_open_ledger_sequence` (it requested `ledger { ledger_index: "open" }`, which rippled rejects), and documents both coverage flows in `CONTRIBUTING.md`. (The unit thresholds — 83% lines / 85% regions / 73% functions — were set earlier in XRPLF#296 and are unchanged here.) ### Context of Change The integration suite was running only `tests/integration_test.rs` (40 transaction tests, all routed through one `test_transaction` helper). Several public APIs sat at 0% integration coverage: the sync wrappers in `src/{account,ledger,transaction}/mod.rs`, `asynch::transaction::submit_and_wait`'s poll loop, the WebSocket request/response path, multisign, the sync faucet generator, and the `clients::json_rpc::JsonRpcClient` sync facade. xrpl-py gates its integration suite at 70%; this PR starts xrpl-rust at 65%, with room to tighten as coverage grows. ### Type of Change - [x] New feature (CI coverage gate) - [x] Tests - [x] Bug fix (`get_latest_open_ledger_sequence`) - [x] Documentation (CONTRIBUTING coverage guide) ## Before / After **Workflow.** - **Before:** `cargo test --test integration_test …` — ran a single test binary, no coverage measured. - **After:** coverage runs in three steps: 1. `cargo llvm-cov --no-report … --test integration_test --test cli_integration --test funding --test utils --test test_utils` — collects coverage across all five integration binaries. 2. `cargo llvm-cov report --lcov --ignore-filename-regex "$COVERAGE_IGNORE_REGEX"` — writes an lcov scoped to integration-territory files. 3. That lcov is uploaded to codecov, which enforces the 65% gate (`codecov.yml → project.integration`), and is also saved as a CI artifact. `COVERAGE_IGNORE_REGEX` is defined once (workflow `env`), so the scoped report and the gate always use the same file list. **Scoped coverage:** 44.52% lines → **70.30% lines**. Notable per-file moves: | File | Before | After | |---|---|---| | `account/mod.rs` (sync) | 0% | 100% | | `ledger/mod.rs` (sync) | 47.37% | 100% | | `transaction/mod.rs` (sync) | 0% | 100% | | `wallet/faucet_generation.rs` | 0% | 100% | | `asynch/ledger/mod.rs` | 62.50% | 94.64% | | `asynch/account/mod.rs` | 55.17% | 95.40% | | `asynch/clients/client.rs` | 0% | 91.67% | | `asynch/clients/websocket/websocket_base.rs` | 12.73% | 78.18% | | `asynch/transaction/submit_and_wait.rs` | 0% | 72.87% | ## Test Plan All tests run against `rippleci/xrpld:develop` (CI's standalone rippled image). New test files / functions: - `tests/transactions/submit_and_wait.rs::test_submit_and_wait_payment` — autofill, sign, submit, poll loop, with a background `ledger_accept` driver. - `tests/utils.rs` — three new WS tests: `server_info` round-trip + close, `account_info` against genesis, three sequential requests on one connection (exercises id-based message routing). - `tests/transactions/sync_wrappers.rs` — 10 tests covering every sync wrapper in `src/{account,ledger,transaction}/mod.rs`, plus `clients::json_rpc::JsonRpcClient` and `wallet::faucet_generation`. Each owns a `tokio::runtime::Runtime` + `EnterGuard` so `reqwest` can find the reactor while `embassy_futures::block_on` parks the thread. - `tests/transactions/multisign_payment.rs::test_multisign_payment` — register a 2-of-2 signer list, autofill with `signers_count=2`, each signer signs a copy, `multisign()` merges, `submit()` confirms `tesSUCCESS`. Verified locally by running the same `cargo llvm-cov` invocation CI uses, with the same exclusion regex; the scoped report clears the 65% gate. --------- Co-authored-by: Phu Pham <ppham@ripple.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
High Level Overview of Change
Brings xrpl-rust unit-test coverage in line with 85% line gate by adding ~120 unit tests to previously-untested or low-coverage files, and reframing what "unit-test coverage" measures.
Tests added
enable_amendment,set_fee,unl_modify) — 0% → 91–94%xchain_add_claim_attestation(0% → 92%),xchain_commit(45% → 95%)bridge,xchain_owned_claim_id,objects/mod.rs) — 0% → 87–100%utils/str_conversionandutils/get_xchain_claim_id— 0% → 98–100%models/results/mod.rsdispatcher (17% → 58%), and low-coverage result typesaccount_tx/ledger_entry/tx/nftoken— 0–34% → 98–100%txn_parserinternals (nodes,parser,utils/mod) — 51–62% → 97–100%check_cancel,escrow_cancel,set_regular_key,ticket_create,payment_channel_fund,check_create) — 68–80% → 97%transaction/mod.rs(calculate_fee_per_transaction_type) — 0% → 44%CI changes
asynch/clients/**), sync wrappers around network calls (account/,ledger/,transaction/mod.rs), and the faucet client. These are exercised by the existing tests/integration/ suite (behind the integration feature flag) and would more naturally fit a separate integration-coverage gate.Context of Change
xrpl-py is currently at 85% unit-test and 70% integration test coverage.
Type of Change
Test Plan
All CI tests pass.
Add integration test coverage threshold