Skip to content

feat: set up codecov for the hardhat repository #6502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 21 commits into
base: v-next
Choose a base branch
from
Open

Conversation

galargh
Copy link
Member

@galargh galargh commented Mar 25, 2025

  • Because this PR includes a bug fix, relevant tests have been included.
  • Because this PR includes a new feature, the change was previously discussed on an Issue or with someone from the team.
  • I didn't do anything of this.

Note

This PR is to be reviewed synchronously on a call.

Warning

Instrumenting the code for coverage (lcov) can have a significant effect on CI job runtime. We should take that into account when reviewing the PR together. Below, you'll find a table with job runtimes with both lcov and junit report generation, just junit report generation and no report generation. The packages affected the most are hardhat-keystore and hardhat.

job runtime (lcov + junit) [s] runtime (junit) [s] runtime [s]
[example-project] ci on macos-13 (Node 22) 206 197 217
[example-project] ci on macos-latest (Node 22) 147 103 107
[example-project] ci on ubuntu-latest (Node 22) 85 79 81
[example-project] ci on windows-latest (Node 22) 157 140 134
[hardhat-errors] ci on macos-13 (Node 22) 57 72 75
[hardhat-errors] ci on macos-latest (Node 22) 47 35 40
[hardhat-errors] ci on ubuntu-latest (Node 22) 25 27 17
[hardhat-errors] ci on windows-latest (Node 22) 68 70 53
[hardhat-ethers-chai-matchers] ci on macos-13 (Node 22) 167 93 104
[hardhat-ethers-chai-matchers] ci on macos-latest (Node 22) 98 55 73
[hardhat-ethers-chai-matchers] ci on ubuntu-latest (Node 22) 77 52 48
[hardhat-ethers-chai-matchers] ci on windows-latest (Node 22) 124 99 91
[hardhat-ethers] ci on macos-13 (Node 22) 79 73 66
[hardhat-ethers] ci on macos-latest (Node 22) 64 58 40
[hardhat-ethers] ci on ubuntu-latest (Node 22) 43 47 35
[hardhat-ethers] ci on windows-latest (Node 22) 125 84 82
[hardhat-ignition-core] ci on macos-13 (Node 22) 130 71 109
[hardhat-ignition-core] ci on macos-latest (Node 22) 66 62 35
[hardhat-ignition-core] ci on ubuntu-latest (Node 22) 41 51 40
[hardhat-ignition-core] ci on windows-latest (Node 22) 98 81 80
[hardhat-ignition-ethers] ci on macos-13 (Node 22) 100 97 85
[hardhat-ignition-ethers] ci on macos-latest (Node 22) 52 68 47
[hardhat-ignition-ethers] ci on ubuntu-latest (Node 22) 49 40 38
[hardhat-ignition-ethers] ci on windows-latest (Node 22) 90 80 70
[hardhat-ignition-ui] ci on macos-13 (Node 22) 72 76 122
[hardhat-ignition-ui] ci on macos-latest (Node 22) 50 40 43
[hardhat-ignition-ui] ci on ubuntu-latest (Node 22) 41 44 40
[hardhat-ignition-ui] ci on windows-latest (Node 22) 74 89 90
[hardhat-ignition-viem] ci on macos-13 (Node 22) 133 101 128
[hardhat-ignition-viem] ci on macos-latest (Node 22) 78 63 64
[hardhat-ignition-viem] ci on ubuntu-latest (Node 22) 54 53 50
[hardhat-ignition-viem] ci on windows-latest (Node 22) 99 107 100
[hardhat-ignition] ci on macos-13 (Node 22) 232 319 163
[hardhat-ignition] ci on macos-latest (Node 22) 140 136 120
[hardhat-ignition] ci on ubuntu-latest (Node 22) 104 102 110
[hardhat-ignition] ci on windows-latest (Node 22) 198 161 163
[hardhat-keystore] ci on macos-13 (Node 22) 642 141 96
[hardhat-keystore] ci on macos-latest (Node 22) 268 71 53
[hardhat-keystore] ci on ubuntu-latest (Node 22) 454 56 52
[hardhat-keystore] ci on windows-latest (Node 22) 465 96 83
[hardhat-mocha] ci on macos-13 (Node 22) 115 95 136
[hardhat-mocha] ci on macos-latest (Node 22) 75 47 60
[hardhat-mocha] ci on ubuntu-latest (Node 22) 64 29 30
[hardhat-mocha] ci on windows-latest (Node 22) 175 75 58
[hardhat-network-helpers] ci on macos-13 (Node 22) 126 74 107
[hardhat-network-helpers] ci on macos-latest (Node 22) 59 42 35
[hardhat-network-helpers] ci on ubuntu-latest (Node 22) 43 32 30
[hardhat-network-helpers] ci on windows-latest (Node 22) 162 78 65
[hardhat-node-test-reporter] ci on macos-13 (Node 22) 100 72 88
[hardhat-node-test-reporter] ci on macos-latest (Node 22) 63 33 32
[hardhat-node-test-reporter] ci on ubuntu-latest (Node 22) 33 32 24
[hardhat-node-test-reporter] ci on windows-latest (Node 22) 94 75 70
[hardhat-node-test-runner] ci on macos-13 (Node 22) 61 80 79
[hardhat-node-test-runner] ci on macos-latest (Node 22) 38 38 35
[hardhat-node-test-runner] ci on ubuntu-latest (Node 22) 28 27 28
[hardhat-node-test-runner] ci on windows-latest (Node 22) 68 133 71
[hardhat-test-utils] ci on macos-13 (Node 22) 102 58 70
[hardhat-test-utils] ci on macos-latest (Node 22) 36 37 26
[hardhat-test-utils] ci on ubuntu-latest (Node 22) 29 22 19
[hardhat-test-utils] ci on windows-latest (Node 22) 65 63 60
[hardhat-typechain] ci on macos-13 (Node 22) 150 78 67
[hardhat-typechain] ci on macos-latest (Node 22) 85 43 39
[hardhat-typechain] ci on ubuntu-latest (Node 22) 55 31 36
[hardhat-typechain] ci on windows-latest (Node 22) 171 158 60
[hardhat-utils] ci on macos-13 (Node 22) 71 84 68
[hardhat-utils] ci on macos-latest (Node 22) 42 56 33
[hardhat-utils] ci on ubuntu-latest (Node 22) 43 33 23
[hardhat-utils] ci on windows-latest (Node 22) 76 76 72
[hardhat-viem] ci on macos-13 (Node 22) 165 80 70
[hardhat-viem] ci on macos-latest (Node 22) 80 47 61
[hardhat-viem] ci on ubuntu-latest (Node 22) 77 38 36
[hardhat-viem] ci on windows-latest (Node 22) 196 123 75
[hardhat-zod-utils] ci on macos-13 (Node 22) 62 79 64
[hardhat-zod-utils] ci on macos-latest (Node 22) 44 34 31
[hardhat-zod-utils] ci on ubuntu-latest (Node 22) 27 26 21
[hardhat-zod-utils] ci on windows-latest (Node 22) 96 62 72
[hardhat] ci on macos-13 (Node 22) 379 173 164
[hardhat] ci on macos-latest (Node 22) 235 94 102
[hardhat] ci on ubuntu-latest (Node 22) 234 63 64
[hardhat] ci on windows-latest (Node 22) 256 148 170
[template-package] ci on macos-13 (Node 22) 62 91 75
[template-package] ci on macos-latest (Node 22) 38 47 29
[template-package] ci on ubuntu-latest (Node 22) 23 22 22
[template-package] ci on windows-latest (Node 22) 58 143 46

This PR sets up Codecov for the Hardhat repository. It will allow us to track the following over time:

  • unit test coverage
  • test flakiness (beta)
  • bundle size (with an asset breakdown)

The Codecov integration will also comment on PRs highlighting the changes to the above metrics in relation to the target branch.

⚠️ In this PR, I also remove our custom bundle comments. Instead, we'll rely on Codecov's bundle analysis. There is a slight regression as the comment from Codecov about the bundle size will not include the information about the number of dependencies. This information will only be available after clicking through to the bundle analysis details, also present in the comment. In my opinion, this is acceptable. Especially since, the codecov integration will allow us to track bundle size over time, something we didn't do before.

⚠️ Please note that the coverage tracking does not currently work for the ignition packages. After discussing this internally, I decided not to fix it now since the ignition packages are soon to be migrated to node test runner anyway. This migration will likely come with the coverage support out of the box.

TODO

*Alternatively, we can set up the CODECOV_TOKEN for the entire organization by copying it from the Codecov Organization Configuration and adding the CODECOV_TOKEN secret to the Organization Secrets


This PR has been originally created #6449

@galargh galargh added the no changeset needed This PR doesn't require a changeset label Mar 25, 2025
Copy link

changeset-bot bot commented Mar 25, 2025

🦋 Changeset detected

Latest commit: bae8a8d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@nomicfoundation/hardhat-test-utils Patch
@nomicfoundation/hardhat-utils Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link

vercel bot commented Mar 25, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
hardhat ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 27, 2025 3:20pm

@galargh galargh requested a review from alcuadrado March 25, 2025 15:29
@github-project-automation github-project-automation bot moved this to Backlog in Hardhat Mar 25, 2025
@github-actions github-actions bot added the status:ready This issue is ready to be worked on label Mar 25, 2025
@schaable schaable added the v-next A Hardhat v3 development task label Apr 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
no changeset needed This PR doesn't require a changeset status:ready This issue is ready to be worked on v-next A Hardhat v3 development task
Projects
Status: Backlog
Development

Successfully merging this pull request may close these issues.

2 participants