Fix Codecov 0%: fix coverage.json, rm gem codecov (#2857)

Our earlier commit e2de938 created the file
coverage/codecov-result.json fixing the
"Found 0 coverage files" error, but the site still showed 0%.
Codecov's backend rejected the upload as "Unusable report ...
incorrect data format": the abandoned codecov 0.6.0 gem's formatter
emits the deprecated v2 "network" report format (a filename list, a
"<<<<<< network" separator, then one line of coverage JSON) which isn't
even valid JSON despite the .json name. The pinned Codecov CLI uploaded
it without complaint because `--plugin noop` means it never parses the
file, so the upload step looked healthy while the backend silently
discarded the data.

This commit changes the approach for recording coverage results. Now we
write coverage/coverage.json with SimpleCov::Formatter::JSONFormatter
(from simplecov_json_formatter), whose output Codecov parses natively,
and point the CLI --file at it. As before this runs in test:coverage_gaps
rather than test_helper.rb, because CI runs with DEFER_COVERAGE set
(swapping in SimpleFormatter) and the merged result is only complete once
all parallel + system runs finish.

This removes the abandoned codecov gem entirely and drops its dead formatter
reference from test_helper.rb (it never ran in CI anyway, since
DEFER_COVERAGE overrode it). The codecov gem was the only thing pulling
in simplecov, including its <0.22 pin, so we now list simplecov and
simplecov_json_formatter explicitly: both are required directly (in
test_helper.rb and test:coverage_gaps respectively), so declaring them
prevents a future simplecov release that stops bundling the JSON
formatter from breaking coverage reporting in a surprising way.

Signed-off-by: David A. Wheeler <dwheeler@dwheeler.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: david-a-wheeler

.circleci/config.yml

@@ -177,8 +177,8 @@ jobs:
     # use. This matches the dependency-pinning policy stated at the top of this
     # file and mirrors the Heroku CLI install in the deploy job: we never run
     # an unpinned, unverified binary fetched over the network. The coverage
-    # JSON itself is produced by SimpleCov::Formatter::Codecov; see
-    # test/test_helper.rb. --disable-search + --file restrict the upload to
+    # JSON itself is produced by SimpleCov::Formatter::JSONFormatter in the
+    # test:coverage_gaps rake task. --disable-search + --file restrict the upload to
     # exactly that file, and --plugin noop skips Codecov's coverage-generation
     # plugins (we already have the report). We do NOT pass --fail-on-error, so
     # a Codecov outage cannot break an otherwise-green build. The upload token,
@@ -208,7 +208,7 @@ jobs:
           echo "** Hash verified. Uploading coverage."
           chmod +x codecov
           ./codecov upload-process --disable-search --plugin noop \
-            --file coverage/codecov-result.json
+            --file coverage/coverage.json
           rm -f codecov
 
   # I haven't found a reliable way to calculate HEROKU_APP just once,

Gemfile

@@ -183,7 +183,6 @@ end
 
 group :test do
   gem 'capybara-slow_finder_errors', require: false # ID slow Capybara finders
-  gem 'codecov', require: false # Report test code coverage
   # Pin minitest < 6.0 until minitest-reporters supports it.
   # Minitest 6.0 introduced breaking changes that cause minitest-reporters 1.7.1
   # to fail silently (tests don't run). Remove this constraint when
@@ -196,7 +195,12 @@ group :test do
   # Note: Updating 'rails-controller-testing' to '1.0.5' causes failures
   gem 'rails-controller-testing', '~> 1.0' # for `assigns` and `assert_template`
   gem 'selenium-webdriver' # Automates browser i/f for Rails system testing
-  # We don't list "simplecov"; code depends on it & brings it in
+  # Test coverage. We require both directly: simplecov in test_helper.rb, and
+  # simplecov_json_formatter in test:coverage_gaps to write coverage/coverage.json
+  # for the Codecov upload. List both so a future simplecov release that stops
+  # bundling the JSON formatter can't break us in a surprising way.
+  gem 'simplecov', require: false
+  gem 'simplecov_json_formatter', require: false
   gem 'webmock', '~> 3.0', require: false # Mock HTTP requests for testing
 end
 

Gemfile.lock

@@ -138,8 +138,6 @@ GEM
       concurrent-ruby
     code_analyzer (0.5.5)
       sexp_processor
-    codecov (0.6.0)
-      simplecov (>= 0.15, < 0.22)
     coderay (1.1.3)
     commonmarker (2.6.3)
       rb_sys (~> 0.9)
@@ -532,7 +530,7 @@ GEM
       concurrent-ruby (~> 1.0, >= 1.0.2)
       logger
     sexp_processor (4.17.5)
-    simplecov (0.21.2)
+    simplecov (0.22.0)
       docile (~> 1.1)
       simplecov-html (~> 0.11)
       simplecov_json_formatter (~> 0.1)
@@ -637,7 +635,6 @@ DEPENDENCIES
   bundler-audit
   capybara-slow_finder_errors
   chartkick (~> 5.2)
-  codecov
   commonmarker (~> 2.6.2)
   dotenv (~> 3.0)
   eslintrb
@@ -690,6 +687,8 @@ DEPENDENCIES
   selenium-webdriver
   sentry-rails
   sentry-ruby
+  simplecov
+  simplecov_json_formatter
   solid_queue (~> 1.1)
   spring (~> 4.1)
   sprockets-rails (= 3.5.2)

lib/tasks/default.rake

@@ -963,20 +963,22 @@ task 'test:coverage_gaps' do
 
   puts "Combined Coverage: #{result.covered_percent.round(2)}%"
 
-  # In CI, write coverage/codecov-result.json from the merged result so the
+  # In CI, write coverage/coverage.json from the merged result so the
   # Codecov CLI (see .circleci/config.yml) has a file to upload. This MUST
   # happen here, not in test_helper.rb: the test workers run with
-  # DEFER_COVERAGE set, which swaps in the minimal SimpleFormatter, so the
-  # Codecov formatter never runs during the test processes. The merged
+  # DEFER_COVERAGE set, which swaps in the minimal SimpleFormatter, so a
+  # report formatter never runs during the test processes. The merged
   # result is only complete once all parallel + system test runs finish.
   #
-  # We call the gem's inner Codecov::SimpleCov::Formatter directly rather
-  # than SimpleCov::Formatter::Codecov, because the latter also invokes the
-  # gem's deprecated built-in network uploader. We upload separately with a
-  # pinned, hash-verified CLI, so we want ONLY the file-writing half here.
+  # We use SimpleCov::Formatter::JSONFormatter (from simplecov_json_formatter,
+  # a dependency of simplecov) because Codecov's backend natively parses its
+  # output. We previously used the abandoned 'codecov' gem's formatter, but
+  # it emits the deprecated "network" report format that the current Codecov
+  # backend rejects as "unusable report ... incorrect data format", silently
+  # yielding 0% coverage on the site even though the upload itself succeeds.
   if ENV['CI']
-    require 'codecov'
-    Codecov::SimpleCov::Formatter.new.format(result)
+    require 'simplecov_json_formatter'
+    SimpleCov::Formatter::JSONFormatter.new.format(result)
   end
 
   # Iterate through files using SimpleCov's own 'missed_lines' logic

test/test_helper.rb

@@ -15,16 +15,12 @@
 # *MUST* state VERY EARLY that we're in the test environment.
 ENV['RAILS_ENV'] ||= 'test'
 
-# Configure SimpleCov formatting before we start it
-if ENV['CI']
-  require 'codecov'
-  SimpleCov.formatters = [
-    SimpleCov::Formatter::HTMLFormatter,
-    SimpleCov::Formatter::Codecov
-  ]
-else
-  SimpleCov.formatters = SimpleCov::Formatter::HTMLFormatter
-end
+# Configure SimpleCov formatting before we start it.
+# The HTML report is for humans; the machine-readable JSON report uploaded to
+# Codecov is written separately by the test:coverage_gaps rake task, since CI
+# runs with DEFER_COVERAGE set (which swaps in SimpleFormatter below) and the
+# merged result is only complete once all parallel + system runs finish.
+SimpleCov.formatters = SimpleCov::Formatter::HTMLFormatter
 
 # Start SimpleCov to track coverage
 # NOTE: If you change SimpleCov configuration (used locally), you may also