Possible issue with partially covered code - Simplecov setup

[luke-hill]

I run Simplecov on site_prism - site-prism/site_prism#33 - This is an area I'm trying to update the coverage % value. For (possibly poor reasons), some code we only unit test and some code we feature test.

• Include the SimpleCov version you are running in your report.
  0.17.x
• Include your ruby -e "puts RUBY_DESCRIPTION".
  ruby -e "puts RUBY_DESCRIPTION"
  ruby: No such file or directory -- 2.3.8p459 (LoadError)
  (But I'm using ruby 2.3 on rvm succesfully)

The link referred has failing outputs. I would like to get the PR's to pass Travis.

The coverage % are as follows

RSpec - 98.4%
Cucumber - 95%
RSpec + Cucumber 99.2%

It could be I've set something up incorrect. But I've followed most of the instructions and tried a few different things. Esentially I want simplecov to only make the check for coverage once, at the very end of the command ran. But it seems to make the check after each rubocop / rspec / cucumber command. I'm not sure why.

[PragTob]

@luke-hill 👋 hi and thanks for the issue report!

How do you run your tests? Usually simplecov checks the coverage "at exit" aka when the testing processes finish (this and the merging). I believe I usually solved this with having one take task run all the tests so that when it finishes, it generates the coverage.

See for instance our default task where we run rubocop, rspec and cucumber all in one: https://github.com/colszowka/simplecov/blob/master/Rakefile#L32

[luke-hill]

https://github.com/site-prism/site_prism/blob/master/Rakefile

[PragTob]

The mentioned PR is merged so I don't think this is a current issue anymore.

@luke-hill if there are still problems please let me know.

[luke-hill]

We still have this "hacky" override in our system. Whereby we test for the lowest bound. So I don't think it's fixed.

I can try make the change again. But I'm not sure it will work.

If you want me to leave this closed I can do, but another ticket would be identical.

[PragTob]

@luke-hill

Can you then please try to phrase the problem again. With the PR merged and everything I don't quite get it. What I understand/think is the following:

• you wanna upgrade the minimum_coverage to reflect the current coverage
• this fails (?) because it aborts your test suite after running rspec but you want it to only measure against rspec + cucumber ?

[luke-hill]

That's correct @PragTob Yes.

So atm my coverage is set to 98.4 - Because that's what rspec covers. But rspec + cucumber together covers almost 100%, so I want to set the value to the higher one.

I'll try it again. But this hasn't changed massively our end. So I think it still fails.

[deivid-rodriguez]

The way I go about this is to use the manual report merging feature, by only enforcing a minimum_coverage after reports have been merged.

So I have a rake task like this:

namespace :coverage do
  desc "Merges all result sets into a single coverage report"
    task :collate do
      ENV["COV_COLLATION"] = "true"

      require "simplecov"

      SimpleCov.collate Dir["coverage/*.json"]
    end
end

and then in the .simplecov file do this

SimpleCov.start

SimpleCov.minimum_coverage 99.2 if ENV["COV_COLLATION"]

[luke-hill]

FYI @PragTob I just did this

# in .simplecov I set the following value
SimpleCov.minimum_coverage 99.1

# Then I cd'd into the project directory
$ bundle exec rake
# lots of lines.....
# lots of lines.....
# lots of lines.....
Finished in 3.29 seconds (files took 0.67888 seconds to load)
308 examples, 0 failures

Randomized with seed 37546

Coverage report generated for RSpec to /home/luke/Code/site_prism/coverage. 1678 / 1702 LOC (98.59%) covered.
Line coverage (98.58%) is below the expected minimum coverage (99.10%).
SimpleCov failed with exit 2
rake aborted!
Specs failed

[deivid-rodriguez]

@PragTob Can you try make the last test suite you run (cucumber I guess) set an environment variable so that when reading .simplecov we know when a minimum coverage needs to be enforced? Something similar to what I wrote about but if you're relying on automerged reports, you don't need to mess with coverage colattion at all.

So, something like

# Rakefile

task "cucumber" do
  sh("ENFORCE_MIN_COVERAGE=true cucumber")
end

# .simplecov

SimpleCov.minimum_coverage 99.1 if ENV["ENFORCE_MIN_COVERAGE"]

Without some external information, I don't think simplecov can do anything about this.

[deivid-rodriguez]

Or even better, remove SimpleCov.minimum_coverage 99.1 from your .simplecov file and add it after all your test tasks run?

[deivid-rodriguez]

Actually my last suggestion wouldn't work, but we could provide a SimpleCov.minimum_coverage! 99.1 to do what I meant.