Ability to see which test suite or even which test file/example/line is covering which line in my code

[TylerRick]

It would be nice if those numbers on the right side of my green lines of code let you hover over them to see this information (or a summary of this information that you could click for the details).

For example, hover over the "4" by line 7 of app/controllers/courses_controller.rb and see:

Covered by:
spec/controllers/courses_controller_spec.rb:17: it assigns all courses as @courses
spec/controllers/courses_controller_spec.rb:23: it renders the "foo" template
features/manage_courses.feature:18: Edit an archived course
features/assign_student_to_course.feature:10: Assigning student to an archived course

Would this be possible?

It would be even cooler if you could click on one of the tests listed and it would jump to the corresponding line in the file view for that test file!

[TylerRick]

See also #69:

currently you can't split results by the test suite they are coming from. It would surely make sense to have different test suites available as separate reports, but the underlying facilities in SimpleCov need to be modified for this to be possible - currently only the one, merged resultset is available after the test suites ran

[colszowka]

Unfortunately, the information coming from the STDLIB Coverage library is not complete enough for that: It only gives a count of hits per line, with no way of saying which code (here being a test) called that particular line. What can be done though is to at least state which test suite hit that line how often. The inner workings of the merging tools inside SimpleCov would have to be reworked for that information to become available however.

[colszowka]

Note to future implementer: Please check that the issue mentioned above, as it makes further suggestions what would be nice to have once this gets tackled.

[tovodeverett]

It would probably slow things down dramatically, but what if one managed to write hooks for common test suites so that on every test the hooks took before and after snapshots of the coverage and then diffed them to figure out what the specific test covered. Of course, this gets into some interesting semantics regarding things like before blocks in RSpec - do before blocks count as part of the test or not? Sometimes what's in the before block is simply setup for the test that shouldn't really count towards coverage, but in other situations the before block contains the code that triggers the actual block under test.

I will be honest - I do some truly evil things with let, lambda, and before blocks in RSpec. It's nice to have the equivalent of dynamic scope on custom testing methods. Using let(:custom_method) { lambda {} } combined with custom_method.call let's me DRY out tests in some interesting ways. I even use this to write test generators using nested lambdas! So frequently there's a lot going on in the before blocks, and there's no easy way to distinguish between what is the fixture and what is the test.

I stumbled across this thread because what I was really looking for wasn't so much per-test information, but rather "what is the coverage for foo/bar/bas.rb as defined by spec/foo/bar/bas_spec.rb". Ideally I would get two coverage figures for every file - one for coverage by anything in the whole test suite, and the other for unit test coverage (i.e. coverage from the unit test for that specific file).

Something else to consider in the rewrite of the merging system would be to better support autotest, so that when it re-runs a limited subset of the specs, it only resets the coverage for the files in question.

Of course, everything seems easy from the peanut gallery!

[tovodeverett]

I realized I could automate my manual process, so in case anyone else finds this thread and wants an implementation of the above, here it is. Basically, it's a script to run rspec over and over again with each of the spec/*/_spec.rb files and then to aggregate the results semi-intelligently. It does its best to identify the file under test for each spec file. If it can't find one, it ignores the spec file. If it can, it runs the spec file with rspec and then snags the simplecov results for the file under test. It does this file by file. At the very end, it runs a comprehensive rspec. Then it overwrites the results from the comprehensive rspec with the file by file ones and generates index_1by1.html from that merged result (leaving the comprehensive rspec results in index.html).

It relies to some extent upon the internals of simplecov and simplecov-html and so may break without warning, but it does let me go get something to eat while it does the dirty work. The only parameters it accepts is a list of spec directories or files to focus on. If you don't give it a list, it covers everything in spec/ except for spec/requests, spec/routes, and spec/views. It does blow away the simplecov cache (i.e. coverage/.resultset.json) while it runs, so any cucumber results will disappear.

The code is not very pretty, but I need to get back to being productive.

#!/usr/bin/ruby

require 'optparse'
require 'find'
require 'simplecov'

class Rspec1by1SimpleCov
  attr_accessor :paths, :individual_results

  def initialize
    @individual_results = {}
    help = false

    opts = OptionParser.new do |opts|
      opts.banner = "Usage: rspec_1by1_simplecov [options] [path1 [path2 [...]]]"
      opts.separator "If no paths are passed, defaults to 'spec'"
      opts.on("--help", "Display this message") { help = true }
    end

    self.paths = opts.parse(*ARGV)
    self.paths = ['spec'] if paths.empty?

    if help
      puts opts.to_s
      exit
    end
  end

  def specs
    unless @specs
      @specs = []
      Find.find(*paths) do |f|
        if File.directory?(f) && %w(spec/requests spec/routes spec/views).include?(f)
          Find.prune
        else
          @specs << f if File.file?(f) && f =~ /_spec\.rb\z/
        end
      end
    end
    @specs
  end

  def run_specs
    specs.each {|spec| run_spec(spec) }
  end

  def run_spec(spec)
    unless file_under_test = spec_file_under_test(spec)
      STDERR.puts "Skipping '#{spec}' - unable to locate file under test"
      return
    end

    purge_resultset
    puts "Running rspec #{spec}"
    system('rspec', spec)
    individual_results[file_under_test] = SimpleCov::ResultMerger.merged_result.original_result[file_under_test]
  end

  def spec_file_under_test(spec)
    file_under_test = spec.dup
    file_under_test.sub!(/\Aspec\//, '') or raise StandardError, "Unable to find spec at beginning of '#{spec}'"
    file_under_test.sub!(/_spec\.rb\z/, '.rb') or raise StandardError, "Unable to find _spec at end of '#{spec}'"
    ['', 'app/'].each do |prefix|
      return File.expand_path("#{prefix}#{file_under_test}") if File.exists?("#{prefix}#{file_under_test}")
    end
    return nil
  end

  def purge_resultset
    resultset_path = SimpleCov::ResultMerger.resultset_path
    File.delete(resultset_path) rescue nil
    raise StandardError, "Unable to remove '#{resultset_path}'" if File.exist?(resultset_path)
  end

  def run_all_specs
    purge_resultset
    puts "Running rspec"
    system('rspec')
  end

  def merge_and_generate_html
    puts "Merging results"

    all_results = SimpleCov::ResultMerger.merged_result
    merged_result = SimpleCov::Result.new(all_results.original_result.merge(individual_results))
    merged_result.command_name = all_results.command_name

    formatter = SimpleCov::Formatter::HTMLFormatter.new
    output_path = formatter.send(:output_path)
    File.rename(File.join(output_path, "index.html"), File.join(output_path, "index_1by1_all.html"))
    formatter.format(merged_result)
    File.rename(File.join(output_path, "index.html"), File.join(output_path, "index_1by1.html"))
    File.rename(File.join(output_path, "index_1by1_all.html"), File.join(output_path, "index.html"))
  end

  def run_everything
    run_specs
    run_all_specs
    merge_and_generate_html
  end
end

rspec1by1simplecov = Rspec1by1SimpleCov.new
rspec1by1simplecov.run_everything

[tyrannosaurus-becks]

To add to this, it's now possible using https://github.com/nebulab/reverse_coverage, but it's not actually possible to use reverse_coverage and simplecov together. So this is still a valid feature request! 😄

Our use case is, we'd like to selectively run RSpec tests. We only want to run the ones that cover files that have changed. If we could understand which test ran against each file, we could build a dictionary like {"file_1.rb": ["file_1a_spec.rb", "file_1b_spec.rb"]} which we could check to decide which RSpecs to run based on the files that changed.