Code coverage is not reported correctly in some circumstances

[johlju]

Checklist

• Issue has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

What is the issue?

When running tests against a module script file that has a using module statement at the top of module script file in some cicumstances the code coverage is not reported accuratly what the tests actually covers.

There are 4 workarounds for this issue

1. (non feasible) comment the line using module in SqlServerDsc.psm1
2. (non feasible) change the order the tests are run
3. switch to use the new code coverage method (not feasible yet due to Code coverage fails in some circumstances when UseBreakpoints is $false #2306).
4. in the tests for Assert-ManagedServiceType add a mock for ConvertFrom-ManagedServiceType.

But what I'm wondering is why I have to add the alternativ 4 workaround for it to work.

When running the code as written in the "How to reproduce" it does not cover the two line shown below. Doing any of the workarounds it will correctly cover 100%.

C:\source\DebugCodeCoverage2> Invoke-Pester -Configuration $pesterConfig
Pester v5.4.0

Starting discovery in 2 files.
Discovery found 4 tests in 33ms.
Starting code coverage.
Code Coverage preparation finished after 8 ms.
Running tests.

Running tests from 'C:\source\DebugCodeCoverage2\Assert-ManagedServiceType.Tests.ps1'
Describing Assert-ManagedServiceType
 Context When types match
   [+] Should not throw an exception 32ms (29ms|3ms)

Running tests from 'C:\source\DebugCodeCoverage2\ConvertFrom-ManagedServiceType.Tests.ps1'
Describing ConvertFrom-ManagedServiceType
 Context When translating to a normalized service types
   [+] Should properly map 'SqlServer' to normalized service type 'DatabaseEngine' 8ms (2ms|6ms)
   [+] Should properly map 'SqlAgent' to normalized service type 'SqlServerAgent' 2ms (2ms|1ms)
   [+] Should properly map 'Search' to normalized service type 'Search' 3ms (2ms|1ms)
Tests completed in 180ms
Tests Passed: 4, Failed: 0, Skipped: 0 NotRun: 0
Processing code coverage result.
Code Coverage result processed in 6 ms.
Covered 71.43% / 75%. 7 analyzed Commands in 1 File.
Missed commands:

File              Class Function                       Line Command
----              ----- --------                       ---- -------
SqlServerDsc.psm1       ConvertFrom-ManagedServiceType   55 $serviceTypeValue = 'SqlServerAgent'
SqlServerDsc.psm1       ConvertFrom-ManagedServiceType   62 $serviceTypeValue = 'Search'

Expected Behavior

The tests to pass and report 100% code coverage as the tests do individually.

Steps To Reproduce

File SqlServerDsc.psm1

# (non feasible) workaround 1 - Comment `using module` from SqlServerDsc.psm1
using module .\DependentClassModule.psm1

function Assert-ManagedServiceType
{
    [OutputType()]
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [Object]
        $ServiceObject,

        [Parameter(Mandatory = $true)]
        [ValidateSet('DatabaseEngine', 'SqlServerAgent', 'Search', 'IntegrationServices', 'AnalysisServices', 'ReportingServices', 'SQLServerBrowser', 'NotificationServices')]
        [System.String]
        $ServiceType
    )

    process
    {
        $normalizedServiceType = ConvertFrom-ManagedServiceType -ServiceType $ServiceObject.Type

        if ($normalizedServiceType -ne $ServiceType)
        {
            # Removed to minimize code
        }
    }
}

function ConvertFrom-ManagedServiceType
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [System.String]
        $ServiceType
    )

    process
    {
        switch ($ServiceType)
        {
            'SqlServer'
            {
                $serviceTypeValue = 'DatabaseEngine'

                break
            }

            'SqlAgent'
            {
                $serviceTypeValue = 'SqlServerAgent'

                break
            }

            'Search'
            {
                $serviceTypeValue = 'Search'

                break
            }
        }

        return $serviceTypeValue
    }
}

File DependentClassModule.psm1

enum Ensure
{
    Present
    Absent
}

File Assert-ManagedServiceType.Tests.ps1

BeforeAll {
    Import-Module -Name './SqlServerDsc.psm1' -Force
}

AfterAll {
    # Unload the module being tested so that it doesn't impact any other tests.
    Get-Module -Name 'SqlServerDsc' -All | Remove-Module -Force
}

Describe 'Assert-ManagedServiceType' -Tag 'Private' {
    BeforeAll {
        $mockServiceObject = @{
            Type = 'SqlServer'
        }
    }

    Context 'When types match' {
        # ------
        # Workaround 3 - In tests for `Assert-ManagedServiceType` add a mock for `ConvertFrom-ManagedServiceType`.
        # ------

        # BeforeAll {
        #     Mock -CommandName ConvertFrom-ManagedServiceType -ModuleName 'SqlServerDsc' -MockWith {
        #         return 'DatabaseEngine'
        #     }
        # }

        It 'Should not throw an exception' {
            $_.MockServiceObject = $mockServiceObject

            InModuleScope -Parameter $_ -ModuleName 'SqlServerDsc' -ScriptBlock {
                Set-StrictMode -Version 1.0

                {
                    $MockServiceObject |
                        Assert-ManagedServiceType -ServiceType 'DatabaseEngine' -ErrorAction 'Stop'
                } | Should -Not -Throw
            }
        }
    }
}

File ConvertFrom-ManagedServiceType.Tests.ps1

BeforeAll {
    Import-Module -Name './SqlServerDsc.psm1' -Force
}

AfterAll {
    # Unload the module being tested so that it doesn't impact any other tests.
    Get-Module -Name 'SqlServerDsc' -All | Remove-Module -Force
}

Describe 'ConvertFrom-ManagedServiceType' -Tag 'Private' {
    Context 'When translating to a normalized service types' {
        BeforeDiscovery {
            $testCases = @(
                @{
                    MockServiceType  = 'SqlServer'
                    MockExpectedType = 'DatabaseEngine'
                }

                @{
                    MockServiceType  = 'SqlAgent'
                    MockExpectedType = 'SqlServerAgent'
                }

                @{
                    MockServiceType  = 'Search'
                    MockExpectedType = 'Search'
                }
            )
        }

        It 'Should properly map ''<MockServiceType>'' to normalized service type ''<MockExpectedType>''' -ForEach $testCases {
            InModuleScope -Parameters $_ -ModuleName 'SqlServerDsc' -ScriptBlock {
                Set-StrictMode -Version 1.0

                # Get the ManagedServiceType
                $managedServiceType = ConvertFrom-ManagedServiceType -ServiceType $MockServiceType

                $managedServiceType | Should -BeOfType [System.String]
                $managedServiceType | Should -Be $MockExpectedType
            }
        }
    }
}

File debug.ps1

$pesterConfig = New-PesterConfiguration -Hashtable @{
    CodeCoverage = @{
        Enabled = $true
        Path = './SqlServerDsc.psm1'
        OutputPath = './Pester_coverage.xml'
        UseBreakpoints = $true
    }
    Run = @{
        Path = @(
            # (non feasible) workaround 2 - Change the order the tests are run
            # Change order and the coverage become 100%
            '.\Assert-ManagedServiceType.Tests.ps1'
            '.\ConvertFrom-ManagedServiceType.Tests.ps1'
        )
    }
    Output = @{
        Verbosity = 'Detailed'
    }
}

Invoke-Pester -Configuration $pesterConfig

Describe your environment

Pester version     : 5.4.0 C:\source\SqlServerDsc\output\RequiredModules\Pester\5.4.0\Pester.psm1                       
PowerShell version : 7.3.2
OS version         : Microsoft Windows NT 10.0.22623.0

Possible Solution?

Unknown :/

[fflaten]

Thanks for the report! I'm able to reproduce this and it's a problem with PowerShell.

I have no idea why using module would cause the breakpoint actions not to run, but we'll have to add it to the long list of weird class behavior. 😞 Would you mind posting an issue in the PowerShell-repo or should I?

Replace debug.ps1 with the following for a neutral repro:

Get-PSBreakpoint | Remove-PSBreakpoint

$null = Set-PSBreakpoint -Script ./SqlServerDsc.psm1 -Line 20 -Action { Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" }
$null = Set-PSBreakpoint -Script ./SqlServerDsc.psm1 -Line 38 -Action { Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" }
$null = Set-PSBreakpoint -Script ./SqlServerDsc.psm1 -Line 40 -Action { Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" }
$null = Set-PSBreakpoint -Script ./SqlServerDsc.psm1 -Line 46 -Action { Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" }
$null = Set-PSBreakpoint -Script ./SqlServerDsc.psm1 -Line 52 -Action { Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" }
$null = Set-PSBreakpoint -Script ./SqlServerDsc.psm1 -Line 58 -Action { Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" }
$null = Set-PSBreakpoint -Script ./SqlServerDsc.psm1 -Line 22 -Action { Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" }

Get-PSBreakpoint

Import-Module -Name './SqlServerDsc.psm1' -Force
& (Get-Module SqlServerDsc) { ConvertFrom-ManagedServiceType -ServiceType 'SqlServer' }
Get-Module -Name 'SqlServerDsc' -All | Remove-Module -Force

Get-PSBreakpoint

Import-Module -Name './SqlServerDsc.psm1' -Force
& (Get-Module SqlServerDsc) { ConvertFrom-ManagedServiceType -ServiceType 'SqlServer' }
& (Get-Module SqlServerDsc) { ConvertFrom-ManagedServiceType -ServiceType 'SqlAgent' }
& (Get-Module SqlServerDsc) { ConvertFrom-ManagedServiceType -ServiceType 'Search' }
Get-Module -Name 'SqlServerDsc' -All | Remove-Module -Force

Output:

 ID Script            Line Command Variable Action
 -- ------            ---- ------- -------- ------
344 SqlServerDsc.psm1   20                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
345 SqlServerDsc.psm1   38                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
346 SqlServerDsc.psm1   40                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
347 SqlServerDsc.psm1   46                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
348 SqlServerDsc.psm1   52                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
349 SqlServerDsc.psm1   58                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
350 SqlServerDsc.psm1   22                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 

Hello world from 345 on line 38:0
Hello world from 346 on line 40:0
Hello world from 349 on line 58:0
DatabaseEngine

# All breakpoints still exist, but none are triggered after the second module import
 ID Script            Line Command Variable Action
 -- ------            ---- ------- -------- ------
344 SqlServerDsc.psm1   20                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
345 SqlServerDsc.psm1   38                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
346 SqlServerDsc.psm1   40                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
347 SqlServerDsc.psm1   46                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
348 SqlServerDsc.psm1   52                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
349 SqlServerDsc.psm1   58                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 
350 SqlServerDsc.psm1   22                   Write-Host "Hello world from $($_.Id) on line $($_.Line):$($_.Column)" 

DatabaseEngine
SqlServerAgent
Search

[johlju]

Would you mind posting an issue in the PowerShell-repo or should I?

@fflaten if you have time I would much appreciate if you could do it. 🙇 But since there is a workaround for issue #2306 this issue no longer as important as the solution for this issue is to move to the new code coverage method (UseBreakpoints = $false). 🤔

[fflaten]

Bug reported: PowerShell/PowerShell#19222