testing-guide.md

🧪 Testing Guide

This document outlines the testing practices and guidelines for the Thunderbird for Android project.

Key Testing Principles:

• Follow the Arrange-Act-Assert (AAA) pattern
• Use descriptive test names
• Prefer fake implementations over mocks
• Name the object under test as testSubject
• Use AssertK for assertions

📐 Test Structure

🔍 Arrange-Act-Assert Pattern

Tests in this project should follow the Arrange-Act-Assert (AAA) pattern:

1. Arrange: Set up the test conditions and inputs
2. Act: Perform the action being tested
3. Assert: Verify the expected outcomes

Example:

@Test
fun `example test using AAA pattern`() {
    // Arrange
    val input = "test input"
    val expectedOutput = "expected result"
    val testSubject = SystemUnderTest()

    // Act
    val result = testSubject.processInput(input)

    // Assert
    assertThat(result).isEqualTo(expectedOutput)
}

Use comments to clearly separate these sections in your tests:

// Arrange
// Act
// Assert

📝 Test Naming

Use descriptive test names that clearly indicate what is being tested. For JVM tests, use backticks:

@Test
fun `method should return expected result when given valid input`() {
    // Test implementation
}

Note: Android instrumentation tests do not support backticks in test names. For these tests, use camelCase instead:

@Test
fun methodShouldReturnExpectedResultWhenGivenValidInput() {
    // Test implementation
}

💻 Test Implementation

🎭 Fakes over Mocks

In this project, we prefer using fake implementations over mocks:

• ✅ Preferred: Create fake/test implementations of interfaces or classes
• ❌ Avoid: Using mocking libraries to create mock objects

Fakes provide better test reliability and are more maintainable in the long run. They also make tests more readable and less prone to breaking when implementation details change.

Mocks can lead to brittle tests that are tightly coupled to the implementation details, making them harder to maintain. They also negatively impact test performance, particularly during test initialization. Which can quickly become overwhelming when an excessive number of tests includes mock implementations.

Example of a fake implementation:

// Interface
interface DataRepository {
    fun getData(): List<String>
}

// Fake implementation for testing
class FakeDataRepository(
    // Allow passing initial data during construction
    initialData: List<String> = emptyList()
) : DataRepository {
    // Mutable property to allow changing data between tests
    var dataToReturn = initialData

    override fun getData(): List<String> {
        return dataToReturn
    }
}

// In test
@Test
fun `processor should transform data correctly`() {
    // Arrange
    val fakeRepo = FakeDataRepository(listOf("item1", "item2"))
    val testSubject = DataProcessor(fakeRepo)

    // Act
    val result = testSubject.process()

    // Assert
    assertThat(result).containsExactly("ITEM1", "ITEM2")
}

📋 Naming Conventions

When writing tests, use the following naming conventions:

• Name the object under test as testSubject (not "sut" or other abbreviations)
• Name fake implementations with a "Fake" prefix (e.g., FakeDataRepository)
• Use descriptive variable names that clearly indicate their purpose

✅ Assertions

Use AssertK for assertions in tests:

@Test
fun `example test`() {
    // Arrange
    val list = listOf("apple", "banana")

    // Act
    val result = list.contains("apple")

    // Assert
    assertThat(result).isTrue()
    assertThat(list).contains("banana")
}

Note: You'll need to import the appropriate AssertK assertions:

• assertk.assertThat for the base assertion function
• Functions from the assertk.assertions namespace for specific assertion types (e.g., import assertk.assertions.isEqualTo, import assertk.assertions.contains, import assertk.assertions.isTrue, etc.)

🧮 Test Types

This section describes the different types of tests we use in the project. Each type serves a specific purpose in our testing strategy, and together they help ensure the quality and reliability of our codebase.

🔬 Unit Tests

Unit tests verify that individual components work correctly in isolation.

What to Test:

• Single units of functionality
• Individual methods or functions
• Classes in isolation
• Business logic
• Edge cases and error handling

Key Characteristics:

• Independent (no external dependencies)
• No reliance on external resources
• Uses fake implementations for dependencies

Frameworks:

• JUnit 4
• AssertK for assertions
• Robolectric (for Android framework classes)

Location:

• Tests should be in the same module as the code being tested
• Should be in the src/test directory or src/{platformTarget}Test for Kotlin Multiplatform
• Tests should be in the same package as the code being tested

Contributor Expectations:

• ✅ All new code should be covered by unit tests
• ✅ Add tests that reproduce bugs when fixing issues
• ✅ Follow the AAA pattern (Arrange-Act-Assert)
• ✅ Use descriptive test names
• ✅ Prefer fake implementations over mocks

🔌 Integration Tests

Integration tests verify that components work correctly together.

What to Test:

• Interactions between components
• Communication between layers
• Data flow across multiple units
• Component integration points

Key Characteristics:

• Tests multiple components together
• May use real implementations when appropriate
• Focuses on component boundaries

Frameworks:

• JUnit 4 (for tests in src/test)
• AssertK for assertions
• Robolectric (for Android framework classes in src/test)
• Espresso (for UI testing in src/androidTest)

Location:

• Preferably in the src/test or src/commonTest, src/{platformTarget}Test for Kotlin Multiplatform
• Only use src/androidTest, when there's a specific need for Android dependencies

Why prefer test over androidTest:

• JUnit tests run faster (on JVM instead of emulator/device)
• Easier to set up and maintain
• Better integration with CI/CD pipelines
• Lower resource requirements
• Faster feedback during development

When to use androidTest:

• When testing functionality that depends on Android-specific APIs that are not available with Robolectric
• When tests need to interact with the Android framework directly

Contributor Expectations:

• ✅ Add tests for features involving multiple components
• ✅ Focus on critical paths and user flows
• ✅ Be mindful of test execution time
• ✅ Follow the AAA pattern
• ✅ Use descriptive test names

📱 UI Tests

UI tests verify the application from a user's perspective.

What to Test:

• User interface behavior
• UI component interactions
• Complete user flows
• Screen transitions
• Input handling and validation

Key Characteristics:

• Tests from user perspective
• Verifies visual elements and interactions
• Covers end-to-end scenarios

Frameworks:

• Espresso for Android UI testing
• Compose UI testing for Jetpack Compose
• JUnit 4 as the test runner

Location:

• In the src/test directory for Compose UI tests
• In the src/androidTest directory for Espresso tests

Contributor Expectations:

• ✅ Add tests for new UI components and screens
• ✅ Focus on critical user flows
• ✅ Consider different device configurations
• ✅ Test both positive and negative scenarios
• ✅ Follow the AAA pattern
• ✅ Use descriptive test names

📸 Screenshot Tests

⚠️ Work in Progress ⚠️

Screenshot tests verify the visual appearance of UI components.

What to Test:

• Visual appearance of UI components
• Layout correctness
• Visual regressions
• Theme and style application

Key Characteristics:

• Captures visual snapshots
• Compares against reference images (golden images)
• Detects unintended visual changes

Frameworks:

• JUnit 4 as the test runner
• Compose UI testing
• Screenshot comparison tools (TBD)

Location:

• Same module as the code being tested
• In the src/test directory

Contributor Expectations:

• ✅ Add tests for new Composable UI components
• ✅ Verify correct rendering in different states
• ✅ Update reference screenshots for intentional changes

🚫 Test Types We Don't Currently Have

This section helps contributors understand our testing strategy and future plans.

End-to-End Tests ✨

• Full system tests verifying complete user journeys
• Tests across multiple screens and features
• Validates entire application workflows

Performance Tests ⚡

• Measures startup time, memory usage, responsiveness
• Validates app performance under various conditions
• Identifies performance bottlenecks

Accessibility Tests ♿

• Verifies proper content descriptions
• Checks contrast ratios and keyboard navigation
• Ensures app is usable by people with disabilities

Localization Tests 🌐

• Verifies correct translation display
• Tests right-to-left language support
• Validates date, time, and number formatting

Manual Test Scripts 📝

• Manual testing by QA team for exploratory testing
• Ensures repeatable test execution
• Documents expected behavior for manual tests

🏃 Running Tests

Quick commands to run tests in the project.

Run all tests:

./gradlew test

Run tests for a specific module:

./gradlew :module-name:test

Run Android instrumentation tests:

./gradlew connectedAndroidTest

Run tests with coverage verification:

./gradlew koverVerifyDebug

Generate an HTML coverage report:

./gradlew koverHtmlReportDebug

📊 Code Coverage

We use the Kover code coverage plugin with project defaults enforced via our Gradle build. Coverage is verified as part of CI and will fail if requirements are not met.

Minimum requirements:

• Branch coverage: at least 70%
• Line coverage: at least 75%

How to check locally:

• Verify coverage (fails the build on violations):

./gradlew koverVerifyDebug

• For a specific module:

  ./gradlew :module-name:koverVerifyDebug

• Generate a human-readable HTML report:

  ./gradlew koverHtmlReportDebug

Temporarily disable coverage globally (e.g., for local runs or exceptional CI cases):

• Using Gradle property:

  ./gradlew check -PcodeCoverageDisabled=true 

• Using environment variable:

  CODE_COVERAGE_DISABLED=true ./gradlew check

Where to find reports:

• HTML: module/build/reports/kover/html/index.html
• XML: module/build/reports/kover/xml/report.xml

Scope:

• Filters are applied automatically to exclude common framework/generated code (Android, Compose, etc.).
• Targets apply to the aggregated totals reported by Kover for each module/variant.

Exemptions:

• New code should meet the coverage targets above.
• Legacy code should be improved over time but is not required to meet targets immediately.

If you believe an exception is warranted (e.g., trivial or generated code), discuss it with maintainers in the PR before requesting an exemption.

Note

High coverage does not guarantee high-quality tests. Focus on meaningful tests that validate behavior and edge cases while meeting the thresholds above.