Skip to content

E2E_FLAKINESS_FIXES.md

Latest commit

 

History

History
218 lines (173 loc) · 5.52 KB

E2E_FLAKINESS_FIXES.md

File metadata and controls

218 lines (173 loc) · 5.52 KB
Raw

E2E Test Flakiness Fixes

Overview

This document outlines the improvements made to fix E2E test flakiness and add robust retry mechanisms.

Changes Made

1. Enhanced Jest Configuration

  • Increased timeout from 30s to 60s for complex operations
  • Added custom test environment with proper setup/teardown
  • Implemented custom test sequencer for stable test ordering
  • Added flakiness detection reporter to identify unstable tests

2. Retry and Stability Utilities

TestRetryHelper

  • Exponential backoff retry logic with configurable parameters
  • Smart retry conditions (don't retry on 4xx errors, retry on network issues)
  • Wait utilities for stable conditions and value stabilization
  • Timeout handling with proper error propagation

TestHttpClient

  • Built-in retry logic for HTTP requests
  • Timeout management with configurable limits
  • Connection pooling awareness for database operations
  • Endpoint readiness checks with polling

TestDatabaseService

  • Connection stability checks before test execution
  • Automatic cleanup between test runs
  • Transaction rollback for failed tests
  • Connection pooling management

3. Custom Test Environment

  • Environment variable management for test isolation
  • Resource cleanup after each test run
  • Database readiness verification
  • External service mocking configuration

4. Flakiness Detection

  • Automated flakiness reporting during test runs
  • Failure rate analysis across multiple test executions
  • Detailed failure pattern identification
  • Recommendations for fixing flaky tests

Usage

Running Standard E2E Tests

npm run test:e2e

Running Stability Tests

# Quick stability check (3 runs)
npm run test:e2e:stability:quick

# Full stability analysis (5 runs)
npm run test:e2e:stability

# Auth-specific stability test
npm run test:e2e:stability:auth

# Extended flakiness detection (10 runs)
npm run test:e2e:flaky

Custom Stability Testing

# Test specific pattern with custom runs
node test/utils/e2e-test-runner.js --pattern "auth.e2e-spec.ts" --runs 8

# Parallel execution (if supported)
node test/utils/e2e-test-runner.js --runs 5 --parallel

Test Structure Improvements

Before (Flaky)

it('should register user', async () => {
  const response = await request(app.getHttpServer())
    .post('/api/auth/register')
    .send(userData);
  expect(response.status).toBe(201);
});

After (Stable)

it('should register user successfully with retries', async () => {
  await retryHelper.withRetry(
    async () => {
      const response = await httpClient.post('/api/auth/register', userData);
      expect(response.status).toBe(201);
      expect(response.body).toHaveProperty('user');
    },
    {
      maxAttempts: 3,
      delayMs: 1000,
      backoffMultiplier: 2,
    },
  );
}, 30000);

Flakiness Detection

The system automatically detects flaky tests based on:

  • Failure rate threshold: 10% or higher
  • Minimum runs: At least 3 executions
  • Pattern analysis: Consistent failure patterns

Sample Flakiness Report

🧪 Running E2E stability test with 5 iterations...

🏃 Run 1/5 starting...
✅ Run 1/5 completed: 8/8 tests passed

🏃 Run 2/5 starting...
✅ Run 2/5 completed: 8/8 tests passed

❌ Run 3/5 completed: 7/8 tests passed
❌ Failed tests: should handle concurrent requests

📊 Stability report saved to: ./test-results/e2e-stability-report-2024-01-15T10-30-00.md

Best Practices for Writing Stable E2E Tests

1. Use Retry Helpers

await retryHelper.withRetry(
  () => httpClient.get('/api/data'),
  { maxAttempts: 3, delayMs: 500 }
);

2. Wait for Conditions

await retryHelper.waitFor(
  () => checkDatabaseReady(),
  { timeout: 10000, description: 'database readiness' }
);

3. Handle Async Operations

await retryHelper.waitForStable(
  () => getQueueLength(),
  { stabilityWindowMs: 1000, checkIntervalMs: 100 }
);

4. Use Explicit Timeouts

it('should complete operation', async () => {
  // Test logic with explicit timeout
}, 30000);

5. Clean Test Data

beforeEach(async () => {
  await testDb.clean();
});

Monitoring and Maintenance

Automated Reporting

  • Flakiness reports are generated automatically
  • Test stability metrics are tracked
  • Recommendations are provided for fixing issues

CI/CD Integration

# In GitHub Actions
- name: Run E2E Stability Tests
  run: npm run test:e2e:stability
  continue-on-error: false

- name: Generate Flakiness Report
  run: npm run test:e2e:flaky
  if: failure()

Troubleshooting

Common Flakiness Causes

  1. Race conditions in async operations
  2. Database connection issues
  3. External service dependencies
  4. Timing-sensitive assertions
  5. Resource cleanup problems

Debugging Flaky Tests

  1. Run stability tests multiple times
  2. Check flakiness reports for patterns
  3. Add detailed logging
  4. Use retry helpers with verbose output
  5. Isolate problematic test cases

Performance Impact

  • Retry logic adds minimal overhead (~100ms per retry)
  • Stability checks ensure reliable test execution
  • Parallel execution available for faster feedback
  • Resource cleanup prevents test interference

Future Improvements

  • Machine learning-based flakiness prediction
  • Automatic test quarantine for flaky tests
  • Performance regression detection
  • Cross-environment stability validation