Skip to content

Testing

Jump to bottom
Eric Fitzgerald edited this page Jan 24, 2026 · 9 revisions

Testing

This guide covers testing strategies, tools, and practices for TMI development including unit tests, integration tests, API tests, and end-to-end tests.

Table of Contents

Testing Philosophy

TMI follows a comprehensive testing approach:

  1. Unit Tests - Fast tests with no external dependencies
  2. Integration Tests - Tests with real database and services
  3. API Tests - Complete API workflow testing with Postman/Newman
  4. E2E Tests - Full user journey testing with Cypress

Test Pyramid

        /\
       /E2E\          Few, slow, expensive
      /------\
     /  API  \        Some, medium speed
    /----------\
   /Integration\     More, medium speed
  /--------------\
 /   Unit Tests  \   Many, fast, cheap
/------------------\

Testing Principles

  • Test business logic thoroughly - Unit test all business rules
  • Test integration points - Verify components work together
  • Test user workflows - Ensure complete features work end-to-end
  • Automate everything - All tests should be automated
  • Fast feedback - Unit tests run in seconds
  • Realistic testing - Integration tests use real databases

Unit Testing

Server Unit Tests (Go)

TMI server uses Go's built-in testing framework.

Running Unit Tests

# Run all unit tests
make test-unit

# Run specific test
go test -v ./api -run TestCreateThreatModel

# Run with coverage
make test-coverage-unit

Writing Unit Tests

Test File Naming: *_test.go

Example Test:

// api/threat_model_test.go
package api

import (
    "testing"
    "github.com/stretchr/testify/assert"
)

func TestCreateThreatModel(t *testing.T) {
    // Arrange
    tm := ThreatModel{
        Name:        "Test Threat Model",
        Description: stringPtr("Test description"),
    }

    // Act
    result, err := createThreatModelLogic(tm)

    // Assert
    assert.NoError(t, err)
    assert.NotEmpty(t, result.ID)
    assert.Equal(t, tm.Name, result.Name)
}

Test Patterns

Table-Driven Tests:

func TestAuthorizationRoles(t *testing.T) {
    tests := []struct {
        name     string
        role     string
        canRead  bool
        canWrite bool
        canDelete bool
    }{
        {"owner", "owner", true, true, true},
        {"writer", "writer", true, true, false},
        {"reader", "reader", true, false, false},
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            assert.Equal(t, tt.canRead, canRead(tt.role))
            assert.Equal(t, tt.canWrite, canWrite(tt.role))
            assert.Equal(t, tt.canDelete, canDelete(tt.role))
        })
    }
}

Mocking External Dependencies:

type MockDatabase struct {
    mock.Mock
}

func (m *MockDatabase) GetThreatModel(id string) (*ThreatModel, error) {
    args := m.Called(id)
    return args.Get(0).(*ThreatModel), args.Error(1)
}

func TestWithMock(t *testing.T) {
    // Create mock
    mockDB := new(MockDatabase)
    mockDB.On("GetThreatModel", "123").Return(&ThreatModel{
        ID: "123",
        Name: "Test",
    }, nil)

    // Use mock in test
    tm, err := mockDB.GetThreatModel("123")

    assert.NoError(t, err)
    assert.Equal(t, "123", tm.ID)
    mockDB.AssertExpectations(t)
}

Web App Unit Tests (Angular/TypeScript)

TMI-UX uses Vitest for unit testing.

Running Unit Tests

# Run all tests
pnpm run test

# Run in watch mode
pnpm run test:watch

# Run with UI
pnpm run test:ui

# Run specific test
pnpm run test -- src/app/pages/tm/tm.component.spec.ts

# Coverage report
pnpm run test:coverage

Writing Unit Tests

Test File Naming: *.spec.ts

Example Component Test:

// src/app/pages/tm/tm.component.spec.ts
import { ComponentFixture, TestBed } from '@angular/core/testing';
import { TmComponent } from './tm.component';
import { ApiService } from '../../core/services/api.service';
import { of } from 'rxjs';

describe('TmComponent', () => {
  let component: TmComponent;
  let fixture: ComponentFixture<TmComponent>;
  let mockApiService: jasmine.SpyObj<ApiService>;

  beforeEach(async () => {
    // Create mock
    mockApiService = jasmine.createSpyObj('ApiService', ['getThreatModels']);

    await TestBed.configureTestingModule({
      imports: [TmComponent],
      providers: [
        { provide: ApiService, useValue: mockApiService }
      ]
    }).compileComponents();

    fixture = TestBed.createComponent(TmComponent);
    component = fixture.componentInstance;
  });

  it('should create', () => {
    expect(component).toBeTruthy();
  });

  it('should load threat models on init', () => {
    // Arrange
    const mockThreatModels = [
      { id: '1', name: 'TM 1' },
      { id: '2', name: 'TM 2' }
    ];
    mockApiService.getThreatModels.and.returnValue(of(mockThreatModels));

    // Act
    component.ngOnInit();

    // Assert
    expect(mockApiService.getThreatModels).toHaveBeenCalled();
    expect(component.threatModels).toEqual(mockThreatModels);
  });
});

Service Test:

// src/app/core/services/api.service.spec.ts
import { TestBed } from '@angular/core/testing';
import { HttpClientTestingModule, HttpTestingController } from '@angular/common/http/testing';
import { ApiService } from './api.service';

describe('ApiService', () => {
  let service: ApiService;
  let httpMock: HttpTestingController;

  beforeEach(() => {
    TestBed.configureTestingModule({
      imports: [HttpClientTestingModule],
      providers: [ApiService]
    });

    service = TestBed.inject(ApiService);
    httpMock = TestBed.inject(HttpTestingController);
  });

  afterEach(() => {
    httpMock.verify();
  });

  it('should fetch threat models', () => {
    const mockThreatModels = [{ id: '1', name: 'TM 1' }];

    service.getThreatModels().subscribe(tms => {
      expect(tms).toEqual(mockThreatModels);
    });

    const req = httpMock.expectOne('/api/threat_models');
    expect(req.request.method).toBe('GET');
    req.flush(mockThreatModels);
  });
});

Integration Testing

Integration tests verify that components work correctly with real databases and services.

Server Integration Tests (Go)

Running Integration Tests

# Run all integration tests (automatic setup and cleanup)
make test-integration

# This automatically:
# 1. Starts PostgreSQL container
# 2. Starts Redis container
# 3. Runs migrations
# 4. Starts server
# 5. Runs tests
# 6. Cleans up everything

Test Configuration

Integration tests use dedicated ports to avoid conflicts:

  • PostgreSQL: Port 5434 (vs 5432 for development)
  • Redis: Port 6381 (vs 6379 for development)
  • Server: Port 8080

Writing Integration Tests

Test File Naming: *_integration_test.go

Example:

// api/threat_model_integration_test.go
package api

import (
    "testing"
    "net/http"
    "net/http/httptest"
    "github.com/stretchr/testify/assert"
)

func TestDatabaseThreatModelIntegration(t *testing.T) {
    suite := SetupIntegrationTest(t)
    defer suite.TeardownIntegrationTest(t)

    // Create threat model
    threatModelData := map[string]interface{}{
        "name": "Integration Test TM",
        "description": "Test with real database",
    }

    req := suite.makeAuthenticatedRequest("POST", "/threat_models", threatModelData)
    w := suite.executeRequest(req)

    assert.Equal(t, http.StatusCreated, w.Code)

    // Verify in database
    var tm ThreatModel
    err := suite.db.First(&tm).Error
    assert.NoError(t, err)
    assert.Equal(t, "Integration Test TM", tm.Name)
}

Test Data Management

Predictable Test Users (using login hints):

func createTestUser(hint string) (*User, string) {
    // Create specific test user 'alice@test.tmi' instead of random
    resp, _ := http.Get(
        "http://localhost:8080/oauth2/authorize?idp=test&login_hint=" + hint
    )

    // Parse token from response
    token := parseTokenFromResponse(resp)
    return &User{Email: hint + "@test.tmi"}, token
}

func TestMultiUserScenario(t *testing.T) {
    alice, aliceToken := createTestUser("alice")
    bob, bobToken := createTestUser("bob")

    // Test with both users
}

Test Patterns

Complete Entity Lifecycle:

func TestThreatModelLifecycle(t *testing.T) {
    suite := SetupIntegrationTest(t)
    defer suite.TeardownIntegrationTest(t)

    // 1. Create
    createReq := suite.makeAuthenticatedRequest("POST", "/threat_models", data)
    createW := suite.executeRequest(createReq)
    assert.Equal(t, http.StatusCreated, createW.Code)
    tmID := parseID(createW.Body)

    // 2. Read
    getReq := suite.makeAuthenticatedRequest("GET", "/threat_models/" + tmID, nil)
    getW := suite.executeRequest(getReq)
    assert.Equal(t, http.StatusOK, getW.Code)

    // 3. Update
    updateReq := suite.makeAuthenticatedRequest("PUT", "/threat_models/" + tmID, updatedData)
    updateW := suite.executeRequest(updateReq)
    assert.Equal(t, http.StatusOK, updateW.Code)

    // 4. Delete
    deleteReq := suite.makeAuthenticatedRequest("DELETE", "/threat_models/" + tmID, nil)
    deleteW := suite.executeRequest(deleteReq)
    assert.Equal(t, http.StatusNoContent, deleteW.Code)

    // 5. Verify deletion
    verifyReq := suite.makeAuthenticatedRequest("GET", "/threat_models/" + tmID, nil)
    verifyW := suite.executeRequest(verifyReq)
    assert.Equal(t, http.StatusNotFound, verifyW.Code)
}

Authorization Testing:

func TestAuthorizationMatrix(t *testing.T) {
    suite := SetupIntegrationTest(t)
    defer suite.TeardownIntegrationTest(t)

    alice, aliceToken := createTestUser("alice")
    bob, bobToken := createTestUser("bob")

    // Alice creates threat model
    tm := createThreatModel(aliceToken)

    // Test reader permissions
    addAuthorization(tm.ID, bob.Email, "reader", aliceToken)

    // Bob can read
    getReq := makeRequestWithToken("GET", "/threat_models/" + tm.ID, nil, bobToken)
    assert.Equal(t, http.StatusOK, suite.executeRequest(getReq).Code)

    // Bob cannot write
    updateReq := makeRequestWithToken("PUT", "/threat_models/" + tm.ID, data, bobToken)
    assert.Equal(t, http.StatusForbidden, suite.executeRequest(updateReq).Code)

    // Bob cannot delete
    deleteReq := makeRequestWithToken("DELETE", "/threat_models/" + tm.ID, nil, bobToken)
    assert.Equal(t, http.StatusForbidden, suite.executeRequest(deleteReq).Code)
}

OpenAPI-Driven Integration Test Framework

TMI uses an OpenAPI-driven integration test framework located in test/integration/. The framework provides:

  • OAuth Authentication: Automated OAuth flows via the OAuth callback stub
  • Request Building: Type-safe request construction with fixtures
  • Response Validation: OpenAPI schema validation for all responses
  • Assertion Helpers: Specialized assertions for API responses

Framework Structure

test/integration/
├── framework/
│   ├── client.go       # HTTP client with authentication
│   ├── oauth.go        # OAuth authentication utilities
│   ├── fixtures.go     # Test data fixtures
│   ├── assertions.go   # Test assertion helpers
│   └── database.go     # Database utilities
├── spec/
│   ├── schema_loader.go    # OpenAPI schema loading
│   └── openapi_validator.go # Response validation
└── workflows/
    ├── example_test.go          # Framework demonstration
    ├── oauth_flow_test.go       # OAuth tests
    ├── threat_model_crud_test.go # Threat model CRUD
    ├── diagram_crud_test.go     # Diagram CRUD
    ├── user_operations_test.go  # User operations
    ├── user_preferences_test.go # User preferences
    └── admin_promotion_test.go  # Admin promotion

Writing Framework Tests

package workflows

import (
    "os"
    "testing"
    "github.com/ericfitz/tmi/test/integration/framework"
)

func TestResourceCRUD(t *testing.T) {
    // Skip if not running integration tests
    if os.Getenv("INTEGRATION_TESTS") != "true" {
        t.Skip("Skipping integration test")
    }

    serverURL := os.Getenv("TMI_SERVER_URL")
    if serverURL == "" {
        serverURL = "http://localhost:8080"
    }

    // Ensure OAuth stub is running
    if err := framework.EnsureOAuthStubRunning(); err != nil {
        t.Fatalf("OAuth stub not running: %v", err)
    }

    // Authenticate
    userID := framework.UniqueUserID()
    tokens, err := framework.AuthenticateUser(userID)
    framework.AssertNoError(t, err, "Authentication failed")

    // Create client
    client, err := framework.NewClient(serverURL, tokens)
    framework.AssertNoError(t, err, "Client creation failed")

    // Use subtests for each operation
    t.Run("Create", func(t *testing.T) {
        fixture := framework.NewThreatModelFixture().
            WithName("Test Model")

        resp, err := client.Do(framework.Request{
            Method: "POST",
            Path:   "/threat_models",
            Body:   fixture,
        })
        framework.AssertNoError(t, err, "Request failed")
        framework.AssertStatusCreated(t, resp)
    })
}

Test Coverage Plan

TMI aims for 100% API coverage (178 operations across 92 paths) organized in three tiers:

Tier Purpose Run Frequency Time Budget
Tier 1 Core workflows (OAuth, CRUD) Every commit < 2 min
Tier 2 Feature tests (metadata, webhooks, addons) Nightly < 10 min
Tier 3 Edge cases & admin operations Weekly < 15 min

For the complete integration test plan including implementation roadmap and coverage tracking, see the source documentation at docs/migrated/developer/testing/integration-test-plan.md.

API Testing

TMI uses Postman collections and Newman for comprehensive API testing.

Running API Tests

# Run all API tests
make test-api

# Or run manually
cd postman
./run-tests.sh

Test Collections

Located in /postman directory:

  • comprehensive-test-collection.json - Main test suite
  • unauthorized-tests-collection.json - 401 error testing
  • threat-crud-tests-collection.json - Threat CRUD operations
  • metadata-tests-collection.json - Metadata operations
  • permission-matrix-tests-collection.json - Authorization testing
  • bulk-operations-tests-collection.json - Batch operations

Test Coverage

API tests cover:

  • 70+ endpoints
  • 91 workflow methods
  • All HTTP status codes (200, 201, 204, 400, 401, 403, 404, 409, 422, 500)
  • Authentication and authorization
  • CRUD operations for all entities
  • Metadata operations
  • Batch operations
  • Error scenarios

Threat Model API Coverage Analysis

A comprehensive coverage analysis tracks test coverage across all 41 threat model paths (105 operations):

Metric Value
Total Threat Model Paths 41
Total Operations 105
Operations with Success Tests ~85 (81%)
Operations with 401 Tests ~25 (24%)
Operations with 403 Tests ~15 (14%)
Operations with 404 Tests ~35 (33%)
Operations with 400 Tests ~30 (29%)

Key Gap Areas:

  • Sub-resource 401 tests (threats, diagrams, metadata endpoints)
  • Authorization 403 tests for writer/reader role scenarios
  • Rate limit (429) and server error (500) tests

Collection Files (test/postman/):

  • comprehensive-test-collection.json - Full workflow tests
  • unauthorized-tests-collection.json - 401 authentication tests
  • permission-matrix-tests-collection.json - Multi-user authorization
  • threat-crud-tests-collection.json - Threat entity CRUD
  • collaboration-tests-collection.json - WebSocket collaboration
  • advanced-error-scenarios-collection.json - 409, 422 edge cases

For complete coverage matrix, gap analysis, and implementation recommendations, see docs/migrated/developer/testing/postman-threat-model-coverage.md.

Writing Postman Tests

Basic Test:

pm.test("Status code is 200", function () {
    pm.response.to.have.status(200);
});

pm.test("Response has threat models", function () {
    const response = pm.response.json();
    pm.expect(response).to.be.an('array');
    pm.expect(response.length).to.be.above(0);
});

Advanced Test with Setup:

// Pre-request Script
const data = {
    name: "Test Threat Model",
    description: "Created by test"
};
pm.collectionVariables.set("threat_model_data", JSON.stringify(data));

// Test Script
pm.test("Threat model created", function () {
    pm.response.to.have.status(201);
    const response = pm.response.json();

    pm.expect(response).to.have.property('id');
    pm.expect(response.name).to.equal("Test Threat Model");

    // Save ID for subsequent tests
    pm.collectionVariables.set("threat_model_id", response.id);
});

End-to-End Testing

TMI-UX uses Cypress for E2E testing.

Running E2E Tests

# Run all E2E tests
pnpm run test:e2e

# Open Cypress GUI
pnpm run test:e2e:open

# Run specific spec
pnpm run test:e2e -- --spec="cypress/e2e/login.cy.ts"

Writing E2E Tests

Test File Naming: *.cy.ts

Example Login Test:

// cypress/e2e/login.cy.ts
describe('Login Flow', () => {
  beforeEach(() => {
    cy.visit('/');
  });

  it('should display login page', () => {
    cy.contains('Sign In').should('be.visible');
  });

  it('should login with test provider', () => {
    cy.contains('Test Login').click();
    cy.url().should('include', '/dashboard');
    cy.contains('Threat Models').should('be.visible');
  });
});

Example Diagram Test:

// cypress/e2e/diagram.cy.ts
describe('Diagram Editor', () => {
  beforeEach(() => {
    cy.login(); // Custom command
    cy.visit('/threat-models/123/diagrams/456');
  });

  it('should add process to diagram', () => {
    // Open shape palette
    cy.get('[data-cy=shape-palette]').click();

    // Select process shape
    cy.get('[data-cy=shape-process]').click();

    // Click on canvas to add
    cy.get('[data-cy=diagram-canvas]').click(200, 200);

    // Verify process added
    cy.get('[data-shape=process]').should('exist');
  });

  it('should edit process label', () => {
    cy.get('[data-shape=process]').first().dblclick();
    cy.get('[data-cy=label-input]').clear().type('Authentication Service');
    cy.get('[data-cy=label-save]').click();

    cy.get('[data-shape=process]').first()
      .should('contain', 'Authentication Service');
  });
});

WebSocket Testing

Manual WebSocket Testing

TMI provides a WebSocket test harness for manual testing:

# Build test harness
make build-wstest

# Run 3-terminal test (alice as host, bob and charlie as participants)
make wstest

# Run monitor mode
make monitor-wstest

# Clean up
make clean-wstest

Automated WebSocket Testing

Test File: postman/collaboration-tests-collection.json

Tests WebSocket functionality:

  • Session creation and joining
  • Diagram operations broadcast
  • Presenter mode
  • Cursor sharing
  • User join/leave events

CATS Security Fuzzing

CATS (Contract-driven Automatic Testing Suite) is a security fuzzing tool that tests API endpoints for vulnerabilities and spec compliance.

What is CATS

CATS automatically generates, runs, and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.

Features:

  • Boundary testing (very long strings, large numbers)
  • Type confusion testing
  • Required field validation
  • Authentication bypass testing
  • Malformed input handling

Running CATS

# Full fuzzing with OAuth authentication
make cats-fuzz

# Fuzz with specific user
make cats-fuzz-user USER=alice

# Fuzz specific endpoint
make cats-fuzz-path ENDPOINT=/addons

# Analyze results
make analyze-cats-results

Public Endpoint Handling

TMI has 17 public endpoints (OAuth, OIDC, SAML) that are intentionally accessible without authentication per RFC specifications:

Public Endpoint Categories:

Category Count RFC/Standard
OIDC Discovery 5 RFC 8414
OAuth Flow 6 RFC 6749
SAML Flow 6 SAML 2.0

Implementation:

  • Marked with x-public-endpoint: true vendor extension in OpenAPI spec
  • CATS uses --skipFuzzersForExtension=x-public-endpoint=true:BypassAuthentication to skip auth bypass tests
  • All other security fuzzers still run on these endpoints

Cacheable Endpoints:

6 discovery endpoints use Cache-Control: public, max-age=3600 (intentionally cacheable per RFC 8414/7517/9728):

  • Marked with x-cacheable-endpoint: true vendor extension
  • CATS uses --skipFuzzersForExtension=x-cacheable-endpoint=true:CheckSecurityHeaders

IDOR False Positive Handling:

Filter parameters (like threat_model_id, addon_id) are not IDOR vulnerabilities - they narrow results, not authorize access. The results parser marks these as false positives.

For complete documentation, see docs/migrated/developer/testing/cats-public-endpoints.md.

Output

Results are stored in test/outputs/cats/:

  • cats-results.db - SQLite database of parsed results
  • report/ - Detailed HTML and JSON reports

Analyzing CATS Results

Test results are saved as individual JSON files (Test*.json) in test/outputs/cats/report/. Each result has one of these statuses:

  • error - Test failed with unexpected behavior
  • warn - Test produced warnings
  • success - Test passed

After running CATS, parse results into a SQLite database for analysis:

# Parse results into SQLite
make parse-cats-results

# Query parsed results (excludes false positives)
make query-cats-results

# Full analysis pipeline
make analyze-cats-results

When reviewing CATS results, categorize each finding as:

Category Description Action
Should Fix (High) Security vulnerability Fix immediately
Should Fix (Medium) API contract violation Fix in next sprint
Should Fix (Low) Minor compliance issue Add to backlog
Should Investigate Unclear behavior Review with team
False Positive Expected/correct behavior Mark as ignored
Should Ignore By design or not applicable Document reason

Example Classifications:

  • False Positive: Server returns 200 for GET / without authorization - this endpoint is intentionally public (security: [])
  • Should Investigate: Unexpected Accept-Language header handling - needs design decision
  • Should Fix (Low): Server returns 400 instead of 405 for unsupported HTTP method
  • Should Fix (Medium): Response Content-Type doesn't match OpenAPI schema

AI-Assisted Analysis: For large result sets, use AI agents to analyze test/outputs/cats/report/ files, classifying each error/warning as "should fix", "should ignore", "false positive", or "should investigate" with priority and reasoning.

Example Security Analysis Report

A comprehensive security analysis report from a CATS fuzzing run (November 2025, CATS v13.3.2) is available at docs/migrated/security/cats-fuzzer-analysis.md. This report demonstrates the analysis approach for CATS results, including:

  • Executive summary with test statistics and key findings
  • Prioritized findings categorized by severity (Must Fix, Should Fix, Ignore)
  • False positive identification with RFC references for expected behavior
  • Recommended action plan with time estimates and file locations

The report analyzed 35,680 test results and identified 2 actionable patterns (security headers and error response content types) while documenting 7 false positive categories.

Key Security Headers Identified:

  • Cache-Control: no-store - Prevents caching of sensitive data
  • X-Content-Type-Options: nosniff - Prevents MIME-type sniffing attacks
  • X-Frame-Options: DENY - Prevents clickjacking attacks
  • Content-Security-Policy: frame-ancestors 'none' - Modern clickjacking protection

See OWASP HTTP Headers Cheat Sheet for current header recommendations.

Historical Findings Analysis

Early CATS fuzzing identified several important issues that have been resolved:

  1. CheckDeletedResourcesNotAvailable Bug - DeleteGroupAndData was looking up groups by name instead of internal_uuid, causing incorrect deletions when multiple groups shared the same name. Fixed in auth/repository/deletion_repository.go.

  2. RemoveFields on oneOf Schemas - CATS doesn't fully understand oneOf constraints. When removing fields from CreateAdministratorRequest (which requires exactly one of email, provider_user_id, or group_name), the API correctly returns 400.

  3. XSS on Query Parameters - XSS warnings on GET requests are false positives for JSON APIs. TMI returns application/json, not HTML, so XSS payloads in query parameters are not exploitable. The parse-cats-results.py script now automatically flags these.

For the complete historical analysis and bug traces, see docs/migrated/developer/testing/cats-findings-plan.md.

CATS False Positives

CATS may flag legitimate API responses as "errors" due to expected behavior patterns. These are not security vulnerabilities - they are correct, RFC-compliant responses or expected API behavior.

The scripts/parse-cats-results.py script automatically detects and marks false positives using the is_false_positive() function, which handles 16+ categories:

Category Response Codes Description
OAuth/Auth 401, 403 Expected authentication failures during fuzzing
Rate Limiting 429 Infrastructure protection, not API behavior
Validation 400 API correctly rejects malformed input
Not Found 404 Expected when fuzzing with random/invalid resource IDs
IDOR 200 Filter parameters and admin endpoints behave correctly
HTTP Methods 400, 405 Correct rejection of unsupported methods
Response Contract Various Header mismatches are spec issues, not security issues
Conflict 409 Duplicate name conflicts from fuzzed values
Content Type 400 Go HTTP layer transport errors (text/plain)
Injection Various JSON API data reflection is not XSS
Header Validation 400 Correct rejection of malformed headers
Transfer Encoding 501 Correct rejection per RFC 7230

Using Filtered Results:

The database provides two views:

  • test_results_view - All tests with is_oauth_false_positive flag
  • test_results_filtered_view - Excludes false positives (recommended)
-- Query actual errors (excluding false positives)
SELECT * FROM test_results_filtered_view WHERE result = 'error';

-- View false positives separately
SELECT * FROM test_results_view WHERE is_oauth_false_positive = 1;

Quick Reference:

Scenario Is False Positive? Reason
401 with "invalid_token" Yes Correct OAuth error response
403 with "forbidden" Yes Correct permission denied
409 on POST /admin/groups Yes Duplicate name from fuzzed values
400 from header fuzzers Yes Correct header validation
429 rate limit Yes Infrastructure protection
404 from boundary fuzzers Yes Expected with invalid IDs
500 with "NullPointerException" No Actual server error

For complete details on all false positive categories, see docs/migrated/developer/testing/cats-oauth-false-positives.md.

Detailed False Positive Categories

The following are documented false positives with detailed explanations:

1. PrefixNumbersWithZeroFields (400 Bad Request)

  • CATS sends numeric values as strings with leading zeros (e.g., "0095")
  • JSON numbers with leading zeros are invalid per RFC 8259
  • The API correctly rejects these malformed inputs

2. NoSQL Injection Detection (201 Created)

  • CATS reports "NoSQL injection vulnerability detected" when payloads like { $where: function() { return true; } } are stored
  • TMI uses PostgreSQL, not MongoDB - NoSQL operators have no effect
  • The payload is stored as a literal string, not executed

3. POST /admin/administrators Validation (400 Bad Request)

  • This endpoint uses oneOf schema requiring exactly one of: email, provider_user_id, or group_name
  • CATS generates bodies that may not satisfy the oneOf constraint
  • The API correctly validates and returns 400 for invalid combinations

4. Connection Errors (Response Code 999)

  • HTTP 999 is not a real status code - it indicates connection errors
  • Often occurs with URL encoding issues (e.g., trailing %)
  • This is a CATS/network issue, not an API bug

5. StringFieldsLeftBoundary on Optional Fields (201 Created)

  • CATS sends empty strings for optional fields and expects 4XX
  • Empty strings on optional fields are valid input
  • The API correctly creates resources with empty optional fields

6. GET Filter Parameters Returning Empty Results (200 OK)

  • CATS sends fuzzing values as filter parameters
  • Returning empty results for non-matching filters is standard REST behavior
  • Endpoints: /admin/groups, /admin/users, /admin/administrators

7. XSS on Query Parameters (200 OK)

  • TMI is a JSON API, not an HTML-rendering application
  • XSS requires HTML context to execute - JSON responses don't render HTML
  • Client applications handle output encoding, not the API

8. POST /admin/groups Duplicate Rejection (409 Conflict)

  • CATS's boundary fuzzers may trigger duplicate group creation
  • 409 Conflict is proper REST semantics for duplicate resources

9. POST /admin/administrators User/Group Not Found (404)

  • CATS generates random values for reference fields
  • 404 is correct when referenced users/groups don't exist

CATS Bugs Fixed in 13.6.0

The following fuzzers were previously skipped due to CATS 13.5.0 bugs but are now re-enabled:

  • MassAssignmentFuzzer: Was crashing with JsonPath.InvalidModificationException on array properties (#191)
  • InsertRandomValuesInBodyFuzzer: Was crashing with IllegalArgumentException: count is negative during HTML report generation (#193)

Ensure you're running CATS 13.6.0 or later to avoid these issues.

Template Injection Protection

TMI validates addon name and description fields for template injection patterns (defense-in-depth):

Pattern Description Example
{{ / }} Handlebars, Jinja2, Angular, Go templates {{constructor.constructor('alert(1)')()}}
${ JavaScript template literals, Freemarker ${alert(1)}
<% / %> JSP, ASP, ERB server templates <%=System.getProperty('user.home')%>
#{ Spring EL, JSF EL expressions #{T(java.lang.Runtime).exec('calc')}
${{ GitHub Actions context injection ${{github.event.issue.title}}

NoSQL syntax is allowed since it's harmless in a SQL (PostgreSQL) context.

CATS Remediation Plan

TMI maintains a detailed remediation plan documenting the analysis and resolution of all CATS fuzzing findings. The plan tracks:

  • 24,211 successes (99.4%), 116 errors (0.5%), 39 warnings (0.2%)
  • Issue categories: False positives, OpenAPI spec issues, potential security issues, SAML documentation, input validation edge cases
  • All issues resolved with documented resolutions

Key resolutions include:

  • IDOR on DELETE /addons/{id}: False positive - admin-only endpoint by design (see api/addon_handlers.go line 207)
  • Admin endpoint HappyPath failures: OpenAPI spec updated with oneOf and maximum constraints
  • WebhookQuota schema mismatch: Added created_at/modified_at fields to schema
  • SAML 400 responses: Added to OpenAPI spec for /saml/{provider}/login and /saml/{provider}/metadata

Skipped fuzzers (false positives):

  • DuplicateHeaders - Server correctly ignores duplicate headers per HTTP spec
  • LargeNumberOfRandomAlphanumericHeaders - Server correctly ignores extra headers
  • EnumCaseVariantFields - Server correctly uses case-sensitive enum validation

For complete remediation details including OpenAPI changes and expected metrics, see docs/migrated/developer/testing/cats-remediation-plan.md.

CATS Test Data Setup

CATS fuzzing can report false positives when testing endpoints that require prerequisite objects (e.g., testing GET /threat_models/{id}/threats/{threat_id} fails with 404 when no threat model exists). TMI addresses this by pre-creating a complete object hierarchy before fuzzing.

Object Hierarchy

threat_model (root)
├── threats
│   └── metadata
├── diagrams
│   └── metadata
├── documents
│   └── metadata
├── assets
│   └── metadata
├── notes
│   └── metadata
├── repositories
│   └── metadata
└── metadata

addons (independent root)
└── invocations

webhooks (independent root)
└── deliveries

client_credentials (independent root)

Creating Test Data

Test data is created automatically when running make cats-fuzz:

# Full fuzzing (includes test data creation via cats-seed)
make cats-fuzz

# Or manually create test data
make cats-create-test-data TOKEN=eyJhbGc... SERVER=http://localhost:8080 USER=alice

The scripts/cats-create-test-data.sh script:

  1. Authenticates via OAuth (uses the OAuth callback stub)
  2. Creates one of each object type with stable IDs
  3. Stores IDs in a reference file (test/outputs/cats/cats-test-data.json)
  4. Generates YAML reference data for CATS (test/outputs/cats/cats-test-data.yml)

CATS Reference Data Format

CATS uses the --refData parameter to substitute path parameters with real IDs. The YAML file uses the all: key for global substitution:

# CATS Reference Data - Path-based format for parameter replacement
all:
  id: <threat_model_uuid>
  threat_model_id: <threat_model_uuid>
  threat_id: <threat_uuid>
  diagram_id: <diagram_uuid>
  document_id: <document_uuid>
  asset_id: <asset_uuid>
  note_id: <note_uuid>
  repository_id: <repository_uuid>
  webhook_id: <webhook_uuid>
  addon_id: <addon_uuid>
  client_credential_id: <credential_uuid>
  key: cats-test-key
  # Admin resource identifiers
  group_id: <group_uuid>
  internal_uuid: <user_internal_uuid>

For complete reference data format documentation, see CATS Reference Data File.

Documentation

Coverage Reporting

Server Coverage

The TMI server provides comprehensive test coverage reporting with both unit and integration test coverage.

Quick Start

# Generate full coverage report (unit + integration + merge + reports)
make test-coverage

# Run only unit tests with coverage
make test-coverage-unit

# Run only integration tests with coverage
make test-coverage-integration

# Generate reports from existing profiles
make generate-coverage

Output Files

Coverage Directory (coverage/):

  • unit_coverage.out - Raw unit test coverage data
  • integration_coverage.out - Raw integration test coverage data
  • combined_coverage.out - Merged coverage data
  • unit_coverage_detailed.txt - Detailed unit test coverage by function
  • integration_coverage_detailed.txt - Detailed integration test coverage
  • combined_coverage_detailed.txt - Detailed combined coverage
  • coverage_summary.txt - Executive summary with key metrics

HTML Reports Directory (coverage_html/):

  • unit_coverage.html - Interactive unit test coverage report
  • integration_coverage.html - Interactive integration test coverage report
  • combined_coverage.html - Interactive combined coverage report

View HTML Report:

open coverage_html/combined_coverage.html

Coverage Goals

  • Unit Tests: Target 80%+ coverage for core business logic
  • Integration Tests: Target 70%+ coverage for API endpoints and workflows
  • Combined: Target 85%+ overall coverage

Key Areas of Focus

High priority areas for coverage:

  1. API Handlers - All HTTP endpoints should be tested
  2. Business Logic - Core threat modeling functionality
  3. Authentication & Authorization - Security-critical code
  4. Database Operations - Data persistence and retrieval
  5. Cache Management - Performance-critical caching logic

Prerequisites

  • Go 1.25 or later
  • Docker (for integration tests with PostgreSQL and Redis)
  • gocovmerge tool (automatically installed if missing)

Test Database Configuration

Coverage integration tests use dedicated ports (see config/coverage-report.yml):

  • PostgreSQL: localhost:5434 (container: tmi-coverage-postgres)
  • Redis: localhost:6381 (container: tmi-coverage-redis)

These ports avoid conflicts with development databases (5432, 6379).

Troubleshooting

Docker Not Available:

# Start Docker on macOS
open -a Docker

# Verify Docker is running
docker info

Database Connection Issues:

# Clean up any existing containers
make clean-everything

# Or manually clean up
docker stop tmi-coverage-postgres tmi-coverage-redis 2>/dev/null
docker rm tmi-coverage-postgres tmi-coverage-redis 2>/dev/null

Coverage Tool Missing:

go install github.com/wadey/gocovmerge@latest

Advanced Usage

Custom Coverage Profiles:

# Test specific packages
go test -coverprofile=custom.out ./api/...

# Test with race detection
go test -race -coverprofile=race.out ./...

# Generate HTML from custom profile
go tool cover -html=custom.out -o custom.html

Coverage Analysis:

# Find functions with zero coverage
go tool cover -func=coverage/combined_coverage.out | awk '$3 == "0.0%" {print $1}'

# Show files sorted by coverage
go tool cover -func=coverage/combined_coverage.out | sort -k3 -n

Web App Coverage

# Generate coverage report
pnpm run test:coverage

# View report
open coverage/index.html

Coverage Configuration: vitest.config.ts

export default defineConfig({
  test: {
    coverage: {
      provider: 'v8',
      reporter: ['text', 'html', 'lcov'],
      exclude: [
        'node_modules/',
        'src/**/*.spec.ts',
        'src/environments/'
      ]
    }
  }
});

Testing Best Practices

1. Test Organization

  • One test file per source file
  • Group related tests with describe blocks
  • Use clear, descriptive test names
  • Follow AAA pattern: Arrange, Act, Assert

2. Test Data

  • Use factories for test data
  • Create minimal test data
  • Clean up after tests
  • Use predictable test users (login hints)

3. Isolation

  • Tests should be independent
  • Don't rely on test order
  • Clean up between tests
  • Mock external dependencies

4. Assertions

  • Test one thing per test
  • Use specific assertions
  • Test both happy path and error cases
  • Verify side effects

5. Performance

  • Keep unit tests fast (<1s each)
  • Use before/after hooks efficiently
  • Parallelize tests when possible
  • Cache test fixtures

6. Maintainability

  • DRY - Don't Repeat Yourself
  • Use helper functions
  • Keep tests simple
  • Update tests with code

Continuous Integration

GitHub Actions

Tests run automatically on:

  • Pull requests
  • Pushes to main branch
  • Scheduled nightly builds

Workflow (.github/workflows/test.yml):

name: Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-go@v2
      - run: make test-unit

  integration-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-go@v2
      - run: make test-integration

  api-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - run: npm install -g newman
      - run: make test-api

Troubleshooting Tests

Integration Tests Fail

# Clean everything and retry
make clean-everything
make test-integration

# Check container logs
docker logs tmi-integration-postgres
docker logs tmi-integration-redis

# Verify ports are free
lsof -ti :5434  # PostgreSQL
lsof -ti :6381  # Redis

API Tests Fail

# Check server is running
curl http://localhost:8080/

# Check authentication
curl -H "Authorization: Bearer TOKEN" http://localhost:8080/threat_models

# Run specific collection
newman run postman/comprehensive-test-collection.json

E2E Tests Fail

# Clear Cypress cache
pnpm run test:e2e:clean

# Run in headed mode to see what's happening
pnpm run test:e2e:open

# Check screenshots
ls cypress/screenshots/

# Check videos
ls cypress/videos/

Next Steps

Home

Releases

Getting Started

Deployment

Operation

Troubleshooting

Development

Integrations

Tools

API Reference

Reference

Clone this wiki locally