Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/python_pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,8 @@ jobs:
run: >
poetry run coverage run -m pytest
--verbose
--timeout=300
--session-timeout=3600
-m "not linting and not super_slow and not flaky"

- name: Print Coverage Report
Expand Down
40 changes: 40 additions & 0 deletions docs/timeout_configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Test Timeout Configuration

This document explains PyAirbyte's test timeout configuration to prevent CI timeouts, particularly on Windows.

## Current Configuration

- **Global test timeout**: 600 seconds (10 minutes) per test
- **CI job timeout**: 60 minutes for pytest jobs
- **pytest-timeout plugin**: v2.4.0 installed

Comment on lines +7 to +10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Version mismatch: plugin version in doc vs dependency.

Doc says “pytest-timeout v2.4.0 installed,” but pyproject pins ^2.3.1. Either bump the dep or reword to “pytest-timeout >=2.3.0,” wdyt? Note: --session-timeout is available since 2.3.0. (pypi.org)

🤖 Prompt for AI Agents
In docs/timeout_configuration.md around lines 7 to 10, the documented
pytest-timeout version (v2.4.0) conflicts with pyproject which pins ^2.3.1;
update the doc to match the dependency by either changing the text to
“pytest-timeout >=2.3.0” (or “pytest-timeout v2.3.x”) or bumping the pyproject
dependency to >=2.4.0, and mention that --session-timeout is available since
2.3.0 if you keep the lower bound.

## Timeout Strategy

### Per-Test Timeouts
- **Unit tests**: 60 seconds (fast execution expected)
- **Integration tests**: 180 seconds (3 minutes for data operations)
- **Slow tests**: Use existing 600 second global timeout

### Session Timeouts
- **Full test suite**: 3600 seconds (1 hour maximum)
- **Windows CI**: Limited to unit tests only to prevent timeouts

Comment on lines +18 to +21
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Windows scope is inaccurate.

Doc says “Windows CI: Limited to unit tests,” but the workflow runs the full matrix on Windows. Shall we update this to reflect current behavior, wdyt?

🤖 Prompt for AI Agents
In docs/timeout_configuration.md around lines 18 to 21, the "Windows CI: Limited
to unit tests only" bullet is inaccurate; replace that line to state that
Windows CI runs the full workflow/matrix (not just unit tests) and is subject to
the same session timeout as other platforms (e.g., "Windows CI: Runs full
matrix; subject to the same 3600s session timeout"), and ensure the phrasing
matches surrounding bullets and CI behavior.

## Usage Examples

```bash
# Run tests with custom per-test timeout
pytest --timeout=120 tests/unit_tests/

# Run integration tests with timeout and duration reporting
poetry run poe test-integration-timeout

# Analyze slow test patterns
poetry run poe test-slow-analysis
```

## Slow Test Analysis

The following integration tests are marked as slow and may cause Windows CI timeouts:
- source-faker tests with 200-300 record scales
- Tests parametrized across multiple cache types (DuckDB, Postgres, BigQuery, Snowflake)
- Docker-based tests (slower on Windows due to Docker performance)
4 changes: 4 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,10 @@ airbyte-mcp = "airbyte.mcp.server:main"
[tool.poe.tasks]
test = { shell = "pytest" }
test-fast = { shell = "pytest --durations=5 --exitfirst -m 'not slow'" }
test-unit-tests = { shell = "pytest tests/unit_tests/" }
test-with-short-timeout = { shell = "pytest --timeout=300 --durations=10" }
test-integration-timeout = { shell = "pytest tests/integration_tests/ --timeout=180 --durations=10 -m 'not super_slow'" }
test-slow-analysis = { shell = "pytest tests/integration_tests/ --collect-only --durations=0 -m slow" }

coverage = { shell = "coverage run -m pytest && coverage report" }
coverage-report = { shell = "coverage report" }
Expand Down
51 changes: 51 additions & 0 deletions scripts/analyze_slow_tests.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
# Copyright (c) 2023 Airbyte, Inc., all rights reserved.

"""Script to analyze slow tests in PyAirbyte and recommend timeout configurations.

This script identifies tests marked as slow and provides recommendations for
per-test timeout limits to prevent CI timeouts on Windows.
"""


def main() -> None:
"""Analyze slow tests and provide timeout recommendations."""
print("=== PyAirbyte Slow Test Analysis ===\n")

print("1. Tests marked with @pytest.mark.slow:")
slow_tests = [
"tests/integration_tests/test_all_cache_types.py::test_faker_read "
"(4 parametrized variants)",
"tests/integration_tests/test_all_cache_types.py::test_append_strategy",
"tests/integration_tests/test_all_cache_types.py::test_replace_strategy",
"tests/integration_tests/test_all_cache_types.py::test_merge_strategy",
"tests/integration_tests/test_all_cache_types.py::test_auto_add_columns",
"tests/integration_tests/test_source_faker_integration.py::test_replace_strategy",
"tests/integration_tests/test_source_faker_integration.py::test_append_strategy",
"tests/integration_tests/test_source_faker_integration.py::test_merge_strategy "
"(2 parametrized variants)",
"tests/integration_tests/test_docker_executable.py::test_replace_strategy",
"tests/integration_tests/test_docker_executable.py::test_append_strategy",
"tests/integration_tests/test_docker_executable.py::test_merge_strategy "
"(2 parametrized variants)",
]

for test in slow_tests:
print(f" - {test}")

print(f"\nTotal slow tests identified: {len(slow_tests)}")

print("\n2. Timeout Recommendations:")
print(" - Current global timeout: 600 seconds (10 minutes)")
print(" - Recommended per-test timeout for integration tests: 180 seconds (3 minutes)")
print(" - Recommended per-test timeout for unit tests: 60 seconds (1 minute)")
print(" - Session timeout for entire test suite: 3600 seconds (1 hour)")

print("\n3. CLI Usage Examples:")
print(" poetry run poe test-integration-timeout # Run integration tests with 3min timeout")
print(" poetry run poe test-with-short-timeout # Run all tests with 5min timeout")
print(" pytest --timeout=120 tests/unit_tests/ # Custom timeout for unit tests")


if __name__ == "__main__":
main()
Loading