feat: Add ACT CLI pytest wrapper for testing resolve-ci-vars action

[aaronsteers]

feat: Add ACT CLI pytest wrapper for testing resolve-ci-vars action

Summary

This PR migrates and adapts the ACT CLI pytest wrapper from PyAirbyte to create a comprehensive Python testing framework specifically for the resolve-ci-vars GitHub Action. The implementation uses the nektos/act CLI tool to run GitHub Actions locally with pytest parameterized tests, enabling thorough testing of the action across different trigger scenarios, input types, and edge cases.

Key Components:

• tests/act_runner.py: Python wrapper class for ACT CLI with action-specific testing methods
• tests/test_resolve_ci_vars_action.py: Comprehensive test scenarios covering static inputs, Jinja2 expressions, and trigger types
• tests/conftest.py: Pytest fixtures and configuration for ACT/Docker availability checks
• .github/workflows/test-with-pytest-act.yml: CI workflow for automated testing
• pytest.ini and requirements.txt: Project configuration and dependencies

The framework tests 20 different scenarios including static input resolution, Jinja2 expression evaluation, various GitHub event triggers (push, PR, workflow_dispatch), and edge cases. All tests currently pass locally in dry-run mode.

Review & Testing Checklist for Human

Critical (High Risk) - 5 items:

• Test actual action execution locally: Run pytest with dry_run=False to verify the action actually executes and produces expected outputs (current tests only validate structure)
• Verify CI workflow functionality: Ensure the GitHub Actions workflow can successfully install ACT CLI medium image (~500MB) and run tests without timeout issues
• Validate action input formatting: Manually verify that action inputs (static_inputs, jinja_inputs) are properly converted to INPUT_* environment variables and parsed correctly by the action
• Test real trigger scenarios: Execute tests with actual GitHub event payloads to ensure the resolve-ci-vars action processes them correctly and produces expected output variables
• Cross-reference with existing tests: Compare test scenarios against the existing bash-based tests in test-action.yml to ensure coverage alignment and no functional gaps

Recommended Local Test Plan:

1. Install requirements: pip install -r requirements.txt
2. Run local tests: pytest tests/ -v
3. Test with real execution: Modify a test to use dry_run=False and verify action outputs
4. Test CI workflow: Push to a test branch and monitor workflow execution

---

Diagram

%%{ init : { "theme" : "default" }}%%
graph TD
    subgraph Legend
        L1["Major Edit"]:::major-edit
        L2["Minor Edit"]:::minor-edit  
        L3["Context/No Edit"]:::context
    end

    Action["action.yml<br/>resolve-ci-vars action"]:::context
    TestWorkflow[".github/workflows/<br/>test-action.yml<br/>existing bash tests"]:::context
    
    ActRunner["tests/act_runner.py<br/>ACT CLI wrapper"]:::major-edit
    TestFile["tests/test_resolve_ci_vars_action.py<br/>pytest test scenarios"]:::major-edit
    Conftest["tests/conftest.py<br/>pytest config"]:::major-edit
    
    PytestIni["pytest.ini"]:::minor-edit
    Requirements["requirements.txt"]:::minor-edit
    CIWorkflow[".github/workflows/<br/>test-with-pytest-act.yml"]:::major-edit
    
    ActRunner -->|"invokes via ACT CLI"| Action
    TestFile -->|"uses wrapper"| ActRunner
    TestFile -->|"tests action via"| TestWorkflow
    CIWorkflow -->|"runs"| TestFile
    Conftest -->|"configures"| TestFile

    classDef major-edit fill:#90EE90
    classDef minor-edit fill:#87CEEB    
    classDef context fill:#FFFFFF

Loading

Notes

• Session Info: Requested by AJ Steers (@aaronsteers) in Devin session: https://app.devin.ai/sessions/b7d8bee893184625994b7ff2401addf0
• ACT CLI Image: Uses medium-sized Docker image (~500MB) as discussed - good balance between functionality and disk usage
• Testing Philosophy: Prioritizes local testing first (all 20 tests pass locally) before CI integration, as requested
• Modular Design: Framework is designed to be spun out as a dedicated Python package for ACT CLI testing scenarios
• Complementary Approach: Intended to complement, not replace, existing bash-based tests in test-action.yml

Potential Concerns:

• All current tests use dry_run=True - real execution testing needed
• ACT CLI dependency adds complexity to CI environment setup
• Event payload generation might be incomplete compared to real GitHub events
• Action input environment variable formatting assumptions need validation

[devin-ai-integration]

Original prompt from AJ Steers

@Devin - Let's investigate testing GitHub actions workflows using the act CLI. I've tested it locally and it works great. Want you to locate the Aaron Steers resolve CI vars GitHub action, and combine ACT with py test parameterized tests. We want py test tests that invoke the workflows using a number of different simulated triggers and a clean Python test interface that can be used in the tests to trigger the workflow in a number of ways.

Keep your implementation modular so that we can spin this out as a dedicated Python package once it's working well. The Python package that will eventually create is going to be a thin wrapper around act, specifically with a focus around testing scenarios. 

Before you start make sure nobody already has a Python package wrapping this tool.

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[aaronsteers]

Delete the test class and use simple test functions