Skip to content

Add CI to validate challenge_config on PRs and master (fixes #98)#133

Open
NEHAJAKATE wants to merge 2 commits intoCloud-CV:masterfrom
NEHAJAKATE:fix/98-ci-pipeline
Open

Add CI to validate challenge_config on PRs and master (fixes #98)#133
NEHAJAKATE wants to merge 2 commits intoCloud-CV:masterfrom
NEHAJAKATE:fix/98-ci-pipeline

Conversation

@NEHAJAKATE
Copy link

@NEHAJAKATE NEHAJAKATE commented Jan 20, 2026

Fixes #98

Problem

EvalAI-Starters currently has no automated validation for challenge_config.yaml/challenge_config.yml on PRs or after merges to master.
This forces maintainers to manually verify changes post-merge, and invalid configs can slip into master.

What this PR adds

GitHub Actions workflow that runs on:

Pull requests targeting master (only when challenge_config.yaml|yml or related challenge packaging files change)

Pushes to master (same path filter)

Lightweight validator script github/ci_validate_challenge_config.py that:

checks YAML syntax

builds evaluation_script.zip

builds challenge_config.zip

optionally validates against the EvalAI API if secrets are provided (EVALAI_AUTH_TOKEN, EVALAI_HOST_TEAM_PK, EVALAI_HOST_URL)

Security / Safety note

This workflow runs on pull_request (not pull_request_target), so it does not receive repository secrets on PRs from forks, which keeps CI safe by default.
Optional EvalAI API validation is only meaningful when secrets are available (typically on trusted contexts like pushes to master or same-repo PRs, depending on repo settings).

Why this approach

Prevents “bad config merged → manual debugging later”

Provides fast feedback in PRs with minimal CI complexity

Avoids exposing secrets to untrusted PR code (GitHub default behavior for forks)

How to test (verified locally)
pip install -r github/requirements.txt
pip install pyyaml
python github/ci_validate_challenge_config.py
echo $?

Expected:

Exit code 0 for valid config

Exit code non-zero if YAML is invalid / packaging fails

@NEHAJAKATE
Copy link
Author

Hi, just checking in to see if any additional changes or tests are needed from my side. Happy to update the PR accordingly. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add CI script using Github Actions for each PR

1 participant