Add CI to validate challenge_config on PRs and master (fixes #98)#133
Open
NEHAJAKATE wants to merge 2 commits intoCloud-CV:masterfrom
Open
Add CI to validate challenge_config on PRs and master (fixes #98)#133NEHAJAKATE wants to merge 2 commits intoCloud-CV:masterfrom
NEHAJAKATE wants to merge 2 commits intoCloud-CV:masterfrom
Conversation
Author
|
Hi, just checking in to see if any additional changes or tests are needed from my side. Happy to update the PR accordingly. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #98
Problem
EvalAI-Starters currently has no automated validation for challenge_config.yaml/challenge_config.yml on PRs or after merges to master.
This forces maintainers to manually verify changes post-merge, and invalid configs can slip into master.
What this PR adds
GitHub Actions workflow that runs on:
Pull requests targeting master (only when challenge_config.yaml|yml or related challenge packaging files change)
Pushes to master (same path filter)
Lightweight validator script github/ci_validate_challenge_config.py that:
checks YAML syntax
builds evaluation_script.zip
builds challenge_config.zip
optionally validates against the EvalAI API if secrets are provided (EVALAI_AUTH_TOKEN, EVALAI_HOST_TEAM_PK, EVALAI_HOST_URL)
Security / Safety note
This workflow runs on pull_request (not pull_request_target), so it does not receive repository secrets on PRs from forks, which keeps CI safe by default.
Optional EvalAI API validation is only meaningful when secrets are available (typically on trusted contexts like pushes to master or same-repo PRs, depending on repo settings).
Why this approach
Prevents “bad config merged → manual debugging later”
Provides fast feedback in PRs with minimal CI complexity
Avoids exposing secrets to untrusted PR code (GitHub default behavior for forks)
How to test (verified locally)
pip install -r github/requirements.txt
pip install pyyaml
python github/ci_validate_challenge_config.py
echo $?
Expected:
Exit code 0 for valid config
Exit code non-zero if YAML is invalid / packaging fails