Implement comprehensive automated testing infrastructure and version management for all repository scripts#17
Conversation
…sitory scripts Co-authored-by: DutchmanNL <7318445+DutchmanNL@users.noreply.github.com>
|
@copilot you forgot (again) to update the change log and readme according to requirements:
ensure this is handled correctly and, if possible/need, update @copilot instruction to avoid this error's in feature PR's (missing update in readme/changelog) |
…re details, enhance PR guidelines Co-authored-by: DutchmanNL <7318445+DutchmanNL@users.noreply.github.com>
You're absolutely right - I missed updating the CHANGELOG.md and README.md as required by the repository guidelines. I've now addressed this with commit ab49038 which: CHANGELOG.md Updates:
README.md Updates:
Enhanced PR Guidelines:
This ensures future PRs will properly document changes in both files as required by the repository standards. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
DutchmanNL
left a comment
DutchmanNL
left a comment
There was a problem hiding this comment.
@copilot , implementing this PR will update the version to 0.4.0, handling accordingly and use the automation/scripting for this purpose and validate all reference are updated correctly
Title for this release: Automated Version handling & validation by GitHub actions
…le from CI workflow Co-authored-by: DutchmanNL <7318445+DutchmanNL@users.noreply.github.com>
Updated the version to 0.4.0 using the automated version management scripts ( Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
|
@copilot testing is failing: https://github.com/DrozmotiX/ioBroker-Copilot-Instructions/actions/runs/17840762631/job/50729644734?pr=17#step:1:40 update framework accordingly and test your changes |
…tions to v4 Co-authored-by: DutchmanNL <7318445+DutchmanNL@users.noreply.github.com>
Fixed the CI testing failures with commit b31c8e1. The issues were:
The workflow now properly sets execute permissions for all test files ( Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
There was a problem hiding this comment.
Pull Request Overview
This PR implements a comprehensive automated testing infrastructure for all repository scripts and introduces version management automation. The primary purpose is to ensure script reliability and prevent regressions through systematic testing of all automation tools used in the ioBroker Copilot Instructions repository.
Key changes:
- Created complete test framework with 54 tests covering all scripts and integration scenarios
- Added GitHub Actions CI/CD workflow for automated testing on repository changes
- Updated repository version to 0.4.0 with comprehensive documentation updates
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test-runner.sh | Main test execution framework providing isolated environments and comprehensive reporting |
| tests/test-update-versions.sh | 11 tests for documentation version synchronization script functionality |
| tests/test-manage-versions.sh | 13 tests for master version management script commands and validation |
| tests/test-integration.sh | 10 integration tests for cross-script workflows and end-to-end scenarios |
| tests/test-extract-version.sh | 9 tests for version and date extraction utility functions |
| tests/test-check-template-version.sh | 11 tests for template version comparison and remote checking features |
| TESTING.md | Comprehensive documentation for the automated testing infrastructure and usage guidelines |
| .github/workflows/test-scripts.yml | GitHub Actions workflow for continuous integration testing |
| template.md | Version bump from 0.3.1 to 0.4.0 |
| README.md | Added testing framework documentation and updated version references |
| CHANGELOG.md | Detailed changelog entry for automated testing infrastructure (Fixes #16) |
| .github/copilot-instructions.md | Updated with mandatory testing requirements and guidelines |
|
|
||
| # Test actual version replacement in README | ||
| test_version_replacement() { | ||
| local original_version=$(grep "Latest Version:" "$TEST_DIR/README.md" | head -1 | sed 's/.*Latest Version:\*\* v*//' | tr -d ' ') |
There was a problem hiding this comment.
The regex pattern for extracting version numbers is duplicated and complex. Consider creating a helper function to extract versions from README.md to improve maintainability and reduce the risk of inconsistent patterns.
| "$TEST_DIR/scripts/update-versions.sh" >/dev/null 2>&1 | ||
|
|
||
| # Check if README was updated | ||
| local updated_version=$(grep "Latest Version:" "$TEST_DIR/README.md" | head -1 | sed 's/.*Latest Version:\*\* v*//' | tr -d ' ') |
There was a problem hiding this comment.
The regex pattern for extracting version numbers is duplicated and complex. Consider creating a helper function to extract versions from README.md to improve maintainability and reduce the risk of inconsistent patterns.
| # Test full version update workflow | ||
| test_version_update_workflow() { | ||
| cd "$TEST_DIR" | ||
|
|
||
| # Store original versions | ||
| local original_template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | ||
|
|
||
| # Update to a test version | ||
| ./scripts/manage-versions.sh update 1.2.3 >/dev/null 2>&1 | ||
|
|
||
| # Check if all files were updated | ||
| local template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | ||
| local readme_version=$(grep "Latest Version:" README.md | head -1 | sed 's/.*Latest Version:\*\* v*//' | tr -d ' ') | ||
| local copilot_version=$(grep "^**Version:**" .github/copilot-instructions.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') |
There was a problem hiding this comment.
Multiple similar grep/sed patterns for version extraction are repeated throughout the test. Consider creating a shared helper function for version extraction to reduce code duplication and improve maintainability.
| # Test full version update workflow | |
| test_version_update_workflow() { | |
| cd "$TEST_DIR" | |
| # Store original versions | |
| local original_template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | |
| # Update to a test version | |
| ./scripts/manage-versions.sh update 1.2.3 >/dev/null 2>&1 | |
| # Check if all files were updated | |
| local template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | |
| local readme_version=$(grep "Latest Version:" README.md | head -1 | sed 's/.*Latest Version:\*\* v*//' | tr -d ' ') | |
| local copilot_version=$(grep "^**Version:**" .github/copilot-instructions.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | |
| # Helper function to extract version from a file using a grep pattern | |
| extract_version_from_file() { | |
| local file="$1" | |
| local pattern="$2" | |
| grep "$pattern" "$file" | head -1 | sed 's/.*\*\* *v*\?//' | tr -d ' ' | |
| } | |
| # Test full version update workflow | |
| test_version_update_workflow() { | |
| cd "$TEST_DIR" | |
| # Store original versions | |
| local original_template_version=$(extract_version_from_file "template.md" "^**Version:**") | |
| # Update to a test version | |
| ./scripts/manage-versions.sh update 1.2.3 >/dev/null 2>&1 | |
| # Check if all files were updated | |
| local template_version=$(extract_version_from_file "template.md" "^**Version:**") | |
| local readme_version=$(extract_version_from_file "README.md" "Latest Version:") | |
| local copilot_version=$(extract_version_from_file ".github/copilot-instructions.md" "^**Version:**") |
| ./scripts/manage-versions.sh update 1.2.3 >/dev/null 2>&1 | ||
|
|
||
| # Check if all files were updated | ||
| local template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') |
There was a problem hiding this comment.
Multiple similar grep/sed patterns for version extraction are repeated throughout the test. Consider creating a shared helper function for version extraction to reduce code duplication and improve maintainability.
| # Test full version update workflow | ||
| test_version_update_workflow() { | ||
| cd "$TEST_DIR" | ||
|
|
||
| # Store original versions | ||
| local original_template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | ||
|
|
||
| # Update to a test version | ||
| ./scripts/manage-versions.sh update 1.2.3 >/dev/null 2>&1 | ||
|
|
||
| # Check if all files were updated | ||
| local template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | ||
| local readme_version=$(grep "Latest Version:" README.md | head -1 | sed 's/.*Latest Version:\*\* v*//' | tr -d ' ') | ||
| local copilot_version=$(grep "^**Version:**" .github/copilot-instructions.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') |
There was a problem hiding this comment.
Multiple similar grep/sed patterns for version extraction are repeated throughout the test. Consider creating a shared helper function for version extraction to reduce code duplication and improve maintainability.
| # Test full version update workflow | |
| test_version_update_workflow() { | |
| cd "$TEST_DIR" | |
| # Store original versions | |
| local original_template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | |
| # Update to a test version | |
| ./scripts/manage-versions.sh update 1.2.3 >/dev/null 2>&1 | |
| # Check if all files were updated | |
| local template_version=$(grep "^**Version:**" template.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | |
| local readme_version=$(grep "Latest Version:" README.md | head -1 | sed 's/.*Latest Version:\*\* v*//' | tr -d ' ') | |
| local copilot_version=$(grep "^**Version:**" .github/copilot-instructions.md | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ') | |
| # Helper function to extract version from a file | |
| # Usage: extract_version_from_file <file> <type> | |
| # type: "template" | "readme" | "copilot" | |
| extract_version_from_file() { | |
| local file="$1" | |
| local type="$2" | |
| case "$type" in | |
| template|copilot) | |
| grep "^**Version:**" "$file" | head -1 | sed 's/.*Version:\*\* *//' | tr -d ' ' | |
| ;; | |
| readme) | |
| grep "Latest Version:" "$file" | head -1 | sed 's/.*Latest Version:\*\* v*//' | tr -d ' ' | |
| ;; | |
| *) | |
| echo "Unknown type: $type" >&2 | |
| return 1 | |
| ;; | |
| esac | |
| } | |
| # Test full version update workflow | |
| test_version_update_workflow() { | |
| cd "$TEST_DIR" | |
| # Store original versions | |
| local original_template_version=$(extract_version_from_file template.md template) | |
| # Update to a test version | |
| ./scripts/manage-versions.sh update 1.2.3 >/dev/null 2>&1 | |
| # Check if all files were updated | |
| local template_version=$(extract_version_from_file template.md template) | |
| local readme_version=$(extract_version_from_file README.md readme) | |
| local copilot_version=$(extract_version_from_file .github/copilot-instructions.md copilot) |
| cp -r "$REPO_ROOT"/* "$TEST_DIR/" 2>/dev/null || true | ||
|
|
||
| # Ensure hidden directories are copied (like .github) | ||
| cp -r "$REPO_ROOT"/.* "$TEST_DIR/" 2>/dev/null || true |
There was a problem hiding this comment.
The copy operations suppress all errors with || true, which could mask legitimate permission or disk space issues. Consider checking for specific expected errors (like 'cannot copy . and ..') while still reporting unexpected failures.
| cp -r "$REPO_ROOT"/* "$TEST_DIR/" 2>/dev/null || true | |
| # Ensure hidden directories are copied (like .github) | |
| cp -r "$REPO_ROOT"/.* "$TEST_DIR/" 2>/dev/null || true | |
| # Copy non-hidden files/directories | |
| if ! cp_output=$(cp -r "$REPO_ROOT"/* "$TEST_DIR/" 2>&1); then | |
| echo -e "${RED}Error copying files to test directory:${NC} $cp_output" | |
| exit 1 | |
| fi | |
| # Ensure hidden directories are copied (like .github) | |
| if ! cp_hidden_output=$(cp -r "$REPO_ROOT"/.* "$TEST_DIR/" 2>&1); then | |
| # Ignore expected errors about . and .., but fail on others | |
| if ! echo "$cp_hidden_output" | grep -qE 'cannot copy .*: No such file or directory|cannot copy .:.*|cannot copy ..:.*'; then | |
| echo -e "${RED}Error copying hidden files to test directory:${NC} $cp_hidden_output" | |
| exit 1 | |
| fi | |
| fi |
| cp -r "$REPO_ROOT"/* "$TEST_DIR/" 2>/dev/null || true | ||
|
|
||
| # Ensure hidden directories are copied (like .github) | ||
| cp -r "$REPO_ROOT"/.* "$TEST_DIR/" 2>/dev/null || true |
There was a problem hiding this comment.
The copy operations suppress all errors with || true, which could mask legitimate permission or disk space issues. Consider checking for specific expected errors (like 'cannot copy . and ..') while still reporting unexpected failures.
| cp -r "$REPO_ROOT"/* "$TEST_DIR/" 2>/dev/null || true | |
| # Ensure hidden directories are copied (like .github) | |
| cp -r "$REPO_ROOT"/.* "$TEST_DIR/" 2>/dev/null || true | |
| # Copy non-hidden files/directories | |
| if ! cp -r "$REPO_ROOT"/* "$TEST_DIR/" 2>err.log; then | |
| if grep -vqE "No such file or directory|cannot stat" err.log; then | |
| echo -e "${RED}Error copying files to test directory:${NC}" | |
| cat err.log | |
| rm -f err.log | |
| exit 1 | |
| fi | |
| rm -f err.log | |
| fi | |
| # Ensure hidden directories are copied (like .github) | |
| if ! cp -r "$REPO_ROOT"/.* "$TEST_DIR/" 2>err.log; then | |
| # Only suppress errors about . and .., which are expected | |
| if grep -vqE "cannot copy .*: No such file or directory|cannot copy .:.*|cannot copy ..:.*" err.log; then | |
| echo -e "${RED}Error copying hidden files to test directory:${NC}" | |
| cat err.log | |
| rm -f err.log | |
| exit 1 | |
| fi | |
| rm -f err.log | |
| fi |
scripts/manage-versions.sh- Master version management with show/check/sync/update commandsscripts/extract-version.sh- Extract version and date information from templatescripts/update-versions.sh- Update documentation with dynamic versionsscripts/check-template-version.sh- Check template version against remotetests/test-runner.sh- Main test execution framework (54 total tests)tests/test-extract-version.sh- 9 tests for version extraction functionalitytests/test-manage-versions.sh- 13 tests for version management commandstests/test-update-versions.sh- 11 tests for documentation updatingtests/test-check-template-version.sh- 11 tests for template version checkingtests/test-integration.sh- 10 integration tests for script interactions.github/workflows/test-scripts.yml- Runs on push, PRs, manual trigger (removed daily schedule)TESTING.md- Comprehensive testing infrastructure documentation.github/copilot-instructions.mdwith automated testing requirementsAll 54 tests pass successfully, providing comprehensive coverage of script functionality, error handling, and integration scenarios.
Fixes #16.
💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.