diff --git a/.github/workflows/TestAll_and_Code_Coverage_CI.md b/.github/workflows/TestAll_and_Code_Coverage_CI.md deleted file mode 100644 index 43895b6a41..0000000000 --- a/.github/workflows/TestAll_and_Code_Coverage_CI.md +++ /dev/null @@ -1,217 +0,0 @@ - -# TestAll and Code Coverage CI - GitHub Actions Workflow - -This repository contains the **Code Coverage CI** GitHub Actions workflow, which automates the process of running MATLAB tests and generating code coverage reports using MOcov and Codecov. The workflow is triggered on pushes and pull requests to the `develop` branch. - -## Table of Contents - -- [Overview](#overview) -- [Prerequisites](#prerequisites) -- [Setup Instructions](#setup-instructions) -- [Workflow Breakdown](#workflow-breakdown) - - [Trigger Configuration](#trigger-configuration) - - [Jobs Definition](#jobs-definition) - - [Steps Breakdown](#steps-breakdown) -- [Project Structure](#project-structure) - - -## Overview - -The **Code Coverage CI** workflow automates testing and code coverage reporting. It ensures that code changes are tested and that code coverage metrics are updated automatically using [MOcov](https://github.com/MOxUnit/MOcov) and [Codecov](https://codecov.io/). - -## Prerequisites - -- **MATLAB R2022a**: The workflow uses MATLAB R2022a. Ensure your code is compatible with this version. -- **MOcov**: An open-source code coverage tool for MATLAB. It will be installed as part of the workflow. -- **Codecov Account**: Sign up for a [Codecov](https://codecov.io/) account to access code coverage reports. -- **Codecov Token**: Obtain a `CODECOV_TOKEN` from Codecov for authentication (required for private repositories). - -## Setup Instructions - -1. **Clone the Repository** (if not already): - - ```bash - git clone https://github.com/opencobra/cobratoolbox.git - ``` - -2. **Add the Workflow File**: - - - Save the provided workflow YAML file as `.github/workflows/codecov.yml` in the repository. - -3. **Store Codecov Token**: - - - In the repository, go to **Settings** > **Secrets and variables** > **Actions**. - - Click on **New repository secret**. - - Add `CODECOV_TOKEN` as the name and paste the Codecov token as the value. - -4. **Add MOcov to Your Project** (Optional for local testing): - - - **Clone MOcov**: - - ```bash - git clone https://github.com/MOxUnit/MOcov.git /path/to/MOcov - ``` - - - **Add MOcov to MATLAB Path**: - - ```matlab - addpath(genpath('/path/to/MOcov')); - ``` - -5. **Create `run_coverage_tests.m` Script**: - - - Add the `run_coverage_tests.m` script to the root of the repository. This script runs tests and generates the coverage report using MOcov. - -6. **Organize Project**: - - - **Source Code**: Place your MATLAB source code in the `src` directory. - - **Tests**: Place test files in the `test` directory. Ensure `testAll.m` or your main test script is in this directory. - -7. **Push Changes**: - - ```bash - git add . - git commit -m "Add Code Coverage CI workflow with MOcov" - git push origin develop - ``` - -## Workflow Breakdown - -### Trigger Configuration - -The workflow is triggered on push events and pull requests to the `develop` branch. - -```yaml -on: - push: - branches: [develop] - pull_request: - branches: [develop] -``` - -### Jobs Definition - -The workflow defines a job named `test` that runs on the latest Ubuntu environment. - -```yaml -jobs: - test: - runs-on: ubuntu-latest -``` - -### Steps Breakdown - -1. **Checkout Repository** - - Checks out thr repository code so the workflow can access it. - - ```yaml - - uses: actions/checkout@v3 - ``` - -2. **Set Up MATLAB** - - Installs MATLAB R2022a on the runner to execute MATLAB scripts and functions. - - ```yaml - - name: Set up MATLAB - uses: matlab-actions/setup-matlab@v1 - with: - release: R2022a - ``` - -3. **Install MOcov** - - Clones the MOcov repository and sets the appropriate permissions. - - ```yaml - - name: Install MOcov - run: | - git clone https://github.com/MOxUnit/MOcov.git /opt/MOcov - sudo chmod -R 755 /opt/MOcov - ``` - -4. **Run MATLAB Tests and Generate Coverage** - - Executes MATLAB tests using MOcov to generate a code coverage report in XML format. - - ```yaml - - name: Run MATLAB tests and generate coverage - run: | - matlab -batch "addpath(genpath(pwd)); run_coverage_tests" - ``` - - **Contents of `run_coverage_tests.m`:** - - ```matlab - function run_coverage_tests() - % Add MOcov to path - addpath(genpath('/opt/MOcov')); - - try - % Run tests and capture results - disp('Running tests with coverage...'); - - % Run tests directly first to capture results - results = runtests('test/testAll.m'); - passed = all([results.Passed]); - - % Now run MOcov with a simpler expression that just runs the tests - test_expression = 'runtests(''test/testAll.m'')'; - - % Run MOcov for coverage analysis - mocov('-cover', '.', ... - '-cover_xml_file', 'coverage.xml', ... - '-expression', test_expression); - - % Check results - if ~passed - error('Some tests failed. Check the test results for details.'); - end - - disp('All tests passed successfully!'); - disp(['Number of passed tests: ' num2str(sum([results.Passed]))]); - - exit(0); - catch e - disp('Error running tests:'); - disp(getReport(e)); - exit(1); - end - end - ``` - -5. **Upload Coverage to Codecov** - - Uploads the generated coverage report to Codecov for analysis. - - ```yaml - - name: Upload coverage to Codecov - uses: codecov/codecov-action@v3 - with: - files: ./coverage.xml - flags: matlab - token: ${{ secrets.CODECOV_TOKEN }} - fail_ci_if_error: true - comment: true - ``` - - - -## Project Structure - -Organize your project as follows: - -``` -your-repository/ -├── .github/ -│ └── workflows/ -│ └── codecov.yml -├── src/ -│ └── (MATLAB source code) -├── test/ -│ └── testAll.m -├── run_coverage_tests.m -└── (Other project files) -``` - diff --git a/.github/workflows/cobratoolboxCI_step1.yml b/.github/workflows/testAllCI_step1.yml similarity index 100% rename from .github/workflows/cobratoolboxCI_step1.yml rename to .github/workflows/testAllCI_step1.yml diff --git a/.github/workflows/cobratoolboxCI_step2.yml b/.github/workflows/testAllCI_step2.yml similarity index 89% rename from .github/workflows/cobratoolboxCI_step2.yml rename to .github/workflows/testAllCI_step2.yml index e825f66892..a5f8b753da 100644 --- a/.github/workflows/cobratoolboxCI_step2.yml +++ b/.github/workflows/testAllCI_step2.yml @@ -36,11 +36,11 @@ jobs: echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV - name: Publish Test Report - uses: ctrf-io/github-test-reporter@v1.0.5 + uses: ctrf-io/github-test-reporter@v1.0.6 with: report-path: 'artifacts/ctrf-report.json' - summary-report: true - failed-report: true + community-report: true + community-report-name: 'cobra-report' issue: ${{ env.PR_NUMBER }} env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/testAll_Continuous_Integration.md b/.github/workflows/testAll_Continuous_Integration.md new file mode 100644 index 0000000000..ed1b2f3aee --- /dev/null +++ b/.github/workflows/testAll_Continuous_Integration.md @@ -0,0 +1,157 @@ +# 🚀 Continuous Integration and Test Reporting for Cobra Toolbox + +## 📌 Overview + +This repository implements a GitHub Actions workflow to automate testing and reporting for pull requests. The setup consists of two workflows: + +1. **`testAllCI_step1`** - Runs MATLAB tests and uploads the test results as artifacts. +2. **`testAllCI_step2`** - Retrieves test results and comments on the corresponding pull request. + +This design ensures security while allowing test reports to be posted on pull requests, including those from forked repositories. + +--- + +## ⚠️ Important Note + +These workflows should be implemented on the **default branch** of the repository (either `master` or `main` in newer repositories) to ensure proper execution and integration. Running workflows on other branches may lead to unexpected behavior, security issues, or failure to post comments on pull requests. + +--- + +## 🔐 Handling Forked Repositories: Why Two Workflows? + +When a pull request originates from a fork, the `pull_request` event runs in the context of the fork, meaning it does not have permission to write to the base repository. This prevents the workflow from posting comments on the pull request. + +Using `pull_request_target` instead of `pull_request` would allow commenting on forked pull requests, but it introduces a significant security risk: the workflow would run with write permissions on the base repository, allowing potential malicious code execution. + +To mitigate this, we split the workflow into two: + +- **The first workflow (`testAllCI_step1`)** only has read permissions and runs the tests. +- **The second workflow (`testAllCI_step2`)** is triggered by the first workflow’s completion and runs in the base repository’s context, allowing it to post a comment securely. + +--- + +## 🔄 Step-by-Step Workflow Execution + +### **1️⃣ testAllCI_step1: Running Tests and Uploading Artifacts** + +This workflow is triggered when a pull request is opened, synchronized, or reopened on the `develop` or `master` branches. It performs the following steps: + +- **Check out merged PR code**: + +```yaml +- name: Check out merged PR code + uses: actions/checkout@v4 +``` + +- **Run MATLAB Tests**: + +```yaml +- name: Run MATLAB Tests + run: | + matlab -batch "run('initCobraToolbox.m'); run('test/testAll.m');" +``` + +- **Convert JUnit to CTRF format**: + +```yaml +- name: Convert JUnit to CTRF + run: | + npx junit-to-ctrf ./testReport.junit.xml -o ./ctrf/ctrf-report.json +``` + +- **Upload CTRF Artifact**: + +```yaml +- name: Upload CTRF Artifact + uses: actions/upload-artifact@v4 + with: + name: testReport + path: ./ctrf/ctrf-report.json +``` + +- **Save PR Number and Upload as an Artifact**: + To ensure that `testAllCI_step2` can correctly comment on the corresponding pull request, we save the PR number as an artifact in `testAllCI_step1`. Since `testAllCI_step2` is triggered by `testAllCI_step1` using `workflow_run`, it does not have direct access to the PR metadata. Uploading the PR number as an artifact allows `testAllCI_step2` to retrieve and use it for posting test results in the correct pull request. + + +```yaml +- name: Save PR Number + run: echo "PR_NUMBER=${{ github.event.pull_request.number }}" >> $GITHUB_ENV + +- name: Upload PR Number as Artifact + run: echo $PR_NUMBER > pr_number.txt + shell: bash + +- name: Upload PR Number Artifact + uses: actions/upload-artifact@v4 + with: + name: pr_number + path: pr_number.txt +``` + +Since this workflow only requires read permissions, it avoids potential security risks when dealing with external contributions from forked repositories. + +--- + +### **2️⃣ testAllCI_step2: Downloading Artifacts and Posting Results** + +This workflow is triggered when `testAllCI_step1` completes successfully. It follows these steps: + +- **Download Test Report Artifact**: +Since GitHub Actions does not allow direct artifact downloads across workflows using `actions/download-artifact`, we use `dawidd6/action-download-artifact@v8` instead. This repository enables downloading artifacts from a previous workflow run by specifying the `run_id`, which is essential when handling artifacts between separate workflows. It follows these steps: +```yaml +- name: Download CTRF Artifact + uses: dawidd6/action-download-artifact@v8 + with: + name: testReport + run_id: ${{ github.event.workflow_run.id }} + path: artifacts +``` + +- **Download PR Number Artifact**: + +```yaml +- name: Download PR Number Artifact + uses: dawidd6/action-download-artifact@v8 + with: + name: pr_number + run_id: ${{ github.event.workflow_run.id }} + path: pr_number +``` + +- **Read PR Number**: + +```yaml +- name: Read PR Number + id: read_pr_number + run: | + PR_NUMBER=$(cat pr_number/pr_number.txt) + echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV +``` + +- **Publish Test Report**: + +The `cobra-report` format is exclusively designed for COBRA Toolbox by COBRA developers and contributed to the `ctrf-io` repository. + +```yaml +- name: Publish Test Report + uses: ctrf-io/github-test-reporter@v1.0.6 + with: + report-path: 'artifacts/ctrf-report.json' + community-report: true + community-report-name: 'cobra-report' + issue: ${{ env.PR_NUMBER }} + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} +``` + +--- + +## ✅ Conclusion + +By structuring the workflows this way, we achieve the following: + +- **Secure execution** without exposing repository write access to forked pull requests. +- **Successful test execution** and result upload. +- **Seamless commenting** on pull requests with test results while mitigating security risks. + +This approach balances **security** and **functionality**, making it a robust solution for continuous integration in repositories that accept contributions from forks. 🚀 diff --git a/src/base/install/prepareTest.m b/src/base/install/prepareTest.m index 0a6706737b..2edcb25060 100644 --- a/src/base/install/prepareTest.m +++ b/src/base/install/prepareTest.m @@ -289,7 +289,7 @@ % Generalized Software check for i = 1:length(requiredSoftwares) software = requiredSoftwares{i}; - + % Depending on the OS, different system commands may be needed % For Linux/Mac, you can generally use `which` or `command -v` % For Windows, `where` can be used to check if software is available @@ -297,7 +297,7 @@ if ispc % Windows [status, ~] = system(['where ' software]); else % Unix-based (Linux/Mac) - [status, ~] = system(['command -v ' software ' > /dev/null']); + [status, ~] = system(['which ' software ' > /dev/null 2>&1']); end if status ~= 0 @@ -305,6 +305,7 @@ end end + % append the error message if ~isempty(missingTBs.License) errorMessage{end + 1} = sprintf('The test requires licenses for the following Toolboxes: %s', strjoin(missingTBs.License, ' and ')); diff --git a/test/testAll.m b/test/testAll.m index 2c93fafc95..b4e8255ed2 100644 --- a/test/testAll.m +++ b/test/testAll.m @@ -208,11 +208,11 @@ totalTime = sum(resultTable.Time); % 1) Wrap in -- typical JUnit format - fprintf(fid, '\n', ... + fprintf(fid, '\n', ... numTests, numFailures, numErrors, totalTime); % 2) A single inside - fprintf(fid, ' \n', ... + fprintf(fid, ' \n', ... numTests, numFailures, numErrors, numSkipped, totalTime); % 3) Loop over each test case diff --git a/test/verifiedTests/analysis/testMultiSpeciesModelling/testCreatePanModels.m b/test/verifiedTests/analysis/testMultiSpeciesModelling/testCreatePanModels.m index acfb26559a..4553ced87d 100644 --- a/test/verifiedTests/analysis/testMultiSpeciesModelling/testCreatePanModels.m +++ b/test/verifiedTests/analysis/testMultiSpeciesModelling/testCreatePanModels.m @@ -14,6 +14,8 @@ fileDir = fileparts(which('testCreatePanModels')); cd(fileDir); +solverPkgs = prepareTest('toolboxes', {'distrib_computing_toolbox'}); + modelList={ 'Abiotrophia_defectiva_ATCC_49176' 'Acidaminococcus_fermentans_DSM_20731' diff --git a/test/verifiedTests/analysis/testPrint/testPrintConstraints.m b/test/verifiedTests/analysis/testPrint/testPrintConstraints.m index eb7cee7373..9998f7504e 100644 --- a/test/verifiedTests/analysis/testPrint/testPrintConstraints.m +++ b/test/verifiedTests/analysis/testPrint/testPrintConstraints.m @@ -29,6 +29,10 @@ diary off text1 = readMixedData('refData_printConstraints.txt'); text2 = readMixedData('printConstraints.txt'); + +% Remove formatting difference +text1 = regexprep(text1, '|', ''); % Remove HTML tags +text2 = regexprep(text2, '|', ''); % Just in case assert(isequal(text1, text2)); % remove the generated file diff --git a/test/verifiedTests/analysis/testSampling/testSampleCbModelRHMC.m b/test/verifiedTests/analysis/testSampling/testSampleCbModelRHMC.m index 1193e6c687..9c14c3e7a9 100644 --- a/test/verifiedTests/analysis/testSampling/testSampleCbModelRHMC.m +++ b/test/verifiedTests/analysis/testSampling/testSampleCbModelRHMC.m @@ -5,12 +5,14 @@ % % initialize the test +solverPkgs = prepareTest('requiredToolboxes', {'statistics_toolbox'}); + fileDir = fileparts(which('testSampleCbModelRHMC')); models = {}; models{1} = getDistributedModel('ecoli_core_model.mat'); models{2} = getDistributedModel('Acidaminococcus_sp_D21.mat'); -% load('Recon1.0model.mat', 'Recon1') -% models{3} = Recon1; +load('Recon1.0model.mat', 'Recon1') +models{3} = Recon1; try S = load('iDopaNeuro1.mat'); names = fieldnames(S); diff --git a/test/verifiedTests/analysis/testThermo/testCycleFreeFlux.m b/test/verifiedTests/analysis/testThermo/testCycleFreeFlux.m index 0106b4aed7..d4de7cfc1b 100644 --- a/test/verifiedTests/analysis/testThermo/testCycleFreeFlux.m +++ b/test/verifiedTests/analysis/testThermo/testCycleFreeFlux.m @@ -16,7 +16,7 @@ % linprog does not seem to work properly on this problem... % quadMinos and dqqMinos seem to have problems with this rproblem too, % leading to suboptimal solutions. -solverPkgs = prepareTest('needsLP', true, 'excludeSolvers',{'matlab','dqqMinos','quadMinos','pdco'}); +solverPkgs = prepareTest('needsLP', true, 'needsQP', true, 'excludeSolvers',{'matlab','dqqMinos','quadMinos','pdco', 'ibm_cplex'}); % save the current path currentDir = pwd; diff --git a/test/verifiedTests/analysis/testTopology/testLrsInterface.m b/test/verifiedTests/analysis/testTopology/testLrsInterface.m index cda0567ac7..e545735137 100644 --- a/test/verifiedTests/analysis/testTopology/testLrsInterface.m +++ b/test/verifiedTests/analysis/testTopology/testLrsInterface.m @@ -13,6 +13,8 @@ % save the current path currentDir = pwd; +solverPkgs = prepareTest('requiredSoftwares',{'lrs'}); + % set the model name modelName = 'test'; diff --git a/test/verifiedTests/analysis/testrFBA/testdynamicRFBA.m b/test/verifiedTests/analysis/testrFBA/testdynamicRFBA.m index 87f3e08e35..79cc1443ef 100644 --- a/test/verifiedTests/analysis/testrFBA/testdynamicRFBA.m +++ b/test/verifiedTests/analysis/testrFBA/testdynamicRFBA.m @@ -22,6 +22,9 @@ %QP solvers QPsolverPkgs = {'tomlab_cplex','ibm_cplex'}; +solverPkgs = prepareTest('requiredSolvers',solverPkgs); + + for k =1:length(solverPkgs) solverLPOK = changeCobraSolver(solverPkgs{k},'LP', 0); diff --git a/test/verifiedTests/base/testSolvers/testChangeIBMCplexParams.m b/test/verifiedTests/base/testSolvers/testChangeIBMCplexParams.m index 8fdf2d0812..c489e2d39e 100644 --- a/test/verifiedTests/base/testSolvers/testChangeIBMCplexParams.m +++ b/test/verifiedTests/base/testSolvers/testChangeIBMCplexParams.m @@ -8,6 +8,8 @@ % save the current path currentDir = pwd; +solverPkgs = prepareTest('requiredSolvers',{'ibm_cplex'}); + %get the warning settings and turn on all warnings cwarn = warning; warning('on');