fix: prevent jscpd and secretlint report-folder race (#3979)

TextReporter runs per linter the moment it finishes, so JsCpdLinter's
rmtree of megalinter-reports/copy-paste fired while secretlint was still
scanning the workspace in project mode. secretlint had already globbed
the jscpd report into its target set, crashed with ENOENT reading it, and
its stack trace was then parsed as SARIF and counted as a finding.

jscpd now writes its report to a temp folder outside the workspace and
copies it in only when clones are found, so MegaLinter never deletes files
other linters are scanning. secretlint gets a generated ignore file in the
report folder that merges the user's patterns with an exclusion for that
folder alone; narrowing a secrets scanner further would hide credentials
baked into build artifacts.

- fix: report a linter with unparsable SARIF as a warning, not a success
  - A linter exiting 0 while emitting a stack trace instead of SARIF was
    counted as zero findings and shown green; affects linters declaring
    can_output_sarif (REPOSITORY_SECRETLINT, REPOSITORY_SEMGREP,
    REPOSITORY_TRIVY). A linter that dies before producing SARIF now fails
    with its raw output instead of a phantom finding.
- fix(secretlint): honor ignore patterns when REPORT_OUTPUT_FOLDER is off
  - The generated ignore file was only written when reports could be
    written; otherwise it fell back to passing a base name, so a
    .secretlintignore under LINTER_RULES_PATH silently applied nothing. It
    now goes to a temp folder outside the workspace, and the
    --secretlintignore=<value> argument form is parsed too.
- fix(secretlint): stop falling back to .gitignore for a missing ignore file
  - A typo in --secretlintignore used to promote .gitignore patterns to
    authoritative status, excluding the very files a secrets scanner exists
    to inspect. It now degrades with a warning and applies no fallback.
- fix(config): always exclude REPORT_OUTPUT_FOLDER from linted directories
  - An EXCLUDED_DIRECTORIES override could drop it, making MegaLinter lint
    its own output while reporters were still writing to it.
- test: cover jscpd report wiring and isolate ambient env in unit tests
- docs: document the secretlint ignore model and report-folder exclusion

Fixes: #3979

Assisted-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Trevor Vaughan <peiriannydd@gmail.com>

Author: trevor-vaughan

CHANGELOG.md

@@ -47,6 +47,15 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
   - Fix outdated links in `docs/descriptors/repository_kingfisher.md`
   - Fix `LINTER_RULES_PATH` not being used to resolve config files for linters using `active_only_if_file_found` (e.g. `REPOSITORY_LS_LINT`, `SPELL_PROSELINT`, `SPELL_VALE`), fixes [#8416](https://github.com/oxsecurity/megalinter/issues/8416)
   - Honor `EXCLUDED_DIRECTORIES` and `ADDITIONAL_EXCLUDED_DIRECTORIES` in changed-files mode (`VALIDATE_ALL_CODEBASE: false`), so files inside excluded directories are pruned from the `git diff` file list the same way they are during full-codebase validation ([#8360](https://github.com/oxsecurity/megalinter/issues/8360))
+  - Fix sporadic `ENOENT` crashes when `COPYPASTE_JSCPD` and `REPOSITORY_SECRETLINT` run in parallel: jscpd now writes its report to a temporary folder outside the linted workspace, and MegaLinter no longer deletes report files while other linters are still scanning them, fixes [#3979](https://github.com/oxsecurity/megalinter/issues/3979). As jscpd no longer removes its report at the end of a clean run, a copy-paste report left by a previous run is now removed when jscpd starts, so the report folder never mixes results from two runs
+  - Fix `REPOSITORY_SECRETLINT` silently narrowing its scan when `REPOSITORY_SECRETLINT_ARGUMENTS` names a `--secretlintignore` file that does not exist: MegaLinter no longer falls back to `.secretlintignore` or `.gitignore` in its place, as those patterns often exclude the very files a secrets scanner must inspect
+  - Fix the `copy-paste` report folder being created unreadable to non-root users (such as a later artifact-upload step), as it inherited the private permissions of the temporary folder jscpd writes to
+  - Stop `REPOSITORY_SECRETLINT` from scanning MegaLinter's own report folder, which could raise false positives on secrets echoed into other linters' reports, using an ignore file generated inside the report folder so nothing is ever written to or deleted from the linted sources
+  - Always exclude `REPORT_OUTPUT_FOLDER` from linted directories, even when `EXCLUDED_DIRECTORIES` is overridden
+  - Report a linter that crashes before producing SARIF output as a failure showing its raw output, instead of a spurious single finding accompanied by a SARIF parsing error
+  - Stop reporting a linter that exits without parsable SARIF output as a clean success: its results could not be counted, so it is now reported as a warning rather than as zero findings. Applies to linters declaring `can_output_sarif` when SARIF output is enabled, including security scanners such as `REPOSITORY_SECRETLINT`, `REPOSITORY_SEMGREP` and `REPOSITORY_TRIVY`
+  - Fix `REPOSITORY_SECRETLINT` ignoring a `.secretlintignore` located in `LINTER_RULES_PATH` (e.g. the default `.github/linters`): only its base name was passed, so secretlint resolved it from the workspace root and applied either no patterns at all or those of a different file with the same name
+  - Keep applying `REPOSITORY_SECRETLINT` ignore patterns when `REPORT_OUTPUT_FOLDER` is disabled, by generating the merged ignore file in a temporary folder outside the linted sources instead of giving up on it
 
 - Reporters
 

README.md

@@ -1135,7 +1135,7 @@ description: List of common variables that you can use to customize MegaLinter b
 | [**ENABLE_ERRORS_LINTERS**](https://github.com/oxsecurity/megalinter/tree/main/docs/config-activation.md)                 | <!-- -->                                      | List of enabled and blocking linters keys [(more info)](https://github.com/oxsecurity/megalinter/tree/main/docs/config-activation.md)                                                                                                                                                                         |
 | [**ENABLE**](https://github.com/oxsecurity/megalinter/tree/main/docs/config-activation.md)                                | <!-- -->                                      | List of enabled descriptors keys [(more info)](https://github.com/oxsecurity/megalinter/tree/main/docs/config-activation.md)                                                                                                                                                                                  |
 | [**ENABLE_LINTERS**](https://github.com/oxsecurity/megalinter/tree/main/docs/config-activation.md)                        | <!-- -->                                      | List of enabled linters keys [(more info)](https://github.com/oxsecurity/megalinter/tree/main/docs/config-activation.md)                                                                                                                                                                                      |
-| **EXCLUDED_DIRECTORIES**                                                                                                  | \[…many values…\]                             | List of excluded directory basenames. They're excluded at any nested level.                                                                                                                                                                                                                                   |
+| **EXCLUDED_DIRECTORIES**                                                                                                  | \[…many values…\]                             | List of excluded directory basenames. They're excluded at any nested level. The value of **REPORT_OUTPUT_FOLDER** is always excluded in addition to this list.                                                                                                                                                |
 | **EXTENDS**                                                                                                               | <!-- -->                                      | Base `mega-linter.yml` config file(s) to extend local configuration from. Can be a single URL or a list of `.mega-linter.yml` config files URLs. Later files take precedence.                                                                                                                                 |
 | **FAIL_IF_MISSING_LINTER_IN_FLAVOR**                                                                                      | `false`                                       | If set to `true`, MegaLinter fails if a linter is missing in the selected flavor                                                                                                                                                                                                                              |
 | **FAIL_IF_UPDATED_SOURCES**                                                                                               | `false`                                       | If set to `true`, MegaLinter fails if a linter or formatter has autofixed sources, even if there are no errors                                                                                                                                                                                                |
@@ -1265,6 +1265,8 @@ ADDITIONAL_EXCLUDED_DIRECTORIES:
   - vendor
 ```
 
+MegaLinter's own report folder (**REPORT_OUTPUT_FOLDER**, `megalinter-reports` by default) is always excluded from linting, even if you override **EXCLUDED_DIRECTORIES**. Linting MegaLinter's own output produces false positives, and the reporters write to that folder while linters are still running.
+
 <!-- config-filtering-section-end -->
 <!-- config-apply-fixes-section-start -->
 <!-- markdown-headers
@@ -1533,7 +1535,7 @@ Allowing `file` or `list_of_files` to be overridden to `project` is mostly for w
 
 Special considerations:
 
-- Linters that are configured to use the `project` lint mode ignore variables like `FILTER_REGEX_INCLUDE` and `FILTER_REGEX_EXCLUDE`, as they are not passed a list of files to lint. For those linters, you must check their documentation to see if a linter can be configured to ignore specific files. For example, the [Secretlint](https://megalinter.io/latest/descriptors/repository_secretlint/) linter ignores files listed in `~/.secretlintignore` by default, or it can be configured to instead ignore files listed in `~/.gitignore` by setting `REPOSITORY_SECRETLINT_ARGUMENTS` to `--secretlintignore .gitignore.`
+- Linters that are configured to use the `project` lint mode ignore variables like `FILTER_REGEX_INCLUDE` and `FILTER_REGEX_EXCLUDE`, as they are not passed a list of files to lint. For those linters, you must check their documentation to see if a linter can be configured to ignore specific files. For example, the [Secretlint](https://megalinter.io/latest/descriptors/repository_secretlint/) linter ignores files listed in `~/.secretlintignore` by default, or it can be configured to ignore files listed in `~/.gitignore` by setting `REPOSITORY_SECRETLINT_ARGUMENTS` to `--secretlintignore .gitignore.` MegaLinter merges the patterns of that file into a generated ignore file, so the file you name is honored wherever it lives. When the report folder sits inside the linted workspace, it is appended to that generated file so MegaLinter's own reports are never scanned.
 
 <!-- config-cli-lint-mode-section-end -->
 <!-- configuration-section-end -->

mega-linter-runner/lib/megalinter-vars.json

@@ -7840,7 +7840,7 @@
       "name": "EXCLUDED_DIRECTORIES",
       "type": "array",
       "title": "Excluded directories",
-      "description": "List of excluded directory basenames replacing MegaLinter default excluded directories. They are excluded at any nested level.",
+      "description": "List of excluded directory basenames replacing MegaLinter default excluded directories. They are excluded at any nested level. The value of REPORT_OUTPUT_FOLDER is always excluded in addition to this list.",
       "examples": [
         [
           ".github",

megalinter/Linter.py

@@ -63,6 +63,9 @@ def __init__(self, params=None, linter_config=None):
         self.linter_speed = 3
         self.can_output_sarif = False
         self.output_sarif = False
+        # Sticky flag: set when SARIF output could not be parsed, so a linter that
+        # exits 0 with unparsable SARIF is not reported as clean (see get_sarif_result_count)
+        self.sarif_parse_failed = False
         # ex: https://eslint.org/
         self.linter_url = (
             "Field 'linter_url' must be overridden at custom linter class level"
@@ -937,6 +940,18 @@ def run(
             if self.cli_lint_mode == "list_of_files":
                 self.update_files_lint_results(self.files, None, None, None, None, None)
 
+        # A linter that exited without parsable SARIF has not been measured, so
+        # counting zero results would report it as clean. Every reporter still shows
+        # it with zero errors and zero warnings, hence the log: the status alone
+        # cannot tell an operator that the results are missing rather than empty.
+        if self.status == "success" and self.sarif_parse_failed is True:
+            self.status = "warning"
+            logging.warning(
+                f"[{self.linter_name}] results could not be counted, as its SARIF "
+                "output was not parsable: reporting this linter as a warning rather "
+                "than as a clean success"
+            )
+
         # Set return code to 0 if failures in this linter must not make the MegaLinter run fail
         if self.return_code != 0:
             # Disable errors: no failure, just warning
@@ -1667,8 +1682,20 @@ def get_sarif_result_count(self, stdout: str, level: str):
                     sarif_output = yaml.safe_load(sarif_file)
                     # SARIF is in stdout
             else:
-                # SARIF is in stdout
-                sarif_output = yaml.safe_load(stdout)
+                # SARIF is in stdout. Require it to actually have SARIF shape:
+                # find_json_in_stdout returns "" unless the payload parses as JSON
+                # containing a "runs" key. A linter that crashed before producing
+                # SARIF leaves a stack trace here, which must surface as a failure
+                # rather than be parsed as results
+                sarif_stdout = utils.find_json_in_stdout(stdout)
+                if sarif_stdout == "":
+                    logging.error(
+                        f"[{self.linter_name}] exited without producing parsable SARIF "
+                        f"output while counting {level}s.\nLinter output: {stdout}"
+                    )
+                    self.sarif_parse_failed = True
+                    return 0
+                sarif_output = json.loads(sarif_stdout)
 
             for run in sarif_output["runs"]:
                 rule_default_level_map = {}
@@ -1711,14 +1738,15 @@ def get_sarif_result_count(self, stdout: str, level: str):
 
             return total_result
         except Exception as e:
-            total_result = 1
+            # Return 0 rather than inventing a finding: get_total_number_errors
+            # already reports 1 error for any non-success linter, so a broken linter
+            # still fails the run without being attributed a phantom result
             logging.error(
-                f"Error while getting total {level}s from SARIF output.\nError:"
-                + str(e)
-                + "\nstdout: "
-                + stdout
+                f"[{self.linter_name}] unable to compute total {level}s from SARIF "
+                f"output.\nError: {str(e)}\nLinter output: {stdout}"
             )
-            return total_result
+            self.sarif_parse_failed = True
+            return 0
 
     # Build the CLI command to get linter version (can be overridden if --version is not the way to get the version)
     def build_version_command(self):

megalinter/descriptors/repository.megalinter-descriptor.yml

@@ -890,6 +890,15 @@ linters:
       - **Team Collaboration**: Shared configuration files ensure consistent secret detection across development teams
       - **Performance Optimized**: Fast scanning suitable for large repositories and automated workflows
       - **Open Source**: Community-driven with transparent detection rules and continuous updates
+
+      **Ignore patterns in MegaLinter:**
+
+      MegaLinter does not pass your ignore file to secretlint directly. It generates a `.secretlintignore-megalinter` file inside **REPORT_OUTPUT_FOLDER** and passes that instead, because secretlint applies only one ignore file.
+
+      - The generated file merges the patterns of your own ignore file: the one named by `REPOSITORY_SECRETLINT_ARGUMENTS: --secretlintignore <file>` if you set it, otherwise your `.secretlintignore`, otherwise your `.gitignore`. Your file is honored wherever it lives, including in **LINTER_RULES_PATH**.
+      - MegaLinter's own report folder is appended last, so it is always excluded. Ignore rules are last match wins, so this cannot be undone by a negation pattern in your own file.
+      - Nothing else is added. MegaLinter's **EXCLUDED_DIRECTORIES** are deliberately not merged in: build artifacts such as `.terraform` or `.terragrunt-cache` can contain real credentials baked in at generation time, and a secrets scanner must still see them.
+      - When **REPORT_OUTPUT_FOLDER** is disabled, the generated file is written to a temporary folder outside your sources instead, so your patterns still apply.
     linter_url: https://github.com/secretlint/secretlint
     linter_repo: https://github.com/secretlint/secretlint
     linter_banner_image_url: https://github.com/secretlint/secretlint/raw/master/docs/assets/SecretLintLP.png
@@ -909,12 +918,9 @@ linters:
       - project
     cli_lint_extra_args_after:
       # secretlint v13 introduced a ripgrep-based file walker that respects
-      # .gitignore by default. --no-gitignore keeps the v12 scan breadth (scan
-      # everything, filter only via .secretlintignore). The .secretlintignore
-      # itself (which excludes .automation/test fixtures containing intentional
-      # fake secrets) is applied via SecretLintLinter.get_ignore_arguments,
-      # which passes it by base name because the v13 walker matches ignore
-      # files by base name and drops an absolute path.
+      # .gitignore by default. --no-gitignore keeps the v12 scan breadth: scan
+      # everything, filter only via the ignore file MegaLinter generates in
+      # SecretLintLinter.build_megalinter_ignore_file.
       # See https://github.com/secretlint/secretlint/releases (v13)
       - "--no-gitignore"
     cli_lint_mode_project_extra_args_after:

megalinter/descriptors/schemas/megalinter-configuration.jsonschema.json

@@ -9261,7 +9261,7 @@
     },
     "EXCLUDED_DIRECTORIES": {
       "$id": "#/properties/EXCLUDED_DIRECTORIES",
-      "description": "List of excluded directory basenames replacing MegaLinter default excluded directories. They are excluded at any nested level.",
+      "description": "List of excluded directory basenames replacing MegaLinter default excluded directories. They are excluded at any nested level. The value of REPORT_OUTPUT_FOLDER is always excluded in addition to this list.",
       "examples": [
         [
           ".github",