Merge branch 'feat/use_manifest_filters_with_gitignore' into 'main'

feat: Use manifest filters with gitignore file

Closes PACMAN-1089

See merge request espressif/idf-component-manager!494

Author: Evgeny Torbin

docs/en/guides/packaging_components.rst

@@ -171,46 +171,56 @@ In CI/CD pipelines, using environment variables to log in is more convenient. Yo
 Filter Component Files
 ======================
 
-As a component developer, you may want to specify which files from the component directory will be uploaded to the ESP Component Registry. There are two ways to achieve this: either by `using a .gitignore file`_ or by `using manifest filters`_.
+As a component developer, you may want to specify which files from the component directory will be uploaded to the ESP Component Registry. This can be achieved by using `manifest filters`_ and a `.gitignore file`_.
 
-.. warning::
+Manifest Filters
+----------------
+
+Your ``idf_component.yml`` manifest may have ``files`` section with ``include`` and ``exclude`` filters. For example:
+
+.. code:: yaml
+
+   files:
+      exclude:
+         - "*.py"          # Exclude all Python files
+         - "**/*.list"     # Exclude `.list` files in all directories
+         - "big_dir/**/*"  # Exclude `big_dir` directory and its content
+      include:
+         - "**/.DS_Store"  # Include files excluded by default
 
-   You are not allowed to use both methods simultaneously.
+Files and directories that are excluded by default can be found `here <https://github.com/espressif/idf-component-manager/blob/main/idf_component_tools/file_tools.py#L16>`_.
 
-Using a .gitignore File
------------------------
+.gitignore File
+---------------
 
-First, you need to specify the ``use_gitignore`` option in the ``idf_component.yml`` manifest file.
+If you have a ``.gitignore`` file in your component directory, you can use it to filter files. All you need to do, is to specify the ``use_gitignore`` option in the ``idf_component.yml`` manifest file.
 
 .. code:: yaml
 
    files:
       use_gitignore: true
 
-Then, patterns specified in the ``.gitignore`` file will be automatically excluded before packaging or uploading the component.
+Patterns specified in the ``.gitignore`` file will be automatically excluded before packaging or uploading the component.
 
 .. code:: yaml
 
    test_dir/   # Exclude files in all `test_dir` directories (including the directories themselves)
 
-More information on how ``.gitignore`` works can be found in the `official documentation <https://git-scm.com/docs/gitignore/en>`_.
+More information on how ``.gitignore`` works can be found in the `official documentation <https://git-scm.com/docs/gitignore>`_.
 
-Using Manifest Filters
-----------------------
-
-In this case, your ``idf_component.yml`` manifest may have ``include`` and ``exclude`` filters. For example:
+You can also use both manifest filters and a ``.gitignore`` file. In this case, the patterns from the ``.gitignore`` file will be applied first. Example:
 
 .. code:: yaml
 
    files:
+      use_gitignore: true
       exclude:
-         - "*.py"          # Exclude all Python files
-         - "**/*.list"     # Exclude `.list` files in all directories
-         - "big_dir/**/*"  # Exclude `big_dir` directory and its content
+         - ".env"          # Exclude `.env` file
       include:
-         - "**/.DS_Store"  # Include files excluded by default
+         - "test_dir/**/*" # Include all files in `test_dir` directory
+                           # which were excluded by `.gitignore`
 
-Files and directories that are excluded by default can be found `here <https://github.com/espressif/idf-component-manager/blob/main/idf_component_tools/file_tools.py#L16>`_.
+When using ``.gitignore``, files specified `here <https://github.com/espressif/idf-component-manager/blob/main/idf_component_tools/file_tools.py#L16>`_ will not be excluded by default.
 
 .. warning::
 

docs/en/reference/manifest_file.rst

@@ -141,32 +141,53 @@ A dictionary with the following options:
 -  ``include``: List of patterns to include.
 -  ``exclude``: List of patterns to exclude.
 
-.. note::
-
-   You cannot use ``.gitignore`` and ``include``/``exclude`` rules at the same time.
-
 Examples:
 
 #. Use ``.gitignore`` to exclude files:
 
-.. code:: yaml
+   .. code:: yaml
 
-   files:
-      use_gitignore: true
+      files:
+         use_gitignore: true
 
-2. Use ``include`` and ``exclude`` patterns:
+#. Use ``include`` and ``exclude`` patterns:
 
-.. code:: yaml
+   .. code:: yaml
+
+      files:
+         exclude:
+            - "*.py"          # Exclude all Python files
+            - "**/*.list"     # Exclude `.list` files in all directories
+            - "big_dir/**/*"  # Exclude `big_dir` directory and its content
+         include:
+            - "**/.DS_Store"  # Include files excluded by default
+
+#. Use both. Consider the following ``.gitignore`` file:
+
+   .. code:: text
 
-   files:
-      exclude:
-         - "*.py"          # Exclude all Python files
-         - "**/*.list"     # Exclude `.list` files in all directories
-         - "big_dir/**/*"  # Exclude `big_dir` directory and its content
-      include:
-         - "**/.DS_Store"  # Include files excluded by default
+      test_dir/
 
-..
+   Then manifest ``idf_component.yml`` might look like:
+
+   .. code:: yaml
+
+      files:
+         use_gitignore: true
+         exclude:
+            - ".env"          # Exclude `.env` file
+         include:
+            - "test_dir/**/*" # Include all files in `test_dir` directory
+                              # which were excluded by `.gitignore`
+
+Filters are applied in the following order:
+
+#. All files are included.
+#. If ``use_gitignore`` is set to ``true``, files are excluded using the ``.gitignore`` file. Otherwise the default exclusion list is used.
+#. If ``exclude`` field is set, files are excluded using the patterns in the ``exclude`` field.
+#. If ``include`` field is set, files are included using the patterns in the ``include`` field, even if they were excluded in the previous steps.
+
+Note that the ``include`` field could be used to override files in ``exclude`` field, ``.gitignore`` file and default exclusion list.
 
 This field is optional and can be omitted if the component contains all files in the root directory with the default list of exceptions.
 
@@ -181,18 +202,6 @@ A list of files and directories excluded by default:
 
 |DEFAULT_EXCLUDE|
 
-Here is an example on how to use the ``files`` field. Note that the ``include`` field could be used to override the default exclusion list.
-
-.. code:: yaml
-
-   files:
-     exclude:
-       - "*.py" # Exclude all Python files
-       - "**/*.list" # Exclude `.list` files in all directories
-       - "big_dir/**/*" # Exclude `big_dir` directory and its content
-     include:
-       - "**/.DS_Store" # Include files excluded by default
-
 .. _manifest-examples:
 
 ``examples``

idf_component_tools/file_tools.py

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: 2022-2024 Espressif Systems (Shanghai) CO LTD
+# SPDX-FileCopyrightText: 2022-2025 Espressif Systems (Shanghai) CO LTD
 # SPDX-License-Identifier: Apache-2.0
 """Set of tools and constants to work with files and directories"""
 
@@ -119,29 +119,25 @@ def exclude_all_directories():
     include_paths('**/*')
 
     if use_gitignore:
-        # Exclude user patterns
-        for pattern in exclude:
-            exclude_paths(pattern)
-
         # Exclude .gitignore patterns
         exclude_gitignore = gitignore_ignored_files(base_path)
         paths.difference_update(exclude_gitignore)
         exclude_all_directories()
     else:
-        # Exclude all defaults, including directories
+        # Exclude defaults
         if exclude_default:
             for pattern in DEFAULT_EXCLUDE:
                 exclude_paths(pattern)
 
-        # Exclude user patterns
-        for pattern in exclude:
-            exclude_paths(pattern)
+    # Exclude manifest patterns
+    for pattern in exclude:
+        exclude_paths(pattern)
 
-        exclude_all_directories()
+    exclude_all_directories()
 
-        # Include everything that was explicitly added
-        for pattern in include:
-            include_paths(pattern)
+    # Include manifest patterns
+    for pattern in include:
+        include_paths(pattern)
 
     return paths
 

idf_component_tools/manifest/models.py

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: 2024 Espressif Systems (Shanghai) CO LTD
+# SPDX-FileCopyrightText: 2024-2025 Espressif Systems (Shanghai) CO LTD
 # SPDX-License-Identifier: Apache-2.0
 import typing as t
 import warnings
@@ -355,20 +355,6 @@ class FilesField(BaseModel):
     include: UniqueStrListField = []
     exclude: UniqueStrListField = []
 
-    @model_validator(mode='before')  # type: ignore
-    @classmethod
-    def validate_files(cls, data: t.Any) -> t.Any:
-        if not isinstance(data, dict):
-            return data
-
-        if 'use_gitignore' in data and data['use_gitignore']:
-            if 'exclude' in data or 'include' in data:
-                raise ValueError(
-                    'Invalid field "files": Cannot use ".gitignore" and "exclude"/"include" at the same time'
-                )
-
-        return data
-
 
 class RepositoryInfoField(BaseModel):
     commit_sha: Annotated[str, Field(pattern=COMMIT_ID_RE)] = None  # type: ignore

tests/manifest/test_validator.py

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: 2023-2024 Espressif Systems (Shanghai) CO LTD
+# SPDX-FileCopyrightText: 2023-2025 Espressif Systems (Shanghai) CO LTD
 # SPDX-License-Identifier: Apache-2.0
 import logging
 import os
@@ -172,19 +172,6 @@ def test_validate_files_invalid_list(self, valid_manifest):
 
         assert errors == ['Invalid field "files:include": Input should be a valid list']
 
-    def test_validate_files_invalid_keys_combination(self, valid_manifest):
-        valid_manifest['files']['use_gitignore'] = True
-        errors = Manifest.validate_manifest(valid_manifest)
-
-        assert errors == [
-            'Invalid field "files": Cannot use ".gitignore" and "exclude"/"include" at the same time'
-        ]
-
-    def test_validate_files_valid_keys_combination(self, valid_manifest):
-        valid_manifest['files']['use_gitignore'] = False
-        errors = Manifest.validate_manifest(valid_manifest)
-        assert not errors
-
     @pytest.mark.parametrize(
         'invalid_tag',
         [

tests/test_file_tools.py

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: 2022-2024 Espressif Systems (Shanghai) CO LTD
+# SPDX-FileCopyrightText: 2022-2025 Espressif Systems (Shanghai) CO LTD
 # SPDX-License-Identifier: Apache-2.0
 import logging
 import os
@@ -225,6 +225,27 @@ def test_no_default_exclude_with_gitignore(assets_path):
     }
 
 
+def test_include_files_excluded_by_gitignore(assets_path):
+    create_gitignore(assets_path, ['examples', '1.txt'])
+    examples_dir = assets_path / 'examples'
+    examples_dir.mkdir()
+    (examples_dir / 'test.txt').write_text('1')
+    test_example = examples_dir / 'test_example'
+    test_example.mkdir()
+    (test_example / 'test.txt').write_text('1')
+
+    assert filtered_paths(assets_path, use_gitignore=True, include=['examples/**/*', '1.txt']) == {
+        assets_path / 'examples' / 'test_example' / 'test.txt',
+        assets_path / 'examples' / 'test_example',
+        assets_path / 'examples' / 'test.txt',
+        assets_path / 'ignore.dir' / 'file.txt',
+        assets_path / 'ignore.me',
+        assets_path / '1.txt',
+        assets_path / '.gitignore',
+        assets_path / '.gitlab-ci.yml',
+    }
+
+
 def test_check_suspicious_component_files(release_component_path, tmp_path, caplog):
     sub = str(tmp_path / 'sub')
     shutil.copytree(release_component_path, sub)