[WIP] feat: Filter `copr_build` jobs based on `paths` and files changed #2780

LecrisUT · 2025-04-16T13:39:46Z

TODO:

Skip the check if /packit command is used
Actually check the files changed
Write new tests or update the old ones to cover new functionality.
Update or write new documentation in packit/packit.dev.

Depends-on: packit/ogr#921

Fixes packit/packit#1997
Fixes #2006

RELEASE NOTES BEGIN

copr_build jobs are triggered only if there are changed files under the paths field

RELEASE NOTES END

packit_service/worker/checker/copr.py

softwarefactory-project-zuul · 2025-04-16T13:47:26Z

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/d8df73178934488f99f72ae42d7d9818

❌ pre-commit FAILURE in 1m 48s
❌ packit-service-tests FAILURE in 3m 31s

softwarefactory-project-zuul · 2025-04-16T13:56:22Z

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/ccd7ecf77cb44d22b2ccb47800d4a1e9

✔️ pre-commit SUCCESS in 1m 52s
❌ packit-service-tests FAILURE in 3m 22s

softwarefactory-project-zuul · 2025-05-23T09:12:33Z

This change depends on a change that failed to merge.

Change packit/ogr#921 is needed.

packit_service/worker/checker/copr.py

LecrisUT · 2025-05-23T09:15:07Z

packit_service/worker/checker/copr.py

+        for changed_file in self.get_files_changed():
+            # Check if any of the files changed are under the paths that are being tracked
+            if any(changed_file.is_relative_to(p) for p in paths):
+                return True
+        return False


Actually, what do we do if we have an empty commit, like the Fedora rebuild commits? Should it be a special case to skip this check?

like the Fedora rebuild commits?

those wouldn't be happening in upstream, so I don't think we have to think about that, or am I missing something?

I mean if they want to have that behaviour. I occasionally do that ¹, but if it's documented that it is or it is not supported, than that's fine as well.

Footnotes

https://github.com/LecrisUT/FedoraRPM-atuin/commit/99555ccec9f6ed94fe474767e3b88f18aa835387 ↩

packit_service/events/github/push.py

softwarefactory-project-zuul · 2025-05-28T09:24:05Z

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/3189a66c4a39486e8f902ac669741fa4

✔️ pre-commit SUCCESS in 1m 54s
❌ packit-service-tests FAILURE in 3m 37s

lbarcziova · 2025-05-28T12:08:55Z

I briefly looked into the code and I am thinking if it wouldn't be more fitting to have the logic for checking this in this method, which would mean it would work generally, not just for Copr build jobs.

LecrisUT · 2025-05-28T13:36:17Z

I briefly looked into the code and I am thinking if it wouldn't be more fitting to have the logic for checking this in this method, which would mean it would work generally, not just for Copr build jobs.

Probably, but it's not as extendable as with the Checker, needing another place to be aware of that check are being done. I see that there already are checks like pr_labels_match_configuration. What about extending the Checker design (or rather reimplement a similar/simpler class) or at least aggregate the checks in a list of lambdas?

lbarcziova · 2025-05-30T08:54:30Z

@LecrisUT can you elaborate more on what you are suggesting? I agree the code I linked could be refactored, but it is something different than the Checker's logic, as that is tight to the handlers, while the checks I linked are meant for actually getting the relevant job configurations to the particular event.

LecrisUT · 2025-05-30T09:21:33Z

@LecrisUT can you elaborate more on what you are suggesting? I agree the code I linked could be refactored, but it is something different than the Checker's logic, as that is tight to the handlers, while the checks I linked are meant for actually getting the relevant job configurations to the particular event.

Indeed, not using the same Checker, but mainly not having all the checks implemented in the function, i.e. something like

    def get_jobs_matching_event(self) -> list[JobConfig]:
        jobs_matching_trigger = []
        checkers = JobChecker.get_all_checkers()
        for job in self.event.packages_config.get_job_views():
            if all(c.check(job) for c in checkers):
                jobs_matching_trigger.append(job)

The Checker design that I like is being able to just create a class and boom a new check is introduced, but I would design it a bit simpler, like

class JobChecker:
    all_checkers: Final[ClassVar[list[type[JobChecker]]]] = []

    def __init_subclass__(cls):
        # Register all checkers when the class is created
        cls.all_checkers.append(cls)

    @classmethod
    def get_all_checkers(cls) -> list[JobChecker]:
        return  [checker_cls() for checker_cls in cls.all_checkers]

    def check(job: JobView) -> bool: ...

(Of course with a better interface, constructor, etc.)

lbarcziova · 2025-05-30T12:08:19Z

@LecrisUT understood, thanks for the explanation! Yes, that sounds like it could improve the readability and extensibility.

lachmanfrantisek · 2025-07-22T11:51:49Z

Hi @LecrisUT , thanks for all the work so far. Do you think it's real to finish this? (Asking for the Copr team that would hugely benefit from this..) Anything we can help with?

LecrisUT · 2025-07-22T15:34:55Z

I can try to make some progress this week.

One question to be answered is #2780 (comment). There are a few ways to approach this:

Use ogr and navigate to the PR and get the files changed from there
Use the webhook metadata directly
Teach ogr to give compare patches ¹ and use the before-after that are already in the webhook payload

I am inclined to go with the 3rd option, just because it will come in handy for the future.

https://github.com/PyGithub/PyGithub/blob/2d4785dd08374147a8eab866eeb704508558336a/github/Repository.py#L1342 ↩

lachmanfrantisek · 2025-07-23T10:43:30Z

Nice, thanks @LecrisUT !

I don't have a strong opinion but using webhook might avoid extra API calls (compared to OGR approach). But on the other hand, not sure if this would be same for other git forges. (With OGR, we might be able to find a way more easily.)

lbarcziova · 2025-07-23T14:14:11Z

My suggestion would be to try to use webhooks directly as it seems the most straightforward (we have also already been having issues with API rate limits).

LecrisUT · 2025-07-29T14:18:29Z

Where do I get the push event? Otherwise, it seems we still need the ogr changes for the pull-request trigger, the webhook does not have equivalent data.

softwarefactory-project-zuul · 2025-07-29T14:24:35Z

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/fce7f73ec1cb408481c1aaf0f4056d20

✔️ pre-commit SUCCESS in 1m 56s
❌ packit-service-tests FAILURE in 3m 26s

betulependule

Nice work. It looks good to me so far. Do you think you'll be able to make progress with this soon?

betulependule · 2025-10-17T08:09:57Z

Where do I get the push event? Otherwise, it seems we still need the ogr changes for the pull-request trigger, the webhook does not have equivalent data.

Could you please elaborate what you mean?

betulependule · 2025-10-17T08:11:03Z

packit_service/worker/checker/copr.py

+        # FIXME: This is probably unnecessary
+        package_config = self.package_config.get_package_config_for(self.job_config)


Why do you think this might be unnecessary?

I was thinking about the usage of the method get_package_config_for. In some other parts I see that the method is not used and I assume it would already be processed when self.package_config is used.

Hm, it seems that the get_package_config_for is used to get the config from a config for multiple packages (usually named packages_config) based on a specific job config. In other existing code, I only see packages_config.get_package_config_for being used, but never package_config.get_package_config_for. I think using self.package_config directly might be the right way to do it, but I'm not exactly sure. Tests should hopefully reveal what it right.

Indeed, worst case this is just a no-op, but if anyone with more experience could chyme in and confirm it quickly then we can resolve this one way or the other. Let's keep this open for a bit.

LecrisUT · 2025-10-17T08:40:27Z

Nice work. It looks good to me so far. Do you think you'll be able to make progress with this soon?

Yes, it's just one blocker to figure out and adding the relevant tests

Where do I get the push event? Otherwise, it seems we still need the ogr changes for the pull-request trigger, the webhook does not have equivalent data.

Could you please elaborate what you mean?

Sure, when I was looking at this, I couldn't find the appropriate mixin that can be used with Checker so that we have the Commit class that would now include the list of commits that is added in f57d857. We have GetPagurePullRequestMixin but I couldn't find the equivalent that would do for upstream PRs.

betulependule · 2025-10-17T09:15:09Z

Sure, when I was looking at this, I couldn't find the appropriate mixin that can be used with Checker so that we have the Commit class that would now include the list of commits that is added in f57d857. We have GetPagurePullRequestMixin but I couldn't find the equivalent that would do for upstream PRs.

Yes, you're right that there are no equivalents for upstream PRs (as far as I can tell). I suppose we'll need a new Mixin class for this, then? @lbarcziova, what do you think? I'm still rather new to this project so I don't understand this part too much just yet 😅.

betulependule · 2025-11-12T08:51:21Z

packit_service/worker/checker/copr.py

+        elif self.job_config.trigger == JobConfigTriggerType.commit:
+            # TODO: Get changed files from webhook push
+            raise NotImplementedError()


This should probably be simple to implement. Both GitLab and GitHub webhook push events specify the commits field, where each item contains the added, modified and removed fields describing what files have been changed.

https://docs.github.com/en/webhooks/webhook-events-and-payloads#push
https://docs.gitlab.com/user/project/integrations/webhook_events/#push-events

However, I am a bit worried about this note from GitHub's docs: A maximum of 3000 changed files will be reported per commit. I am concerned that if a user's commit includes more than 3000 changed files, then the webhook will not report all and the checker might ignore some file changes, resulting in an obscure bug... Maybe there should be a check whether the amount of changes matches 3000, because if it does, then that means that the webhook may have under-reported what files were changes and we should find the rest of changed files using a different method.

Or for simplicity, if 3000 changes are reported in one of the fields added, modified or removed, return the root directory ./ to make sure the checker doesn't ignore any files not reported by the webhook push event.

That comment is the same issue as #2780 (comment). I will try to join you on figuring out how to implement this. If you have any ideas from diving in the code, do let me know

However, I am a bit worried about this note from GitHub's docs: A maximum of 3000 changed files will be reported per commit.

My gut instinct is if it ends up with that many change files, we might be having other issues on the side. If they do that big of a change, then might as well either trigger the build for everything or let the users manually trigger it and we just report " 👀 > 3000 files being changed!? Help me 🙏 "

Oh, you're referring to this?

Where do I get the push event?

I wasn't sure what you meant by the question back then. Sorry about that. I suppose you don't know how to access the push object containing the webhook push event info? I can try to find out how to access it. 👀

Yep, any hints on that is appreciated, since it also kinda relates to #2860.

Alright, I'll take a look. 👍 Thanks for your work so far.

Ok, I think I figured it out a bit. It should come from ConfigFromEventMixin and it would have either commit, or PR or equivalent event results. Commit I think it is pretty straightforward, but PRs would be tricky since it can also take in comments. Maybe we could ignore all non-push, non-pr-update events (i.e. make the checker always true then). Updated the PR with a quick sketch that I have so far in case you get some ideas around this as well :)

ConfigFromEventMixin looks quite promising. Good work. 👍

Maybe we could ignore all non-push, non-pr-update events (i.e. make the checker always true then).

Yes, I think that makes sense.

I looked into this as well and in case ConfigFromEventMixin doesn't work, we might still have the event data available in the Checker class.

packit-service/packit_service/worker/jobs.py

Lines 197 to 203 in 3ce7c70

parser = nested_get(

Parser.MAPPING,

source,

event_type,

default=Parser.parse_event,

)

event_object: Optional[Event] = parser(event)

Above is where the event (GitLab and GitHub push events, PR events and other) are parsed. SteveJobs returns a list of TaskResult, which still has the event variable storing what we need (it contains the parsed event, which would contain the property commits of type CommitInfo).

I believe that the event variable is eventually used to initialize the Checker class, which the class AreFilesChanged inherits from. The property self.data is initialized with some properties of the event variable and the commits variable is lost.

packit-service/packit_service/worker/checker/abstract.py

Line 30 in 3ce7c70

self.data = EventData.from_event_dict(event)

If needed, we could store the entire event variable inside the Checker class (self.event = event) to make it available in AreFilesChanged.

Using ConfigFromEventMixin seems like a cleaner solution, so let's try that first. 👍

betulependule · 2025-11-12T09:17:26Z

packit_service/worker/checker/copr.py

+        if self.job_config.trigger == JobConfigTriggerType.pull_request:
+            changes = self.copr_build_helper.pull_request_object.changes


I double checked the definitions of GitHub and GitLab definitions of the MR / PR webhook events and you are definitely right that the information related to what files were changed is not present.

Do I understand correctly that your current approach is the following?

In parser.py, once a PR / MR webhook event is received, use ogr to get a list of commits (cast as CommitInfo) contained in this PR / MR.

Inside the AreFilesChanged class, take the list of commits and aggregate the three lists of files added, modified and removed.

In the current implementation, it does not go through the ogr and with the changes in the parser.py (parse_github_push_event) it would be materializing the webhook data to a more complete python object with the included files changed (but I did not properly test it, the failing CI should give us some reference for where this is connected).

2. Inside the AreFilesChanged class, take the list of commits and aggregate the three lists of files added, modified and removed.

That is what it is supposed to happen (replacing self.copr_build_helper.pull_request_object.changes, that is still from the ogr required change, I will drop it to make it clearer). As soon as we can get the output of parse_github_push_event in this checker class using a mixin, then that would unlock everything that we were missing here.

Signed-off-by: Cristian Le <[email protected]>

softwarefactory-project-zuul · 2025-11-12T15:45:36Z

Build failed.
https://softwarefactory-project.io/zuul/t/packit-service/buildset/771c89aa6d1d4d67a65c6ac54086de2b

✔️ pre-commit SUCCESS in 1m 45s
❌ packit-service-tests FAILURE in 5m 51s

betulependule · 2025-11-13T09:55:04Z

packit_service/worker/checker/copr.py

+            if not isinstance(pr_event, (github.pr.Action, gitlab.mr.Action)):
+                # TODO: What about comments?
+                raise NotImplementedError()
+            # TODO: How to handle PRs?


How about we create another Mixin class, which will use the existing get_pr_files_diff function in ogr to retrieve the files diff? It should return a dictionary, so we would need to process it into a list of files changed.

// dictionary representing files diff files_changed = project.get_pr_files_diff(pr_id=id_from_pr_event)

It does not seem to be available for github and gitlab? https://github.com/search?q=repo%3Apackit%2Fogr%20get_pr_files_diff&type=code And I think the internal API would not make it efficient to query that format. Maybe we go through packit/ogr#921?

Oh, you're right. In that case using packit/ogr#921 sounds good to me.

LecrisUT commented Apr 16, 2025

View reviewed changes

packit_service/worker/checker/copr.py Outdated Show resolved Hide resolved

LecrisUT force-pushed the feat/paths branch from 3f39b0b to 1d8e795 Compare April 16, 2025 13:52

LecrisUT force-pushed the feat/paths branch from 1d8e795 to fa55b7e Compare May 23, 2025 09:10

LecrisUT commented May 23, 2025

View reviewed changes

usercont-release-bot added this to Packit Kanban Board May 28, 2025

github-project-automation bot moved this to new in Packit Kanban Board May 28, 2025

LecrisUT commented May 28, 2025

View reviewed changes

packit_service/events/github/push.py Outdated Show resolved Hide resolved

lbarcziova moved this from new to in-progress in Packit Kanban Board May 29, 2025

lbarcziova mentioned this pull request Jul 23, 2025

When using monorepo, react only on changes done to configured paths #2006

Open

2 tasks

LecrisUT force-pushed the feat/paths branch from 4af08c0 to 5deb470 Compare July 29, 2025 14:16

LecrisUT force-pushed the feat/paths branch from 5deb470 to c309f30 Compare July 29, 2025 14:20

TomasTomecek assigned betulependule Oct 6, 2025

betulependule reviewed Oct 17, 2025

View reviewed changes

betulependule reviewed Nov 12, 2025

View reviewed changes

LecrisUT added 3 commits November 12, 2025 15:57

Prepare a checker to check for paths changes

22eccf6

Save the commit info from the webhook

0c617a6

Signed-off-by: Cristian Le <[email protected]>

[WIP] Implement AreFilesChanged

fd28dfc

Signed-off-by: Cristian Le <[email protected]>

LecrisUT force-pushed the feat/paths branch from c309f30 to fd28dfc Compare November 12, 2025 15:39

betulependule reviewed Nov 13, 2025

View reviewed changes

		# FIXME: This is probably unnecessary
		package_config = self.package_config.get_package_config_for(self.job_config)

	parser = nested_get(
	Parser.MAPPING,
	source,
	event_type,
	default=Parser.parse_event,
	)
	event_object: Optional[Event] = parser(event)

		if self.job_config.trigger == JobConfigTriggerType.pull_request:
		changes = self.copr_build_helper.pull_request_object.changes

[WIP] feat: Filter copr_build jobs based on paths and files changed #2780

Are you sure you want to change the base?

[WIP] feat: Filter copr_build jobs based on paths and files changed #2780

Uh oh!

Conversation

LecrisUT commented Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented Apr 16, 2025

Uh oh!

softwarefactory-project-zuul bot commented Apr 16, 2025

Uh oh!

softwarefactory-project-zuul bot commented May 23, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Uh oh!

softwarefactory-project-zuul bot commented May 28, 2025

Uh oh!

lbarcziova commented May 28, 2025

Uh oh!

LecrisUT commented May 28, 2025

Uh oh!

lbarcziova commented May 30, 2025

Uh oh!

LecrisUT commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lbarcziova commented May 30, 2025

Uh oh!

lachmanfrantisek commented Jul 22, 2025

Uh oh!

LecrisUT commented Jul 22, 2025

Footnotes

Uh oh!

lachmanfrantisek commented Jul 23, 2025

Uh oh!

lbarcziova commented Jul 23, 2025

Uh oh!

LecrisUT commented Jul 29, 2025

Uh oh!

softwarefactory-project-zuul bot commented Jul 29, 2025

Uh oh!

betulependule left a comment

Choose a reason for hiding this comment

Uh oh!

betulependule commented Oct 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LecrisUT commented Oct 17, 2025

Uh oh!

betulependule commented Oct 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

[WIP] feat: Filter `copr_build` jobs based on `paths` and files changed #2780

[WIP] feat: Filter `copr_build` jobs based on `paths` and files changed #2780

LecrisUT commented Apr 16, 2025 •

edited

Loading

LecrisUT commented May 30, 2025 •

edited

Loading