Skip to content

Avoid scheduling multiple identical jobs #2912

@a-wai

Description

@a-wai

Is your feature request related to a problem? Please describe.
Child nodes are generally created on state transitions of a given node. However, the logic used is naive enough that it can potentially create multiple instance of the same job/test: as child nodes are created only by looking at the state field of updated nodes, without keeping track of their previous state, they could be triggered on any update.

For instance, if a job wants to update the timeout of its parent node, or add an artifact to it, a "node updated" event will be sent. If said parent node is in state available, then all of its child nodes will be created again.

We ran into this issue on staging a few weeks ago, where test jobs were created when the corresponding kbuild node went to result: pass, without looking at the state field. As kbuilds were going through multiple state changes after their result was set to pass, identical child nodes were created following each state transition. This has been mitigated by triggering test jobs creation on state: available instead, but it's still fragile.

Describe the solution you'd like
It would be interesting to check if an identical job already exists by ensuring there's no existing node with the same combination of:

  • parent
  • kind
  • name
  • runtime
  • platform

If the search using those parameters return at least one result, then node creation should be skipped. Those fields should be enough to ensure all test jobs are unique (kind might not even be needed here), but I might be missing something.

Describe alternatives you've considered
I thought about keeping track of the nodes' states internally and only act on changes affecting the field mentioned in the scheduler entry (e.g. create child node only when a kbuld node's state goes from running to available, not on updates that didn't affect state).

Additional context
This problem only occurred during development, and very rarely. However, I believe we might run into such trouble more often as the project grows bigger and more complex, and therefore I think it should be addressed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions