Skip to content

Only process 1 graph request at a time for a given run#887

Merged
lewisbirks merged 6 commits intojenkinsci:mainfrom
lewisbirks:reduce-graph-processing
Jul 10, 2025
Merged

Only process 1 graph request at a time for a given run#887
lewisbirks merged 6 commits intojenkinsci:mainfrom
lewisbirks:reduce-graph-processing

Conversation

@lewisbirks
Copy link
Contributor

@lewisbirks lewisbirks commented Jul 9, 2025

An initial look into addressing #886, for #862 the initial computationally expensive part is creating the PipelineNodeTreeScanner and as such I have created a way of ensuring that all incoming requests use the same parent PipelineNodeGraphAdapter.
This required some additional synchronisation code to ensure that concurrent modifications aren't made to the underlying data.

Testing done

curl --parallel --parallel-immediate --parallel-max 30 --config urls.txt

where urls.txt contains about 90 entries that look like

url = http://localhost:8080/jenkins/job/demo/69/pipeline-overview/tree
output = ./results/tree/01.json
url = http://localhost:8080/jenkins/job/demo/69/pipeline-overview/allSteps
output = ./results/steps/01.json

These files can then be checked to all be the same using a diff

for f1 in ./results/steps/*; do
    for f2 in ./results/steps/*; do
        if [[ "$f2" > "$f1" ]]; then
            if ! diff -B "$f1" "$f2" ; then
                echo "Files $f1 and $f2 differ"
            fi
        fi
    done
done
echo "Checks complete"

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests that demonstrate the feature works or the issue is fixed

Signed-off-by: Lewis Birks <22620804+lewisbirks@users.noreply.github.com>
@lewisbirks
Copy link
Contributor Author

lewisbirks commented Jul 9, 2025

Thinking about it, a better approach might be to wrap the XXXApi objects instead as that would remove the need to worry about mutation. However this does still present some issues. The current way of working expects all of these to be single use objects, as seen with the need to deal the FlowNodeWrapper being mutated. If we cached at an Api object level we would want to somehow use the same PipelineNodeTreeScanner or PipelineNodeGraphAdapter between api objects as that is what is doing a lot of the heavy computation

@lewisbirks lewisbirks added the enhancement New feature or request label Jul 9, 2025
…NodeGraphAdaptor

Signed-off-by: Lewis Birks <22620804+lewisbirks@users.noreply.github.com>
Signed-off-by: Lewis Birks <22620804+lewisbirks@users.noreply.github.com>
Signed-off-by: Lewis Birks <22620804+lewisbirks@users.noreply.github.com>
Signed-off-by: Lewis Birks <22620804+lewisbirks@users.noreply.github.com>
Signed-off-by: Lewis Birks <22620804+lewisbirks@users.noreply.github.com>
@lewisbirks lewisbirks marked this pull request as ready for review July 10, 2025 10:12
@lewisbirks lewisbirks requested a review from a team as a code owner July 10, 2025 10:12
Copy link
Member

@timja timja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, tested locally with curl running multiple requests at the same time and can see only one gets 'built'.

@lewisbirks lewisbirks merged commit 390f33a into jenkinsci:main Jul 10, 2025
17 checks passed
@lewisbirks lewisbirks deleted the reduce-graph-processing branch July 10, 2025 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants