Slow (2 hours) for 7000 targets

[jjh42]

Hi,

I'm not quite sure how to get a simple reproduction. In our repo (bazel 7.1.1, bazel-diff 7.0) when we have a large number (7000) targets change (due to e.g. changing a build flag) then bazel-diff get-impacted-targets step takes 2 hours to complete.

We are running a setup very similar to the example script in this repo.

I'm just wondering if you had any pointer how we could try and determine what's going on. From other comments, I don't think this is expected.

[tinder-maxwellelliott]

Hello there @jjh42,

Thanks for reporting this. Is there any way you could give #193 a try? You will need to set the --modified-filepaths flag seen here

bazel-diff/cli/src/main/kotlin/com/bazel_diff/cli/GenerateHashesCommand.kt

Line 140 in 64b3ae2

names = ["-m", "--modified-filepaths"],

to get the speed up

[jjh42]

Thanks for your response. I will give that a try. I think I can just use bazel's git_override to get that particular commit.

Btw is there a reason you don't merge that branch in / any downsides?

[tinder-maxwellelliott]

I don't see any downsides from this approach, it should be just as precise with dramatically less file reads. I will work on trying to use this internally to help with validation

[tinder-maxwellelliott]

Did this end up working for you?

[jjh42]

Hi @tinder-maxwellelliott. Unfortunately it didn't seem to help us much. Still taking over 2 hours.

Just to confirm we get a list of targets that changed with

git diff --name-only "$previous_revision" "$final_revision" > "$modified_file_paths"

and then the two generate-hashes we add

--modified-filepaths=$modified_file_paths

[tinder-maxwellelliott]

Are you using cquery for your execution? That can explain the slowdown

[jjh42]

(sorry for the slow response, I didn't see a notification about this reply).

I don't think so? This is the script we were trying.

#!/bin/bash
set -e
echo "Starting bazel diff $(date)"
# Path to your Bazel WORKSPACE directory
workspace_path=$1
# Path to your Bazel executable
bazel_path=$2
# Starting Revision SHA
previous_revision=$3
# Final Revision SHA
final_revision=$4
impacted_targets_path=$5
starting_hashes_json="/tmp/starting_hashes.json"
final_hashes_json="/tmp/final_hashes.json"
bazel_diff="/tmp/bazel_diff"
modified_file_paths="/tmp/modified_file_paths.txt"

"$bazel_path" run //:bazel-diff --script_path="$bazel_diff"

echo git -C "$workspace_path" checkout "$previous_revision" --quiet
git -C "$workspace_path" checkout "$previous_revision" --quiet
git stash

# This should provide a list of files that were modified between the two revisions
git diff --name-only "$previous_revision" "$final_revision" > "$modified_file_paths"

echo "Generating Hashes for Revision '$previous_revision'"
$bazel_diff generate-hashes --modified-filepaths=$modified_file_paths -w "$workspace_path" -b "$bazel_path" $starting_hashes_json --includeTargetType

git -C "$workspace_path" checkout -f "$final_revision" --quiet
git stash

echo "Generating Hashes for Revision '$final_revision'"
$bazel_diff generate-hashes --modified-filepaths=$modified_file_paths -w "$workspace_path" -b "$bazel_path" \
   $final_hashes_json --includeTargetType

echo "Determining Impacted Targets $(date)"`

[tinder-maxwellelliott]

What happens if you execute the jar directly instead of via Bazel?