Skip to content

Commit 380c55a

Browse files
andrewgazelkakhaneliman
andrewgazelka
authored and
khaneliman
committed
files: batch link creation and target checks
The linkGeneration and checkLinkTargets activation steps forked several processes per managed file: a dirname command substitution plus mkdir and ln for every link, and a readlink per existing target during the collision check. On platforms where fork+exec costs a few milliseconds (notably darwin) this dominates activation time: a profile with 365 links spent about 3.4s in linkGeneration and 1.1s in checkLinkTargets. Classify targets with bash builtins, resolve all existing symlinks with a single readlink -z call, create missing parent directories with one mkdir -p, and group new links into one ln -sfn -t call per target directory. Only targets occupied by a regular file or directory fall back to the original per-file path, preserving the backup command, extension and overwrite handling, the identical-content skip, the failure on a real directory in the way (previously via ln -T), and dry-run output. Both batched readlink calls check that they consumed one record per operand before acting on the results. readlink prints nothing for an operand that vanished since the classification loop, and its exit status is lost through the process substitution, so a missing record would silently shift every later result onto the wrong source: the link step would leave those links pointing at the old generation, and the collision check would skip a foreign file that is about to be clobbered. A short count now fails activation and asks for a retry. On the same 365-link profile the two steps now take about 0.2s on a full relink and under 0.05s when the generation is unchanged, with an identical resulting tree.
1 parent bf9ce9f commit 380c55a

2 files changed

Lines changed: 146 additions & 24 deletions

File tree

‎modules/files.nix‎

Lines changed: 85 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,91 @@ in
191191
192192
newGenFiles="$1"
193193
shift
194+
195+
# Classify every target using bash builtins only: even one forked
196+
# process per file costs several milliseconds on some platforms
197+
# (notably darwin), which dominates activation time when a profile
198+
# carries hundreds of links. Targets occupied by a regular file or
199+
# directory keep the original per-file handling (backup and
200+
# identical-content skip) on the slow path below.
201+
declare -a symlinkTargets=() symlinkSources=()
202+
declare -a linkSources=() linkDirs=()
203+
declare -a slowSources=()
194204
for sourcePath in "$@" ; do
205+
relativePath="''${sourcePath#$newGenFiles/}"
206+
targetPath="$HOME/$relativePath"
207+
if [[ -L "''${targetPath%/*}" ]] ; then
208+
# The parent directory is itself a symlink (e.g. a stale
209+
# whole-directory link from an older layout). The batched
210+
# `ln -n -t` below would refuse it ("Not a directory"), while
211+
# the original per-file `ln -T` traverses it; keep upstream
212+
# behavior on the slow path.
213+
slowSources+=("$sourcePath")
214+
elif [[ -L "$targetPath" ]] ; then
215+
symlinkTargets+=("$targetPath")
216+
symlinkSources+=("$sourcePath")
217+
elif [[ -e "$targetPath" ]] ; then
218+
slowSources+=("$sourcePath")
219+
else
220+
linkSources+=("$sourcePath")
221+
linkDirs+=("''${targetPath%/*}")
222+
fi
223+
done
224+
225+
# Resolve all existing symlinks with a single readlink call and
226+
# relink only those not already pointing at the new generation.
227+
if [[ ''${#symlinkTargets[@]} -gt 0 ]] ; then
228+
i=0
229+
while IFS= read -r -d "" currentSource ; do
230+
if [[ "$currentSource" != "''${symlinkSources[i]}" ]] ; then
231+
linkSources+=("''${symlinkSources[i]}")
232+
linkDirs+=("''${symlinkTargets[i]%/*}")
233+
fi
234+
i=$(( i + 1 ))
235+
done < <(readlink -z -- "''${symlinkTargets[@]}")
236+
237+
# readlink prints no record for an operand that vanished since
238+
# the classification loop above, and its exit status is lost
239+
# through the process substitution. A missing record shifts
240+
# every later result onto the wrong source, so the links after
241+
# it would be compared against someone else's target and left
242+
# stale. The record count is the only signal that happened.
243+
if [[ $i -ne ''${#symlinkTargets[@]} ]] ; then
244+
errorEcho "A link target changed while resolving symlinks; retry activation."
245+
exit 1
246+
fi
247+
fi
248+
249+
# Create all missing parent directories in one mkdir call.
250+
declare -A missingDirs=()
251+
for targetDir in "''${linkDirs[@]}" ; do
252+
[[ -d "$targetDir" ]] || missingDirs[$targetDir]=1
253+
done
254+
if [[ ''${#missingDirs[@]} -gt 0 ]] ; then
255+
run mkdir -p $VERBOSE_ARG -- "''${!missingDirs[@]}" || exit 1
256+
fi
257+
258+
# Group the pending links by parent directory, one ln call per
259+
# directory. The link name always equals the source basename, and
260+
# -f -n together replace a stale symlink even when it points at a
261+
# directory (the case -T guarded against in the per-file version;
262+
# a regular directory in the way takes the slow path instead and
263+
# fails there just like it always did).
264+
declare -A dirBatches=()
265+
for i in "''${!linkSources[@]}" ; do
266+
dirBatches[''${linkDirs[i]}]+="$i "
267+
done
268+
for targetDir in "''${!dirBatches[@]}" ; do
269+
batch=()
270+
for i in ''${dirBatches[$targetDir]} ; do
271+
batch+=("''${linkSources[i]}")
272+
done
273+
run ln -sfn $VERBOSE_ARG -t "$targetDir" -- "''${batch[@]}" || exit 1
274+
done
275+
276+
# Slow path: the target exists and is not a symlink. This is the
277+
# original per-file logic, kept verbatim for the rare collisions.
278+
for sourcePath in "''${slowSources[@]}" ; do
195279
relativePath="''${sourcePath#$newGenFiles/}"
196280
targetPath="$HOME/$relativePath"
197281
if [[ -e "$targetPath" && ! -L "$targetPath" ]] ; then
@@ -209,7 +293,7 @@ in
209293
fi
210294
211295
if [[ -e "$targetPath" && ! -L "$targetPath" ]] && cmp -s "$sourcePath" "$targetPath" ; then
212-
# The target exists but is identical – don't do anything.
296+
# The target exists but is identical - don't do anything.
213297
verboseEcho "Skipping '$targetPath' as it is identical to '$sourcePath'"
214298
else
215299
# Place that symlink, --force

‎modules/files/check-link-targets.sh‎

Lines changed: 61 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,39 @@ forcedPaths=(@forcedPaths@)
1010

1111
newGenFiles="$1"
1212
shift
13+
14+
# Check a target that already exists and is not a symlink owned by Home
15+
# Manager.
16+
function checkCollision() {
17+
local sourcePath="$1"
18+
local targetPath="$2"
19+
20+
if cmp -s "$sourcePath" "$targetPath"; then
21+
# First compare the files' content. If they're equal, we're fine.
22+
warnEcho "Existing file '$targetPath' is in the way of '$sourcePath', will be skipped since they are the same"
23+
elif [[ ! -L "$targetPath" && -n "$HOME_MANAGER_BACKUP_COMMAND" ]] ; then
24+
# Next, try to run the custom backup command. Assume this always succeeds.
25+
verboseEcho "Existing file '$targetPath' exists and differs from '$sourcePath'. '$HOME_MANAGER_BACKUP_COMMAND' will be used to backup the file."
26+
elif [[ ! -L "$targetPath" && -n "$HOME_MANAGER_BACKUP_EXT" ]] ; then
27+
# Next, try to move the file to a backup location if configured and possible
28+
backup="$targetPath.$HOME_MANAGER_BACKUP_EXT"
29+
if [[ -e "$backup" && -z "$HOME_MANAGER_BACKUP_OVERWRITE" ]] ; then
30+
collisionErrors+=("Existing file '$backup' would be clobbered by backing up '$targetPath'")
31+
elif [[ -e "$backup" && -n "$HOME_MANAGER_BACKUP_OVERWRITE" ]] ; then
32+
warnEcho "Existing file '$targetPath' is in the way of '$sourcePath' and '$backup' exists. Backup will be clobbered due to HOME_MANAGER_BACKUP_OVERWRITE=1"
33+
else
34+
warnEcho "Existing file '$targetPath' is in the way of '$sourcePath', will be moved to '$backup'"
35+
fi
36+
else
37+
# Fail if nothing else works
38+
collisionErrors+=("Existing file '$targetPath' would be clobbered")
39+
fi
40+
}
41+
42+
# Classify every target using bash builtins only: even one forked process
43+
# per file costs several milliseconds on some platforms (notably darwin),
44+
# which dominates activation time when a profile carries hundreds of links.
45+
declare -a linkTargets=() linkSources=()
1346
for sourcePath in "$@" ; do
1447
relativePath="${sourcePath#$newGenFiles/}"
1548
targetPath="$HOME/$relativePath"
@@ -24,32 +57,37 @@ for sourcePath in "$@" ; do
2457

2558
if [[ -n $forced ]]; then
2659
verboseEcho "Skipping collision check for $targetPath"
27-
elif [[ -e "$targetPath" \
28-
&& ! "$(readlink "$targetPath")" == $homeFilePattern ]] ; then
29-
# The target file already exists and it isn't a symlink owned by Home Manager.
30-
if cmp -s "$sourcePath" "$targetPath"; then
31-
# First compare the files' content. If they're equal, we're fine.
32-
warnEcho "Existing file '$targetPath' is in the way of '$sourcePath', will be skipped since they are the same"
33-
elif [[ ! -L "$targetPath" && -n "$HOME_MANAGER_BACKUP_COMMAND" ]] ; then
34-
# Next, try to run the custom backup command. Assume this always succeeds.
35-
verboseEcho "Existing file '$targetPath' exists and differs from '$sourcePath'. '$HOME_MANAGER_BACKUP_COMMAND' will be used to backup the file."
36-
elif [[ ! -L "$targetPath" && -n "$HOME_MANAGER_BACKUP_EXT" ]] ; then
37-
# Next, try to move the file to a backup location if configured and possible
38-
backup="$targetPath.$HOME_MANAGER_BACKUP_EXT"
39-
if [[ -e "$backup" && -z "$HOME_MANAGER_BACKUP_OVERWRITE" ]] ; then
40-
collisionErrors+=("Existing file '$backup' would be clobbered by backing up '$targetPath'")
41-
elif [[ -e "$backup" && -n "$HOME_MANAGER_BACKUP_OVERWRITE" ]] ; then
42-
warnEcho "Existing file '$targetPath' is in the way of '$sourcePath' and '$backup' exists. Backup will be clobbered due to HOME_MANAGER_BACKUP_OVERWRITE=1"
43-
else
44-
warnEcho "Existing file '$targetPath' is in the way of '$sourcePath', will be moved to '$backup'"
45-
fi
46-
else
47-
# Fail if nothing else works
48-
collisionErrors+=("Existing file '$targetPath' would be clobbered")
49-
fi
60+
elif [[ -L "$targetPath" && -e "$targetPath" ]] ; then
61+
linkTargets+=("$targetPath")
62+
linkSources+=("$sourcePath")
63+
elif [[ -e "$targetPath" ]] ; then
64+
checkCollision "$sourcePath" "$targetPath"
5065
fi
5166
done
5267

68+
# Resolve all existing symlinks with a single readlink call. A link into a
69+
# Home Manager generation is ours; anything else is a collision candidate.
70+
if [[ ${#linkTargets[@]} -gt 0 ]] ; then
71+
i=0
72+
while IFS= read -r -d "" currentSource ; do
73+
if [[ ! "$currentSource" == $homeFilePattern ]] ; then
74+
checkCollision "${linkSources[i]}" "${linkTargets[i]}"
75+
fi
76+
i=$(( i + 1 ))
77+
done < <(readlink -z -- "${linkTargets[@]}")
78+
79+
# readlink prints no record for an operand that vanished since the
80+
# classification loop above, and its exit status is lost through the
81+
# process substitution. A missing record shifts every later result onto
82+
# the wrong source, so a foreign symlink could be checked against the
83+
# wrong target and pass as ours. The record count is the only signal
84+
# that happened.
85+
if [[ $i -ne ${#linkTargets[@]} ]] ; then
86+
errorEcho "A link target changed while resolving symlinks; retry activation."
87+
exit 1
88+
fi
89+
fi
90+
5391
if [[ ${#collisionErrors[@]} -gt 0 ]] ; then
5492
errorEcho "Please do one of the following:
5593
- In standalone mode, use 'home-manager switch -b backup' to back up"\

0 commit comments

Comments
 (0)