Repository navigation
Expand file tree
/
Copy pathdev
More file actions
executable file
·2076 lines (1922 loc) · 116 KB
/
Copy pathdev
File metadata and controls
executable file
·2076 lines (1922 loc) · 116 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env rad
---
Muster's one door: build, test, and lint.
Run with no flags to take the full gate. CI runs exactly that and nothing else, so the
command a contributor types and the command that gates a merge cannot drift apart.
Flags are clustered short bools, so the common combinations stay one token:
'./dev -t' tests, './dev -tl' tests and lints, './dev -fbt' reformats then builds and
tests.
---
args:
deps d bool # Fetch and build the pinned libghostty.
build b bool # Compile everything.
test t bool # Run the suite.
lint l bool # Report style violations, changing nothing.
format f bool # Rewrite sources into canonical style.
clean c bool # Delete build artifacts.
proto bool # Regenerate the seam's Swift types, whether or not the schema changed.
bundle bool # Assemble muster.app around the built binary. Not in the gate.
notarize bool # Sign for release, send it to Apple, and staple. Needs credentials; not in the gate.
contract bool # Run the app against a real muster-daemon. Needs a GUI session; not in the gate.
ssh bool # Run the remote tier against the devenv container. Needs docker; not in the gate.
claude_code bool # Drive the installed Claude Code for one turn through its hooks and its screen. Reaches the network; not in the gate.
codex bool # Drive the installed Codex through its hooks, its screen and the doorbell. Reaches the network; not in the gate.
opencode bool # Drive OpenCode 1.18+ on a free model through its plugin, its screen and the doorbell. Reaches the network; not in the gate.
linux bool # Run the daemon's and detection's suites on Linux, in docker. Not in the gate.
perf bool # Check daemon startup and the per-unit budgets in perf/baseline.json. Not in the gate.
record bool # With --perf, record this run as the new baseline instead of judging it.
anyway bool # With --perf, measure on a busy machine anyway rather than refusing.
latency bool # Time input-to-glyph through muster-daemon. Not in the gate.
doctor bool # Say what this repo's own tooling has left running on this machine.
// Every Zig call goes through this, which runs the build deps/zig.pin names and fetches it
// once per machine, so a checkout carries its own compiler and a pin move breaks no other.
ZIG = "tools/zig"
ZIG_PIN = "deps/zig.pin"
GHOSTTY_REPO = "https://github.com/ghostty-org/ghostty"
GHOSTTY_DIR = "deps/ghostty"
XCFRAMEWORK = "deps/ghostty/macos/GhosttyKit.xcframework"
BUILT_STAMP = "deps/.built"
// Muster's own changes to libghostty, applied in name order on top of the pin and carried
// for good rather than until upstream takes them: each is additive C API, kept small so a
// re-pin rebases it rather than rewriting it (docs/mip/0003-own-daemon.md, section 5).
GHOSTTY_PATCHES = "deps/ghostty-patches"
// Both builds of the pin: the surface API the shell renders through, and the VT the
// core encodes keys with and the tests read grids from - as a dylib for anything that
// shares a process with the surface API, and as an archive for the daemon, which does not.
//
// The Linux daemons take an archive each, built from the same patched checkout into a prefix of
// its own (MIP-3, section 12) - under zig's name for the target, which muster-vt's build script
// maps cargo's name onto.
LINUX_DAEMONS = {
"x86_64-unknown-linux-musl": "x86_64-linux-musl",
"aarch64-unknown-linux-musl": "aarch64-linux-musl",
}
// What the daemon gives the shells it starts, from the same pin: Ghostty's terminfo and its
// shell integration. A directory beside the daemon rather than bytes inside it, because the
// bash and zsh scripts are GPLv3 and the daemon is not (packaging/muster-daemon-data/README.md).
DAEMON_DATA = "deps/ghostty/zig-out/muster-daemon-data"
DEP_ARTIFACTS = [
XCFRAMEWORK,
"deps/ghostty/zig-out/lib/libghostty-vt.a",
"deps/ghostty/zig-out/include/ghostty/vt.h",
"deps/ghostty/zig-out/x86_64-linux-musl/lib/libghostty-vt.a",
"deps/ghostty/zig-out/aarch64-linux-musl/lib/libghostty-vt.a",
"deps/ghostty/zig-out/share/terminfo/78/xterm-ghostty",
// The entry's source, which a pane's ssh installs on a host without it.
"deps/ghostty/zig-out/share/terminfo/ghostty.terminfo",
]
GHOSTTY_APP_BUNDLES = [
"deps/ghostty/macos/build/ReleaseLocal/Ghostty.app",
"deps/ghostty/zig-out/Ghostty.app",
]
// Short and outside the toolchain's output directories, because it is the one artifact a
// person types the path of - and `swift package clean` still takes it with the rest.
BUNDLE = ".build/muster.app"
// The app's identity to the OS, and what a notification permission grant will attach to.
BUNDLE_ID = "dev.amterp.muster"
// What the bundle is called once it leaves this machine. `.build/muster.app` is lowercase
// because it is a path somebody types; nobody types the path of an app they downloaded, and
// what a person sees in /Applications should be the name the app calls itself.
RELEASE_APP = "Muster.app"
// The daemon's own bundle, nested in the app's Contents/Library the way a helper application
// is. It exists so the daemon can be started through Launch Services rather than spawned, and
// that is not packaging pedantry: macOS charges a pane's protected request to the responsible
// process, only a process Launch Services started is its own, and the daemon is started and
// never stopped - so this is what makes one permission grant cover every pane across every
// relaunch instead of lapsing when the app quits (docs/observations/macos-26.4.1.md).
//
// A name of its own rather than the app's, because both appear in System Settings under
// Privacy and two rows reading "Muster" are two rows nobody can tell apart when they want to
// revoke one.
DAEMON_BUNDLE = "MusterSessions.app"
DAEMON_BUNDLE_ID = "dev.amterp.muster.sessions"
DAEMON_BUNDLE_NAME = "Muster Sessions"
ICNS = "packaging/Muster.icns"
ENTITLEMENTS = "packaging/muster-release.entitlements"
THIRD_PARTY = "licenses/THIRD-PARTY.md"
GHOSTTY_LICENSES = "licenses/GHOSTTY-EMBEDDED.md"
PROTO = "proto/muster.proto"
PROTO_OUT = "Sources/MusterMac/Generated"
// Gitignored, beside the code it describes: the hash of the schema that produced it. The
// Rust side needs no equivalent - cargo tracks the same file itself.
PROTO_STAMP = "Sources/MusterMac/Generated/.proto.sha256"
// What the gate's Rust suite printed and the load while it ran, for explaining a red run. Under
// target/, so each checkout has its own and nothing commits them.
GATE_RUST_LOG = "target/gate/rust-tests.log"
GATE_LOAD = "target/gate/load.txt"
// When each step of this run began, for the table every run ends with. A file rather than a
// variable, so a red run's timing outlives it beside the suite's log.
GATE_STEPS = "target/gate/steps-{uuid_v4()}.txt"
RUN_STARTED = now().epoch.millis
// Every command captured here names its targets `stdout`, `stderr` and `code`, and never
// anything else - not even `_`.
//
// Rad fills a capture by name when every target is spelled one of those three, and by
// position otherwise - and v0.12 reorders the positional form from (code, stdout, stderr)
// to (stdout, stderr, code). A named capture reads the same either way, so bumping
// deps/rad.pin does not mean re-reading every one.
//
// It is worth being strict about because the failure is quiet. `uname -s` captured
// positionally binds the exit code on one side and "Darwin" on the other, and the gate
// died at the `.trim()` two lines later, naming a type rather than a command.
//
// Where one result is wanted, this reads it off the command instead - `$`cmd`.stdout` -
// which has no slots to get wrong and lets the value keep a name that says what it is.
// The captures that remain are the ones that also need the other stream swallowed: an
// accessor captures only the stream it names, and the rest still reach the terminal.
fn announce(name):
print(bold("\n▸ {name}"))
mark(name)
// Notes that `name` began now. A part of a step is named "<step>: <part>" and is reported under
// it. Nothing here can fail a run: a file that cannot be written costs only the timing.
fn mark(name):
line = "{now().epoch.millis}\t{name}\n"
// Made again here because --clean deletes target/ part way through a run.
write_file(GATE_STEPS, line, append=true) catch:
quiet $`mkdir -p target/gate` catch:
pass
write_file(GATE_STEPS, line, append=true) catch:
pass
// How long each step took, off the marks: a step lasts until the next one begins and the last
// until now, and a part until the next mark of either kind. The total is from RUN_STARTED.
// Nothing recorded each step's time before this, so a gate's ten minutes could only be divided
// by subtraction.
fn report_steps():
read = read_file(GATE_STEPS) catch:
return
marks = []
for line in read.content.split_lines():
fields = line.split("\t")
if len(fields) == 2:
marks += [[parse_int(fields[0]), fields[1]]]
if len(marks) == 0:
return
ended = now().epoch.millis
print(bold("\n▸ time"))
for step in marks with loop:
part = is_part(step[1])
end = ended
for later in marks[loop.idx + 1:]:
if part or not is_part(later[1]):
end = later[0]
break
seconds = (end - step[0]) / 1000.0
label = part ? " {step[1].split(": ")[1]}" : step[1]
print(dim("{seconds:>7.1}s {label}"))
total = (ended - RUN_STARTED) / 1000.0
print("{total:>7.1}s total")
print(dim(" marks: {GATE_STEPS}"))
fn is_part(name):
return matches(name, ": ", partial=true)
// What the machine is doing, sampled once and said out loud at the top of every run.
//
// Per-OS, so this is the shell's answer to hand over rather than the core's - the same as
// the log path and the screens. A load average means nothing without a core count beside
// it, and on Apple silicon the count that matters is the *performance* cores: past that
// many runnable threads the scheduler starts putting work on the efficiency cores, which
// run this kind of code at roughly half the speed. That is the line between a run that is
// merely slow and a run whose numbers lie, and it is where --perf refuses.
//
// Returns [cores, fast_cores, load1, load5, load15]. Nothing here can fail a run: a machine
// that will not answer yields zeros, and zero reads as "unknown" at every use. Refusing to
// build because a sysctl was missing would be absurd.
fn machine():
stdout, stderr, code = quiet $`uname -s` catch:
stdout = ""
if stdout.trim() == "Darwin":
stdout, stderr, code = quiet $`sysctl -n hw.ncpu` catch:
stdout = ""
cores = parse_int(stdout.trim()) catch:
cores = 0
// hw.perflevel0 is the performance half of an Apple silicon chip, and is absent on
// Intel Macs - where there is one kind of core and hw.ncpu already counted it.
stdout, stderr, code = quiet $`sysctl -n hw.perflevel0.logicalcpu` catch:
stdout = ""
fast = parse_int(stdout.trim()) catch:
fast = cores
stdout, stderr, code = quiet $`sysctl -n vm.loadavg` catch:
stdout = ""
loads = three_loads(stdout)
return [cores, fast, loads[0], loads[1], loads[2]]
stdout, stderr, code = quiet $`nproc` catch:
stdout = ""
cores = parse_int(stdout.trim()) catch:
cores = 0
stdout, stderr, code = quiet $`cat /proc/loadavg` catch:
stdout = ""
loads = three_loads(stdout)
// Nothing here reports a performance/efficiency split, so every core counts as a fast one.
return [cores, cores, loads[0], loads[1], loads[2]]
// The three averages out of a load-average line, whatever else is around them.
//
// One parser for two shapes rather than one per platform: macOS answers with the numbers
// between braces and /proc/loadavg follows them with a running/total count and a pid. Both
// are whitespace-separated, so taking the first three fields that parse as a number covers
// each without knowing which it is reading. A line neither of them recognises yields zeros,
// because a diagnostic that cannot read the machine should say nothing rather than stop it.
fn three_loads(text):
found = []
for field in text.split(" "):
if len(found) < 3:
// A load average is never negative, so this stands in for "not a number".
value = parse_float(field.trim()) catch:
value = -1.0
if value >= 0:
found = found + [value]
while len(found) < 3:
found = found + [0.0]
return found
// The one line that would have turned seventy-four invisible minutes into something anybody
// could see (a_29iw10a58). On every run including narrowed ones, because a number raised only
// when it is bad is a number people learn to scroll past, and because the run it explains is
// usually the one nobody thought to question.
fn report_machine(cores, fast, load1, load5, load15):
if cores == 0:
print(dim("machine: this OS would not say how many cores it has or what its load is"))
return
counted = cores == fast ? "{cores} cores" : "{cores} cores ({fast} performance)"
line = "machine: {counted}, load {load1:.2} {load5:.2} {load15:.2}"
if load1 < fast:
print(dim(line))
return
// Not dim, because at this point the run is sharing every fast core with something else
// and how long it takes says nothing about the code.
print(yellow("{line} - the fast cores are already committed, so this run will be slow"))
// What to do about a red suite, said where it ends: the re-run of only what failed, and the
// machine's load when it was past the fast cores.
//
// Not a refusal and not a longer deadline. Parallel worktrees running the gate at once is a
// normal way to work here, and a test that passes alone and fails at load 30 reads as a broken
// test unless the run says otherwise - so an agent re-runs the whole gate rather than one test,
// and learns nothing either way (a_2LTQZhuFV). The load is the highest the run saw: sampled at
// the start and every five seconds through the Rust suite.
fn explain_red_suite(suite, fast, start_load):
report_steps()
if suite == "Rust":
stdout, stderr, code = quiet $`python3 tools/failed-tests.py {GATE_RUST_LOG}` catch:
stdout = ""
if stdout.trim() != "":
print_err(red("\nThe Rust suite failed. Re-run only what failed, on its own:"))
for command in stdout.trim().split_lines():
print_err(" {command}")
peak = peak_load(suite == "Rust" ? GATE_LOAD : "", start_load)
if fast == 0 or peak < fast:
return
print_err(yellow("\nThe machine's load reached {peak:.1} while the {suite} suite ran, past its {fast} performance cores."))
print_err(yellow("A test that timed out or saw things out of order may have been starved of CPU rather than"))
print_err(yellow("wrong. Run it on its own before reading the failure as a bug - and read the failure either"))
print_err(yellow("way, since passing alone shows only that it can pass. ./dev --doctor says what else is running."))
// The highest one-minute load among `start` and the samples in `samples`, if there are any.
fn peak_load(samples, start):
peak = start
if samples == "":
return peak
sampled = read_file(samples) catch:
return peak
for line in sampled.content.split_lines():
load = three_loads(line)[0]
if load > peak:
peak = load
return peak
// What this repo's own tooling has left running, named rather than counted.
//
// Reports and never fails. A check that could stop a build is one people stop running, and
// this one exists to be run at the moment somebody is already suspicious.
fn run_doctor():
doctor_processes()
doctor_muster_daemons()
doctor_orphaned_panes()
doctor_herdr()
doctor_devenv()
// Every muster-daemon on this machine, whoever started it, with the socket it serves.
//
// What each one holds is `muster daemons`' to say, from inside a window: it dials each socket
// with the daemon's own protocol, which a build script cannot speak. This half is the one the
// daemons' records cannot give - the ones nothing wrote down, a test's included.
fn doctor_muster_daemons():
print(bold("\nmuster daemons"))
stdout, stderr, code = quiet $`pgrep -a -x muster-daemon` catch:
stdout = ""
pids = stdout.trim().split_lines().filter(fn(pid) pid.trim() != "")
if len(pids) == 0:
print(dim(" none running"))
return
print(dim(" pid age socket"))
for pid in pids:
print(" {doctor_muster_daemon(pid.trim())}")
print(dim(" What each holds: muster daemons, from a pane. End one with: kill $(lsof -t <socket>.lock)"))
fn doctor_muster_daemon(pid):
stdout, stderr, code = quiet $`ps -o etime= -p {pid}` catch:
stdout = ""
age = stdout.trim()
if age == "":
blank = ""
return "{pid:<7}{blank:<13}gone - it exited while this was being written"
stdout, stderr, code = quiet $`lsof -a -p {pid} -U` catch:
stdout = ""
socket = ""
for line in stdout.split_lines():
fields = line.split(" ").filter(fn(field) field != "")
if len(fields) > 0 and fields[-1].ends_with(".sock"):
socket = fields[-1]
if socket == "":
return "{pid:<7}{age:<13}no socket - starting, handing over, or on its way out"
if not get_path(socket).exists:
return "{pid:<7}{age:<13}{socket} unreachable, its socket path is gone"
scratch = socket.starts_with("/tmp/") or socket.starts_with("/private/tmp/")
where = scratch ? "scratch" : "kept"
return "{pid:<7}{age:<13}{socket} {where}"
// Test panes' programs still running after their daemon has gone, which neither the list above
// nor `muster daemons` can show, since nothing serves them.
//
// A pane started to run a command is a shell whose arguments hold `$MUSTER_PANE_COMMAND`. An
// ordinary pane shell exits when its terminal hangs up; one running a command that ignores the
// hang-up does not. Its parent being launchd says nothing on its own, since every pane a handoff
// adopted has launchd for a parent too, and neither does its terminal, which ps names the same
// once its daemon is gone. A working directory under the harness's scratch root does: a test
// pane starts there and the owner's never do. Before test daemons ended with their tests, a
// killed test run left these running for weeks.
fn doctor_orphaned_panes():
print(bold("\ntest pane programs without a daemon"))
stdout, stderr, code = quiet $`ps -axo pid=,ppid=,etime=,command=` catch:
print(dim(" ps would not answer, so nothing here says what is running"))
return
words = [line.split(" ").filter(fn(field) field != "") for line in stdout.split_lines()]
commands = words.filter(fn(fields) len(fields) >= 4 and fields[1] == "1" and "$MUSTER_PANE_COMMAND" in join(fields[3:], " "))
rows = [[fields[0], fields[2], doctor_cwd(fields[0])] for fields in commands]
rows = rows.filter(fn(row) row[2].starts_with("/tmp/muster-test/") or row[2].starts_with("/private/tmp/muster-test/"))
// A pane of a daemon still serving its scratch root is that daemon's, adopted in a handoff.
daemons = stdout.split_lines().filter(fn(line) "muster-daemon" in line)
rows = rows.filter(fn(row) not doctor_served(row[2], daemons))
if len(rows) == 0:
print(dim(" none running"))
return
print(dim(" pid age cwd"))
for row in rows:
print(" {row[0]:<7}{row[1]:<13}{row[2]}")
print(dim(" Each ignores its hang-up, so it needs: kill -9 <pid>"))
// Whether a muster-daemon is running on the scratch root a test pane's directory is in: the
// harness gives each daemon a root, `/tmp/muster-test/<root>/`, holding its socket and the home
// its panes start in.
fn doctor_served(cwd, daemons):
root = cwd.split("muster-test/")[1].split("/")[0]
return len(daemons.filter(fn(line) "muster-test/{root}/" in line)) > 0
fn doctor_cwd(pid):
stdout, stderr, code = quiet $`lsof -a -p {pid} -d cwd -Fn` catch:
return ""
for line in stdout.split_lines():
if line.starts_with("n"):
return line[1:]
return ""
// Load says the machine is busy; this says what is eating it.
//
// The two are different questions and a_29iw10a58 needed both answered: eight orphaned busy
// loops at 89% of a core each were invisible in a gate that only ever got slower, and half an
// hour is not obviously wrong for a run that builds two toolchains.
fn doctor_processes():
print(bold("\nbusiest processes"))
// Sorted with `sort` rather than with ps, because the flag that sorts by CPU is spelled
// differently on macOS and Linux and the first column is the same number on both.
stdout, stderr, code = quiet $`ps -Ao pcpu,etime,comm | sort -rn | head -5` catch:
print(dim(" ps would not answer, so nothing here says what is running"))
return
print(dim(" %cpu elapsed command"))
for line in stdout.split_lines():
fields = line.split(" ").filter(fn(field) field != "")
if len(fields) >= 3:
// Reassembled rather than printed as ps laid it out: sorting strips the alignment
// ps had, and a command name is a path or three words depending on the process.
command = join(fields[2:], " ")
print(" {fields[0]:<7}{fields[1]:<13}{command.truncate(76)}")
// The herdr daemons a Muster from before muster-daemon left on this machine.
//
// Built out of the OS rather than out of herdr's API, because the API cannot answer it. You can
// ask a socket what it holds and you can ask the OS which sockets a process has open, and there
// is no method that gets from one to the other - the schema has server.stop, reload_config,
// live_handoff and agent_manifests, and nothing that answers "which process are you"
// (a_28YghIUw2). So this walks pids to sockets and dials each socket back.
//
// The hazard is real and it is the reason this prints a cwd rather than a count: of twenty-odd
// daemons on this machine one held somebody's working agent, and it was neither the oldest nor
// the youngest. Sorting by age picks the wrong process.
// `pgrep -a`, and the flag is the whole of a_2I8fSrotB. Without it pgrep leaves out its own
// ancestors, and this command is run from a Muster pane - whose ancestry is launchd, the app's
// daemon, the pane's shell, rad, pgrep. So the one daemon holding live agents was the one
// process the census structurally could not report, every time somebody ran it where it is
// useful. Measured 2026-09-03: 8 pids without the flag, 9 with, and the extra one held every
// pane in the window the command was typed into.
fn doctor_herdr():
print(bold("\nherdr daemons"))
stdout, stderr, code = quiet $`pgrep -a -x herdr` catch:
stdout = ""
pids = stdout.trim().split_lines().filter(fn(pid) pid.trim() != "")
if len(pids) == 0:
print(dim(" none running"))
return
rows = [doctor_daemon(pid.trim()) for pid in pids]
// Counted apart, because they are different things to somebody deciding what to end. A
// daemon holds sessions; a bridge is one pane's connection to one, and ending it takes that
// pane's picture away and nothing else. A count that added them up read as twenty daemons
// on a machine that had two.
bridges = rows.filter(fn(row) row[1])
print(" {len(rows) - len(bridges)} daemon(s), {len(bridges)} pane bridge(s)")
print(dim(" pid age socket"))
for row in rows:
print(" {row[0]}")
print(dim(" A Muster from before its own daemon started these, and no window of this one shows"))
print(dim(" their panes. End a daemon, and every pane it holds, with: kill $(lsof -t <socket>)"))
// One daemon: how old, which socket, and what is in its panes.
//
// `lsof -a` rather than `lsof`. Without -a, lsof ORs its selection options, so `lsof -p <pid> -U`
// returns every unix socket on the machine and each pid appears to hold whichever one is read
// first. Over twenty daemons that produces twenty identical rows, which reads as a finished
// inventory rather than as a broken one - measured, on this machine, on a_28YghIUw2.
// Answers a row and whether the process is a pane's bridge rather than a daemon.
fn doctor_daemon(pid):
stdout, stderr, code = quiet $`ps -o etime=,args= -p {pid}` catch:
stdout = ""
fields = stdout.trim().split(" ").filter(fn(field) field != "")
// ps knows nothing about it, so it exited between the census and this row. Said outright,
// because the alternative row is "no API socket", which reads as a stray somebody should
// look at.
if len(fields) == 0:
blank = ""
return ["{pid:<7}{blank:<13}gone - it exited while this was being written", false]
age = fields[0]
argv = len(fields) > 1 ? join(fields[1:], " ") : ""
// Named for what it is rather than for what it lacks. Most herdr processes on a machine
// running Muster are these - one per open pane - and each used to draw a row saying "no API
// socket, a client rather than a daemon", which is true and leaves the reader to work out
// that it is a pane's connection and not a stray. Which pane, because that is what makes it
// safe to leave alone.
if "terminal session" in argv:
return ["{pid:<7}{age:<13}a bridge for pane {doctor_bridged_pane(argv)}", true]
stdout, stderr, code = quiet $`lsof -a -p {pid} -U` catch:
stdout = ""
socket = ""
for line in stdout.split_lines():
fields = line.split(" ").filter(fn(field) field != "")
if len(fields) > 0 and fields[-1].ends_with("/herdr.sock"):
socket = fields[-1]
if socket == "":
// A herdr that is neither serving nor bridging. Worth a row rather than a silent
// omission: it is still a process somebody may be about to kill.
return ["{pid:<7}{age:<13}no API socket, and not a pane's bridge either", false]
// A daemon still bound to a socket whose path has been deleted is one nothing can reach
// any more, which is a different fact from one that declined to answer - and it is the
// one that says ending this process costs nobody anything.
if not get_path(socket).exists:
return ["{pid:<7}{age:<13}{socket} unreachable, its socket path is gone", false]
scratch = socket.starts_with("/tmp/") or socket.starts_with("/private/tmp/")
where = scratch ? "scratch" : "kept"
return ["{pid:<7}{age:<13}{socket} {where}", false]
// The pane a bridge is painting, out of the command line that started it.
//
// `herdr terminal session control w6:p30 --cols 93 --rows 34`, so the word after the mode. Read
// positionally because there is no flag naming it, and answered as a question mark rather than
// guessed at when the shape is not that: a wrong pane name is worse than none, since the whole
// use of this row is deciding whether the process belongs to something you are looking at.
fn doctor_bridged_pane(argv):
words = argv.split(" ").filter(fn(word) word != "")
take_next = false
for word in words:
if take_next:
return word
take_next = word == "control" or word == "observe"
return "?"
// The containers the remote tier leaves up, named here because nothing else names them.
//
// Every checkout's rather than only this one's, the way every muster-daemon is listed: the
// question is what is costing this machine, and a worktree that has since been deleted leaves
// its container running with nothing left to stop it from. The prefix also catches the one
// container every checkout shared before each got its own, which carries no checkout label.
fn doctor_devenv():
print(bold("\ndevenv containers"))
stdout, stderr, code = quiet $`docker ps --filter 'name=^muster-devenv' --format '\{\{.Names\}\}\t\{\{.Ports\}\}\t\{\{.Label "muster.checkout"\}\}'` catch:
print(dim(" docker is not answering, so nothing this repo starts in it is up"))
return
// Not trimmed as a whole: the shared container's row ends in the tab before its empty
// label, and trimming the last row would take that field away.
rows = stdout.split_lines().filter(fn(row) row.trim() != "")
if len(rows) == 0:
print(dim(" not running"))
return
root = get_path(".").full_path
orphans = []
for row in rows:
name, ports, checkout = row.split("\t")
if checkout == root:
print(" {name} {ports} this checkout")
else if checkout == "":
print(" {name} {ports} {yellow("shared by every checkout, from before each had its own")}")
orphans += [name]
else if get_path(checkout).exists:
print(" {name} {ports} {checkout}")
else:
print(" {name} {ports} {yellow("{checkout}, which no longer exists")}")
orphans += [name]
print(dim(" ./dev --ssh leaves its checkout's up deliberately - a devenv is a thing you keep."))
print(dim(" Stop it from that checkout with: ./devenv/devenv down"))
for name in orphans:
print(dim(" Nothing will stop {name} for you: docker rm -f {name}"))
fn fail_empty_suite(language, cure):
print_err(red("No {language} tests were discovered."))
print_err("An empty suite passes, so this is a failure rather than a green run.")
print_err(cure)
exit(1)
DAEMON_STARTUP_TEST = "lifecycle::a_spawned_daemon_answers_its_first_request_within_budget"
// What the default gate is expected to skip, and what runs it instead.
//
// A suite that stops running is invisible in a count, which makes it worse than a red one:
// `swift test` reports the same total whether a GUI suite is there and skipping or has been
// deleted, and cargo keeps its ignored tests in a per-binary line nothing reads. So the whole
// of what once said such a suite still existed was one hand-written print - a second
// such suite needed a second line, and the day somebody wrote neither, a suite would stop
// running behind a green gate.
//
// Counted rather than listed test by test, because a count is what moves in both directions
// that matter: a suite deleted takes its skips with it, and an `#[ignore]` on a test that
// should be running adds one. The check prints the names it actually found when the number
// disagrees, so what moved is on screen rather than in somebody's head.
//
// The two languages are asked differently because they answer differently. cargo will say what
// it is going to skip before it runs, so Rust is asked up front. swift will not - its list and
// its run both report 319 whether or not a suite skipped itself, which is the observation this
// card was raised on - so Swift is asked twice instead: discovery says the suite still exists,
// and the run's own skip records say it actually skipped.
//
// The doctest row is the check earning its place on the day it was written: it was drafted
// from a count of `#[ignore]` in the tree, and the runner answered with one more than that.
// A ```ignore example is skipped too, and no grep of the source would have said so.
SKIPPED_SUITES = [
{ "language": "Rust", "suite": "muster-ssh's devenv tests", "skipped": 7,
"note": "need the devenv container; ./dev --ssh runs them" },
{ "language": "Rust", "suite": "muster-daemon-client's devenv test", "skipped": 1,
"note": "needs the devenv container; ./dev --ssh runs it" },
{ "language": "Rust", "suite": "muster-daemon's ssh and sudo devenv tests", "skipped": 2,
"note": "need the devenv container; ./dev --ssh runs them" },
{ "language": "Rust", "suite": "muster-seam's remote handover", "skipped": 1,
"note": "needs the devenv container; ./dev --ssh runs it" },
{ "language": "Rust", "suite": "muster-seam's devenv window: driving it, and an image upload",
"skipped": 2, "note": "need the devenv container; ./dev --ssh runs them" },
{ "language": "Rust", "suite": "muster-seam's laptop and devenv agents sharing a group",
"skipped": 1, "note": "needs the devenv container; ./dev --ssh runs it" },
{ "language": "Rust", "suite": "muster-seam's devenv restarted under a tab interleaving it with the laptop",
"skipped": 1, "note": "needs the devenv container; ./dev --ssh runs it" },
{ "language": "Rust", "suite": "fresh_session's doc example", "skipped": 1,
"note": "an illustration in a doc comment rather than a test, so nothing runs it" },
{ "language": "Rust", "suite": "muster-daemon's resume through a wrapper", "skipped": 1,
"note": "macOS hides a system binary's environment, so MUSTER_RESUME cannot be read off the fake agent; ./dev --linux and CI's linux job run it" },
{ "language": "Rust", "suite": "muster-daemon's echo latency", "skipped": 1,
"note": "a measurement, not a property; cargo test --release -p muster-daemon --test daemon -- flood::echo_latency --ignored --nocapture prints it" },
{ "language": "Rust", "suite": "muster-daemon's first answer", "skipped": 1,
"note": "needs a quiet machine; ./dev --perf runs it" },
{ "language": "Rust", "suite": "muster-daemon's live Claude Code checks", "skipped": 13,
"note": "reach the network and spend model turns; ./dev --claude-code runs them" },
{ "language": "Rust", "suite": "muster-daemon's live Codex checks", "skipped": 9,
"note": "reach the network and spend model turns; ./dev --codex runs them" },
{ "language": "Rust", "suite": "muster-daemon's live OpenCode checks", "skipped": 2,
"note": "reach the network on a free model; ./dev --opencode runs them" },
{ "language": "Rust", "suite": "muster-contract's launch checks", "skipped": 18,
"note": "need a logged-in session; ./dev --contract runs them" },
{ "language": "Swift", "suite": "SurfaceGUITests", "skipped": 25,
"note": "need a logged-in session; ./dev --contract runs them" },
]
// How many tests this language's entries in the manifest account for.
fn expected_skips(language):
total = 0
for row in SKIPPED_SUITES:
if row.language == language:
total = total + row.skipped
return total
// Holds the manifest to what the run actually left out.
//
// `found` is the names when the runner will give them and an empty list when it will only give
// a number, which is the difference between the two languages here rather than a preference.
fn check_skipped(language, skipped, found, cure):
expected = expected_skips(language)
if skipped == expected:
for row in SKIPPED_SUITES:
if row.language == language:
print(dim("skipped: {row.suite} - {row.note}"))
return
print_err(red("The {language} suite left out {skipped} test(s); ./dev's manifest expects {expected}."))
print_err("One of the two is wrong, and until it is settled the gate is reporting on a")
print_err("suite whose size it cannot account for - which is the shape a suite that")
print_err("silently stopped running has.")
if len(found) > 0:
print_err("What it actually left out:")
for name in found:
print_err(" {name}")
print_err(cure)
exit(1)
// Exit 2 means the run could not judge itself - no baseline, no corpus - and exit 1 means
// it judged and the answer was no. Collapsing the two would send someone hunting a
// regression that never happened.
fn fail_perf(stdout, stderr, code):
print(stdout.trim())
if code == 2:
print_err(red("The perf run could not judge itself, so nothing was verified."))
print_err(stderr.trim())
exit(1)
print_err(red("A per-unit budget regressed, or a benchmark stopped running."))
print_err(stderr.trim())
print_err("Either something on a hot path got slower, or the machine is busy - the")
print_err("median column says which. If the cost is deliberate and acceptable:")
print_err(" ./dev --perf --record")
exit(1)
// The remote tier passes over this test on macOS because macOS hides
// the environment of /bin/sh, which its fake agent runs under, so it cannot pass there; the tier
// runs every ignored test to reach the remote ones, and reached this one too. ./dev --linux and
// CI's linux job run it.
MACOS_ONLY_IGNORED = "resume::an_agent_started_through_a_wrapper_comes_back_through_it"
// The remote tier: a real ssh master to a real machine that starts out with nothing on it.
//
// Out of the gate because it needs docker and a container, which is the same line the
// contract tier sits on - not because it is slow or flaky (docs/testing.md, tiered by what
// a tier can reach). The container is the devenv, so what gets tested and what gets
// developed against are one artifact.
//
// The container carries no daemon, which is what a person setting one up actually has, and
// the tests install this build's the way the app does.
fn run_ssh_tier():
// Recreated rather than reused, and that is the point rather than tidiness: the fixture
// these tests need is a machine with no daemon on it, and a container left from an earlier
// run has the one those tests installed.
$`./devenv/devenv down`
$`./devenv/devenv up` catch:
print_err(red("The devenv container would not start, so the remote tier ran nothing."))
print_err("Impact: 'local and remote in one window' is unverified by this run.")
print_err("Check that docker is running, then: ./devenv/devenv up")
exit(1)
// Absolute, because cargo runs an integration test from its own package root and a
// repo-relative key silently becomes 'no such file' and then 'permission denied'.
root = $`pwd`.stdout.trim()
key = "{root}/devenv/.ssh/id_ed25519"
// This checkout's own container, on a port derived from its path, so another worktree's
// run of this tier neither shares nor recreates it. Asked of the script that derives it
// rather than worked out twice.
port = $`./devenv/devenv port`.stdout.trim()
// The container is a local fixture on a port nobody else uses, and its key changes every
// time it is rebuilt, so pinning its host key would mean editing a file per rebuild.
options = "-p {port} -i {key} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
// Spelled out at both call sites rather than built into a variable first: Rad quotes an
// interpolated value as one shell word, so a variable holding the whole prefix arrives as
// a command name nothing can run.
//
// muster-contract is left out because its ignored tests are the contract tier's: they
// launch the app, and need a GUI session rather than a container.
stdout = $`MUSTER_DEVENV_HOST=dev@localhost MUSTER_DEVENV_SSH_OPTIONS={options} cargo test --workspace --exclude muster-contract -- --ignored --list --skip {MACOS_ONLY_IGNORED} --skip {DAEMON_STARTUP_TEST}`
names = stdout.split("\n").filter(fn(line) line.ends_with(": test"))
if len(names) == 0:
print_err(red("The remote tier discovered no tests."))
print_err("An empty tier passes, so this is a failure rather than a green run.")
print_err("Check that the #[ignore] tests in crates/muster-ssh/tests/ssh/,")
print_err("crates/muster-daemon-client/tests/daemon_client/devenv.rs and")
print_err("crates/muster-daemon/tests/daemon/devenv_terminfo.rs still exist.")
exit(1)
print(dim("{len(names)} remote test(s) discovered"))
// --no-fail-fast, because cargo otherwise stops at the first test binary with a failure, and
// the seam's remote tests come after the daemon's: one failure there hid every one of them
// from this tier from 2026-10-28, when MACOS_ONLY_IGNORED landed, until it was skipped.
$`MUSTER_DEVENV_HOST=dev@localhost MUSTER_DEVENV_SSH_OPTIONS={options} cargo test --workspace --exclude muster-contract --no-fail-fast -- --ignored --skip {MACOS_ONLY_IGNORED} --skip {DAEMON_STARTUP_TEST}`
// Said here because this is where it matters and nothing else says it. The container is
// left up deliberately - a devenv is a thing you keep - but it goes on costing the machine
// after the tier that started it has finished, and the tier most likely to run next is the
// one that cannot survive it: a_28UsunMWa measured frame.decode at 1.67 ns/byte with this
// container up against 1.44 with it stopped, both against a 0.83 baseline. Somebody running
// the out-of-gate tiers in sequence gets a perf failure that nothing in the code caused.
print(dim("\nThis checkout's devenv container is still up on port {port}, deliberately - a devenv is a"))
print(dim("thing you keep. It costs this machine something while it runs, and ./dev --perf"))
print(dim("reads slow against its baseline until it stops. Stop it with:"))
print(dim(" ./devenv/devenv down"))
// The daemon's and detection's own suites, run on Linux rather than only compiled for it.
//
// cargo builds each test for the target and hands it to tools/linux-run/run, which runs it in a
// Debian container with this checkout mounted at its own path. Both architectures: the one
// docker runs natively first and in full, then the other under emulation, without the three
// tests whose subject emulation replaces - two timing budgets, which would time the emulator, and
// the live process probe, which reads the emulator's argv (qemu-x86_64 /usr/bin/sleep ...) where
// Linux would give the program's. The native run holds the daemon to all three.
EMULATED_TIMING = "a_spawned_daemon_answers_its_first_request_within_budget"
EMULATED_WRITE_BUDGET = "a_write_that_never_finishes_holds_up_no_request"
EMULATED_ARGV = "the_system_reports_a_real_process_group"
// The emulated run takes three test threads rather than one per CPU docker's VM has. The
// daemon's tests share one binary, so its default runs five at a time, and under emulation on
// a busy Mac a restore of many panes then outlasted the suite's patience: at the default it
// failed twice in three runs, both with the machine's load past 40, and took 113 s on a quiet
// one; at three it passed at load 156 and at load 15 alike, in 153 s both times.
EMULATED_THREADS = 3
fn run_linux_tier():
stdout, stderr, code = quiet $`docker info --format '\{\{.Architecture\}\}'` catch:
print_err(red("docker is not answering, so the Linux tier ran nothing."))
print_err("Impact: the daemon and detection are unverified on Linux by this run.")
print_err("Start Docker Desktop (open -a Docker), then run ./dev --linux again.")
exit(1)
native = stdout.trim() == "x86_64" ? "x86_64-unknown-linux-musl" : "aarch64-unknown-linux-musl"
emulated = LINUX_DAEMONS.keys().filter(fn(target) target != native)
for target in [native] + emulated:
listed = $`cargo test -p muster-daemon -p muster-detect -p muster-msg --target {target} -- --list`
names = listed.split("\n").filter(fn(line) line.ends_with(": test"))
if len(names) == 0:
fail_empty_suite("Linux ({target})", "Check that crates/muster-daemon/tests/ and crates/muster-detect/tests/ still exist.")
if target == native:
print(dim("{len(names)} test(s) on {target}, natively"))
$`cargo test -p muster-daemon -p muster-detect -p muster-msg --target {target} --no-fail-fast`
else:
print(dim("{len(names)} test(s) on {target}, emulated, {EMULATED_THREADS} at a time, less {EMULATED_TIMING}, {EMULATED_WRITE_BUDGET} and {EMULATED_ARGV}"))
$`cargo test -p muster-daemon -p muster-detect -p muster-msg --target {target} --no-fail-fast -- --test-threads {EMULATED_THREADS} --skip {EMULATED_TIMING} --skip {EMULATED_WRITE_BUDGET} --skip {EMULATED_ARGV}`
// The app finds its helpers beside its own executable - right for a shipped bundle, and
// wrong for two build systems with two output directories and a downloaded daemon. Rather
// than teach the app where cargo puts things, the build puts them where the app already
// looks. The awkwardness stays here instead of in the code that has cost the most time
// when it was wrong.
//
// The daemon is one of them: the app starts its own rather than asking somebody to, so a
// build that produced no daemon would produce an app that opens an empty window.
fn stage_helpers(profile):
swift_bin = $`swift build -c {profile} --show-bin-path`.stdout.trim()
stage(target = "target/{profile}/muster-bridge", into = "{swift_bin}/muster-bridge")
// The CLI, staged under a different name than cargo built it under: the app's own executable
// is already `muster`, and the two would collide in one directory. The app puts a link called
// `muster` in ~/.muster/bin pointing here, which is what a pane finds on its PATH.
stage(target = "target/{profile}/muster", into = "{swift_bin}/muster-cli")
// The daemon, and the data it hands every shell, where the app looks for them: beside the
// bridge, as they sit in a bundle.
stage(target = "target/{profile}/muster-daemon", into = "{swift_bin}/muster-daemon")
$`rsync -a --delete {DAEMON_DATA}/ {swift_bin}/muster-daemon-data/`
// The daemons a remote install sends (MIP-3, section 12), laid out as DaemonLocation.swift
// and muster-daemon-client's `install` expect them: one directory per Linux build.
for target in LINUX_DAEMONS.keys():
carried = "{swift_bin}/daemons/{carried_name(target)}"
$`mkdir -p {carried}`
stage(target = "target/{target}/{profile}/muster-daemon", into = "{carried}/muster-daemon")
stage(target = "target/{target}/{profile}/muster", into = "{carried}/muster")
// Each harness's adapter, where the CLI looks beside itself (`muster harness install`) and
// where a remote install takes them from: the same place relative to `muster-cli` as in a
// bundle's Resources.
$`rsync -a --delete extras/ {swift_bin}/extras/`
// What the app calls a Linux build it carries: `linux-x86_64` for x86_64-unknown-linux-musl.
fn carried_name(target):
return "linux-{target.split("-")[0]}"
// Copies a binary into place without disturbing anyone running the old one.
//
// Via a temporary name and a rename, because a plain `cp` writes through the file the
// running process has mapped: macOS invalidates its pages, SIGKILLs it, and then SIGKILLs
// every later exec of the path until the signature is re-validated. That is not theoretical
// here - the app starts a daemon from this directory and the whole point of it is to outlive
// the app, so a rebuild would otherwise kill the agents somebody is running. A rename swaps
// the directory entry and leaves the old inode alone.
fn stage(target, into):
$`cp {target} {into}.staging && mv -f {into}.staging {into}`
// Assembles the built binary into something the platform can launch.
//
// SwiftPM has no bundle product, so this is hand-rolled, and it is out of the gate because
// nothing in the gate needs one. What it buys is everything a bare binary in .build cannot
// be: a Dock icon, a double-click, `open -a`, and the launchd descriptor ceiling that only
// a GUI-launched process ever meets (DescriptorLimit.swift).
//
// Signing and notarization are a separate question and belong with going public. This is
// about launching it like an app on the machine that built it.
fn build_bundle():
swift_bin = $`swift build -c release --show-bin-path`.stdout.trim()
contents = "{BUNDLE}/Contents"
// Assembled from nothing every time. A bundle that accumulated across builds could
// carry a stale dylib beside a fresh binary, which fails at launch with a dyld error
// naming a symbol rather than the build that produced it.
$`rm -rf {BUNDLE}`
$`mkdir -p {contents}/MacOS {contents}/Frameworks {contents}/Resources`
// Helpers in MacOS/ rather than Resources/, because PaneCommand.bridge resolves them
// beside the running executable and one rule has to cover the dev build and the bundle.
$`cp {swift_bin}/muster {contents}/MacOS/muster`
$`cp {swift_bin}/muster-bridge {contents}/MacOS/muster-bridge`
$`cp {swift_bin}/muster-cli {contents}/MacOS/muster-cli`
$`cp target/release/libmuster.dylib {contents}/Frameworks/libmuster.dylib`
// Resources rather than Library, because a Linux ELF is not code to codesign: it is sealed
// as a resource like the icon, and is only ever run on another machine.
$`rsync -a --copy-links {swift_bin}/daemons/ {contents}/Resources/daemons/`
// Each harness's adapter, which `muster harness install` hands the harness by this path, so
// it is the installed app's own and survives an update in place.
$`rsync -a extras/ {contents}/Resources/extras/`
// muster-daemon's agent detection is a port of herdr's, and Apache-2.0 asks that its terms
// travel with the binary. The copy is checked in rather than downloaded so that assembling
// a bundle stays offline and so the text cannot silently stop matching what ships.
$`cp licenses/herdr/LICENSE {contents}/Resources/LICENSE-herdr.txt`
$`cp NOTICE {contents}/Resources/NOTICE.txt`
// The Rust crates and swift-protobuf are linked into the binaries above rather than
// copied beside them, so nothing else in this directory tells you they are here. The
// second file is the same problem one layer down: libghostty is MIT, and the archive it
// links in carries ten other projects' object code with it, measured rather than assumed.
$`cp {THIRD_PARTY} {contents}/Resources/THIRD-PARTY.md`
$`cp {GHOSTTY_LICENSES} {contents}/Resources/GHOSTTY-EMBEDDED.md`
// Rendered from packaging/icon.svg and committed, so a bundle needs no image toolchain.
// tools/make-icns.sh is what regenerates it.
$`cp {ICNS} {contents}/Resources/Muster.icns`
write_info_plist(contents)
write_daemon_bundle(contents, swift_bin)
// The app is the one binary here that loads a dylib of ours, libmuster: libghostty-vt is
// linked statically into everything that uses it. Its compiled-in rpaths are relative to
// target/release (Package.swift), which does not exist inside a bundle, so it learns where
// the bundle keeps libmuster instead, and is re-signed afterwards: editing a Mach-O
// invalidates its signature, and macOS refuses to launch the result.
daemon = "{contents}/Library/{DAEMON_BUNDLE}/Contents/MacOS/muster-daemon"
drop_checkout_rpaths("{contents}/MacOS/muster")
$`install_name_tool -add_rpath @executable_path/../Frameworks {contents}/MacOS/muster`
// Before signing, which touches most of the same paths and would otherwise report a missing
// daemon as a codesign error naming a file - true, and several steps away from what to do
// about it.
check_bundle_is_whole(contents)
sign_bundle(swift_bin, contents)
// Run once, because a file being in place says nothing about whether it loads: a daemon
// that cannot load what it links is present, signed and dead at launch, and a window
// shows that only as panes that never render. It links nothing of this checkout's, so
// this is the bundle's own daemon starting, even on the machine that built it.
said = quiet $`{daemon} --version`.stdout.trim() catch:
print_err(red("The bundled muster-daemon does not start."))
print_err("Impact: every pane in this bundle's window stays blank.")
print_err("Check: run {daemon} --version and read what dyld says it could not load;")
print_err("a dylib it was linked against that the bundle does not carry is the usual cause.")
exit(1)
print("assembled {BUNDLE} ({said}) - open it, or drag it to the Dock")
// The install names the daemon's socket, and a bundle put in place of an installed Muster of
// another install starts an empty daemon beside the one holding its agents. Said here because
// this is the last moment anyone is looking. Unset is the checkout's own install, as
// crates/muster-daemon-proto/build.rs decides.
if get_env("MUSTER_INSTALL") == "":
print(dim("Built for this checkout's own install, so it runs a daemon of its own beside any"))
print(dim("installed Muster's. To replace the installed Muster, use tools/install-main."))
// Deletes the rpaths that point into this checkout, so a bundled binary loads only what the
// bundle carries. Left in, the binary would load from target/ on the machine that built it, and
// a bundle missing a dylib would pass every check here and fail on anybody else's.
fn drop_checkout_rpaths(binary):
listing, _, _ = quiet $`otool -l {binary}`
for line in listing.split("\n"):
text = line.trim()
if not text.starts_with("path "):
continue
path = text[5:].split(" (offset")[0]
if "target/" in path or "deps/ghostty" in path:
$`install_name_tool -delete_rpath {path} {binary}`
// Everything a launched bundle has to be able to reach, checked at the moment it is assembled.
//
// **This would not have caught `1d7ace3`, and saying so is the point.** That commit moved the
// daemon into the helper bundle and dropped the copy that had been going into `Contents/MacOS/`,
// and the bundle it produced was correct - what was wrong was the bridge, which went on looking
// beside itself. A layout check cannot see that. `./dev --contract` launches an assembled bundle
// now, and that is the check that would have (kan a_2Hnh3g0Y5).
//
// What this guards is the failure mode `1d7ace3` demonstrated is possible: a copy quietly leaving
// `build_bundle`. Every path below was written a few lines above, so nothing here fails while the
// script is intact - it fails when somebody deletes a line, which is what happened. Signing
// catches the daemon's absence today as a side effect of signing that exact path; this says the
// requirement outright, so a signing change cannot take the guard with it.
fn check_bundle_is_whole(contents):
daemon = "{contents}/Library/{DAEMON_BUNDLE}/Contents/MacOS/muster-daemon"
required = [
// The app, the CLI a pane finds on its PATH, and the bridge every pane's frames arrive
// through. PaneCommand.bridge resolves the last two beside the running executable.
"{contents}/MacOS/muster",
"{contents}/MacOS/muster-bridge",
"{contents}/MacOS/muster-cli",
// The daemon and what it gives every shell, where DaemonLocation.swift looks for them.
// A bundle without the daemon opens a window in which no pane renders.
daemon,
"{contents}/Library/{DAEMON_BUNDLE}/Contents/Resources/muster-daemon-data/README.md",
// What a remote install sends a Linux machine. Without them a devenv with no daemon
// on it shows no panes, and says the app carries none.
"{contents}/Resources/daemons/linux-x86_64/muster-daemon",
"{contents}/Resources/daemons/linux-aarch64/muster-daemon",
// And the CLI the same install puts on that machine, without which an agent there has
// no `muster msg` to message anyone with.
"{contents}/Resources/daemons/linux-x86_64/muster",
"{contents}/Resources/daemons/linux-aarch64/muster",
// The adapters `muster harness install` installs, here and on every SSH machine.
"{contents}/Resources/extras/.claude-plugin/marketplace.json",
"{contents}/Resources/extras/.agents/plugins/marketplace.json",
"{contents}/Frameworks/libmuster.dylib",
"{contents}/Info.plist",
"{contents}/Library/{DAEMON_BUNDLE}/Contents/Info.plist",
]
missing = required.filter(fn(path) not get_path(path).exists)
if len(missing) > 0:
print_err(red("The assembled bundle is missing {len(missing)} file(s) it needs to run."))
for path in missing:
print_err(" {path}")
print_err("Impact: a launched Muster fails at whatever needs the missing file, and a")
print_err("missing daemon in particular opens a window where every pane renders nothing")
print_err("and says only that muster-daemon could not be started.")
print_err("Usual cause: a step in build_bundle stopped copying something, or a path in")
print_err("it moved without the rule in Sources/MusterMac that resolves it moving too.")
exit(1)
// The daemon, as a bundle Launch Services can start.
//
// muster-daemon is a plain command-line binary and this makes an application out of it, which
// sounds like a costume and is the whole mechanism. macOS decides who to charge a pane's
// protected request to when that pane is spawned: a spawned child inherits its spawner's
// answer, and only a process Launch Services started is its own. Spawned, the daemon is charged
// to Muster until Muster exits and then to nothing that can be named; opened, it is charged to
// this bundle for as long as it lives, and every pane it ever makes with it.
//
// The id is the one it has always had, so the grants people gave it stay attached to what they
// are charged to.
//
// The Info.plist below carries the same usage descriptions the app's does. macOS reads a
// usage string from the *responsible* process's bundle, and once the daemon is opened rather
// than spawned that is this one - a missing string is not a worse prompt but a kill.
//
// Its data sits in Resources rather than beside the executable, because a code signature
// treats everything in MacOS/ as code; the app tells the daemon where it is.
//
// LSUIElement rather than LSBackgroundOnly. Both keep it out of the Dock and the app switcher;
// the difference is that a UIElement application may put something on screen, and a permission
// prompt is something on screen. A background-only daemon that could never present one would
// turn every un-granted permission into a silent denial, which is the failure this whole card
// is about.
//
// NSAppSleepDisabled, because an application with no window of its own is what App Nap exists
// to slow down, and it would coalesce the daemon's timers and lower its priority while somebody
// types into one of its panes.