Skip to content

Commit 72c394e

Browse files
Kyu-seokclaude
andauthored
dev-branch-servers: orphans finds Kafka and Redis leftovers (#418) (#423)
* dev-branch-servers: orphans finds Kafka and Redis leftovers (#418) A drop with codebite-kafka or codebite-redis stopped drops the database and leaves the topics, groups and claimed Redis index, which no later drop found. orphans now also lists, under the database name, each Kafka topic and group and each Redis index 1-15 whose name or owner key gives a branch database that no longer exists and is no live worktree's; drop <name> deletes them, and a rerun of plain drop in the worktree finds its own. The skip lines end with that command. kafka_owner is the one rule for a topic or group: the text before its first dash must have branch_db_name's shape (codebite_, a-z0-9 runs, _ and 8 hex digits). The shared servers' names (submission-events, codebite-worker, codebite-sse-<uuid>) never do, and a database whose name is a prefix of another's no longer matches the other's topics. drop <name> matches the name literally, no longer as a grep pattern. With Postgres or the Docker daemon down, orphans and drop say so in one line and exit 1, where they printed a docker error and then "none"; require_postgres names the compose service it starts. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * ship-branch: a drop that skips Kafka or Redis is no failure (#418) The comment above drop_branch_database said only orphans finds what a drop leaves; since #418 that is true for a skipped Kafka or Redis too. Such a drop exits 0 and prints its own line naming what it left and the drop <name> that deletes it, so ship-branch adds no warning: "couldn't drop the branch database" would be false, the database is dropped. A test pins that the line passes through with no warning. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> * dev-branch-servers: a failed lookup or odd name deletes nothing (#418) Review of #418. A database lookup that failed after pg_isready passed was read as "no database exists", so orphans listed an existing database's topics and Redis index and drop deleted them; plain drop read its own failed lookup as "none" the same way. Both now exit. drop <name> checks the name's shape before matching it, since grep took a name of several lines as several names. orphans with Kafka or Redis stopped and nothing found no longer says nothing was left there. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
1 parent 35537db commit 72c394e

4 files changed

Lines changed: 522 additions & 67 deletions

File tree

‎scripts/dev-branch-servers.sh‎

Lines changed: 189 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,10 @@
1414
# migrations on it, starting no server: for reading a problem
1515
# (CODEBITE_DB=<it> scripts/problem-text.sh) as the branch has it
1616
# scripts/dev-branch-servers.sh drop drop the branch database, Kafka topics and groups, Redis index
17-
# (a Kafka or Redis container that isn't running: says what is left behind, skips it)
18-
# scripts/dev-branch-servers.sh orphans list the branch databases whose worktree is gone
17+
# (a Kafka or Redis container that isn't running: says what is left behind
18+
# and the command that drops it later, skips it)
19+
# scripts/dev-branch-servers.sh orphans list the branch databases whose worktree is gone, and the dropped ones
20+
# whose Kafka topics and groups or Redis index a skipped drop left behind
1921
# scripts/dev-branch-servers.sh drop [<name>] with a name, drop one of those, from any checkout
2022
# scripts/dev-branch-servers.sh name print the branch database name
2123
#
@@ -24,8 +26,9 @@
2426
# database codebite_<branch>_<hash of this worktree's path> in the codebite-postgres container,
2527
# migrated by the backend on start (or by migrate, with no server); its COMMENT names this worktree, and drop refuses a
2628
# database whose comment names another
27-
# Kafka topics and consumer groups prefixed with that name (app.kafka.topic.*, app.kafka.group-prefix),
28-
# so neither side takes a share of the other's submissions or results
29+
# Kafka topics and consumer groups prefixed with that name and a dash (app.kafka.topic.*,
30+
# app.kafka.group-prefix), so neither side takes a share of the other's submissions or
31+
# results; the shared servers' own names never start with a branch database name (kafka_owner)
2932
# Redis a database index of its own (a fresh database's user ids collide with the shared one's),
3033
# claimed by an owner key holding the database name; drop flushes only that index, never 0
3134
# processes only those started with CODEBITE_DEV_BRANCH=<this database> are ever stopped
@@ -61,6 +64,31 @@ branch_db_name() {
6164
printf 'codebite_%s_%s\n' "$s" "$hash"
6265
}
6366

67+
# The branch database of the checkout at $1 (its top level): named for its branch, or for its
68+
# directory when detached.
69+
checkout_database() {
70+
local branch
71+
branch=$(git -C "$1" branch --show-current)
72+
branch_db_name "${branch:-$(basename "$1")}" "$1"
73+
}
74+
75+
# Whether $1 has the shape branch_db_name gives: codebite_, runs of a-z0-9 joined by single
76+
# underscores, _ and 8 hex digits, at most 63 characters. The letters are spelled out, since a
77+
# range like a-z can take in other characters in some locales.
78+
is_branch_db_name() {
79+
local w=abcdefghijklmnopqrstuvwxyz0123456789
80+
local pattern="^codebite_[$w]+(_[$w]+)*_[0123456789abcdef]{8}\$"
81+
[[ ${#1} -le 63 && "$1" =~ $pattern ]]
82+
}
83+
84+
# The branch database whose Kafka topic or consumer group $1 is (start names each <database>-<name>,
85+
# and a database name has no dash); empty for any other name. The shared servers' own have no
86+
# branch database name before their first dash: submission-events, codebite-worker.
87+
kafka_owner() {
88+
local db=${1%%-*}
89+
if [[ "$1" == "$db"-?* ]] && is_branch_db_name "$db"; then echo "$db"; fi
90+
}
91+
6492
# The COMMENT a branch database carries: the worktree that made it.
6593
db_owner_comment() {
6694
printf 'codebite-dev-branch:%s' "$1"
@@ -303,24 +331,44 @@ database_state() {
303331
psql_admin "SELECT 'x' || coalesce(shobj_description(oid, 'pg_database'), '') FROM pg_database WHERE datname = '$db'"
304332
}
305333

334+
# Every database's name. A lookup that fails, or lacks the postgres database every server has,
335+
# exits: read as "none exist", it would make each existing branch database's topics and Redis
336+
# index look left behind, for drop <name> to delete.
337+
existing_databases() {
338+
local names
339+
if ! names=$(psql_admin "SELECT datname FROM pg_database") || ! grep -qxF postgres <<<"$names"; then
340+
echo "dev-branch-servers.sh: couldn't list the databases in codebite-postgres; rerun" >&2
341+
exit 1
342+
fi
343+
echo "$names"
344+
}
345+
306346
kafka_bin() {
307347
local tool=$1
308348
shift
309349
docker exec codebite-kafka "/opt/kafka/bin/$tool" --bootstrap-server localhost:9092 "$@"
310350
}
311351

352+
# The topics (kafka-topics.sh as $1) or consumer groups (kafka-consumer-groups.sh) of branch
353+
# database $2, by kafka_owner.
354+
kafka_names_of() {
355+
local name owner
356+
{ kafka_bin "$1" --list 2>/dev/null || true; } | while read -r name; do
357+
owner=$(kafka_owner "$name")
358+
if [[ -n "$owner" && "$owner" == "$2" ]]; then echo "$name"; fi
359+
done
360+
}
361+
312362
# Sets the globals the subcommands read: root (this checkout), main (the main checkout), db (the
313363
# branch database), prefix (its Kafka topic and group prefix), state (the ports, pids and logs)
314364
# port_locks (every start's port claims) and state_base (every branch database's state).
315365
setup() {
316366
root=$(git rev-parse --show-toplevel)
317367
main=$(main_checkout)
318-
local branch
319-
branch=$(git -C "$root" branch --show-current)
320368
state_base="${TMPDIR:-/tmp}"
321369
state_base="${state_base%/}/codebite-dev-branch"
322370
port_locks="$state_base/.ports"
323-
use_database "$(branch_db_name "${branch:-$(basename "$root")}" "$root")"
371+
use_database "$(checkout_database "$root")"
324372
}
325373

326374
# Sets db to branch database $1, and the globals named for it: prefix and state.
@@ -410,9 +458,16 @@ stop_all() {
410458
return 0
411459
}
412460

461+
# Exits, saying in one line what to start or wait for, unless Postgres in codebite-postgres answers.
413462
require_postgres() {
414-
docker exec codebite-postgres pg_isready -q -U codebite \
415-
|| { echo "codebite-postgres is not running: docker compose -f infra/docker-compose.yml up -d" >&2; exit 1; }
463+
local down
464+
down=$(db_down_reason)
465+
if [[ -z "$down" ]]; then
466+
docker exec codebite-postgres pg_isready -q -U codebite && return 0
467+
down="codebite-postgres runs but doesn't accept connections yet; rerun in a few seconds"
468+
fi
469+
echo "dev-branch-servers.sh: $down" >&2
470+
exit 1
416471
}
417472

418473
# Exits, saying what to start, unless the Kafka and the Redis container both run; a Docker daemon
@@ -627,13 +682,19 @@ cmd_drop() {
627682
echo "the branch servers are running; stop them first: scripts/dev-branch-servers.sh stop" >&2
628683
exit 1
629684
fi
630-
local comment
685+
require_postgres
686+
local comment found
631687
comment=$(db_owner_comment "$root")
632-
case "$(database_verdict "$(database_state)" "$comment")" in
688+
# A failed lookup is not "none": that would delete the topics and index of a database still there.
689+
if ! found=$(database_state); then
690+
echo "dev-branch-servers.sh: couldn't look up database $db in codebite-postgres; rerun" >&2
691+
exit 1
692+
fi
693+
case "$(database_verdict "$found" "$comment")" in
633694
create)
634695
# Nothing was ever started here (start makes the state directory, start and migrate the
635-
# database): nothing to look for in Kafka or Redis.
636-
if [[ ! -d "$state" ]]; then
696+
# database), or an earlier drop skipped a stopped Kafka or Redis: only that is looked for.
697+
if [[ ! -d "$state" && -z "$(leftovers_of_db)" ]]; then
637698
echo "database $db: none; nothing to drop"
638699
return 0
639700
fi
@@ -643,7 +704,16 @@ cmd_drop() {
643704
echo "database $db isn't this worktree's (its comment isn't '$comment'); dropping nothing" >&2
644705
exit 1 ;;
645706
esac
646-
drop_branch_data
707+
drop_branch_data "scripts/dev-branch-servers.sh drop in this worktree, or drop $db once the worktree is gone"
708+
}
709+
710+
# $db's Kafka topics and groups and Redis index, in whichever of the two containers runs.
711+
leftovers_of_db() {
712+
if [[ "$(container_state codebite-kafka)" == running ]]; then
713+
kafka_names_of kafka-topics.sh "$db"
714+
kafka_names_of kafka-consumer-groups.sh "$db"
715+
fi
716+
if [[ "$(container_state codebite-redis)" == running ]]; then owned_redis_index; fi
647717
}
648718

649719
drop_database() {
@@ -653,22 +723,23 @@ drop_database() {
653723

654724
# Deletes $db's Kafka topics and groups, flushes its Redis index and removes its state and port
655725
# claims: everything but the database. A container that doesn't run is asked once and its cleanup
656-
# skipped with one line, so the rest still happens and the drop still succeeds.
726+
# skipped with one line ending in $1, the command that cleans up later, so the rest still happens
727+
# and the drop still succeeds.
657728
drop_branch_data() {
658-
local topics groups g
659-
if container_runs_or_says codebite-kafka "topics and groups prefixed ${prefix%-} left behind, and no later drop finds them"; then
660-
topics=$(kafka_bin kafka-topics.sh --list | grep "^$prefix" || true)
729+
local later=$1 topics groups g
730+
if container_runs_or_says codebite-kafka "topics and groups prefixed ${prefix%-} left behind; once it runs: $later"; then
731+
topics=$(kafka_names_of kafka-topics.sh "$db")
661732
if [[ -n "$topics" ]]; then
662733
kafka_bin kafka-topics.sh --delete --topic "$(echo "$topics" | paste -sd, -)" >/dev/null
663734
echo "Kafka topics: deleted $(echo "$topics" | paste -sd' ' -)"
664735
fi
665-
groups=$(kafka_bin kafka-consumer-groups.sh --list 2>/dev/null | grep "^$prefix" || true)
736+
groups=$(kafka_names_of kafka-consumer-groups.sh "$db")
666737
for g in $groups; do
667738
kafka_bin kafka-consumer-groups.sh --delete --group "$g" >/dev/null 2>&1 || true
668739
done
669740
if [[ -n "$groups" ]]; then echo "Kafka groups: deleted $(echo "$groups" | paste -sd' ' -)"; fi
670741
fi
671-
if container_runs_or_says codebite-redis "the Redis index of $db left claimed, and no later drop finds it"; then
742+
if container_runs_or_says codebite-redis "the Redis index of $db left claimed; once it runs: $later"; then
672743
flush_owned_redis_index
673744
fi
674745
release_ports
@@ -692,26 +763,108 @@ orphan_databases() {
692763
return 0
693764
}
694765

766+
# The branch database of each worktree of this repository whose directory is still there (rev-parse fails
767+
# in a removed one).
768+
live_worktree_databases() {
769+
local key path top
770+
git -C "$main" worktree list --porcelain | while read -r key path; do
771+
[[ "$key" == worktree ]] || continue
772+
top=$(git -C "$path" rev-parse --show-toplevel 2>/dev/null) || continue
773+
checkout_database "$top"
774+
done
775+
}
776+
777+
# What branch databases that are gone left behind (a drop skips a stopped Kafka or Redis): one
778+
# "<database>\ttopic", "<database>\tgroup" or "<database>\tredis <index>" line for each Kafka
779+
# topic, consumer group and Redis index 1-15 whose name (kafka_owner) or owner key names a branch
780+
# database that doesn't exist and is no live worktree's. Looks only in the containers that run.
781+
orphan_leftovers() {
782+
local known name what i owner
783+
known=$(existing_databases) || exit 1
784+
known+=$'\n'$(live_worktree_databases)
785+
{
786+
if [[ "$(container_state codebite-kafka)" == running ]]; then
787+
for what in topic:kafka-topics.sh group:kafka-consumer-groups.sh; do
788+
{ kafka_bin "${what#*:}" --list 2>/dev/null || true; } | while read -r name; do
789+
owner=$(kafka_owner "$name")
790+
if [[ -n "$owner" ]]; then printf '%s\t%s\n' "$owner" "${what%%:*}"; fi
791+
done
792+
done
793+
fi
794+
if [[ "$(container_state codebite-redis)" == running ]]; then
795+
for (( i = 1; i <= 15; i++ )); do
796+
owner=$(redis_cmd "$i" GET "$REDIS_OWNER_KEY")
797+
if is_branch_db_name "$owner"; then printf '%s\tredis %s\n' "$owner" "$i"; fi
798+
done
799+
fi
800+
} | while IFS=$'\t' read -r name what; do
801+
grep -qxF -- "$name" <<<"$known" || printf '%s\t%s\n' "$name" "$what"
802+
done
803+
}
804+
695805
cmd_orphans() {
696-
local list
806+
require_postgres
807+
local list leftovers down
697808
list=$(orphan_databases)
698-
if [[ -z "$list" ]]; then
699-
echo "no branch database outlives its worktree"
700-
return 0
809+
leftovers=$(orphan_leftovers) || exit 1
810+
if [[ -z "$list$leftovers" ]]; then
811+
if [[ -z "$(container_down_reason codebite-kafka)$(container_down_reason codebite-redis)" ]]; then
812+
echo "no branch database outlives its worktree, and no dropped one left Kafka topics or a Redis index"
813+
else
814+
echo "nothing found: no branch database outlives its worktree, and no dropped one left what could be looked at"
815+
fi
816+
else
817+
if [[ -n "$list" ]]; then
818+
echo "branch databases whose worktree is gone:"
819+
printf '%s\n' "$list" | while IFS=$'\t' read -r name path; do
820+
echo " $name (was $path)"
821+
done
822+
fi
823+
if [[ -n "$leftovers" ]]; then
824+
echo "branch databases already dropped whose Kafka topics or Redis index are left:"
825+
printf '%s\n' "$leftovers" | LC_ALL=C sort | awk -F'\t' '
826+
function many(n, what) { return n " " what (n > 1 ? "s" : "") }
827+
function flush( s) {
828+
if (db == "") return
829+
if (t) s = many(t, "Kafka topic")
830+
if (g) s = s (s == "" ? "" : ", ") many(g, "Kafka group")
831+
if (r != "") s = s (s == "" ? "" : ", ") "Redis index " r
832+
print " " db " (" s ")"
833+
}
834+
$1 != db { flush(); db = $1; t = 0; g = 0; r = "" }
835+
$2 == "topic" { t++ }
836+
$2 == "group" { g++ }
837+
$2 ~ /^redis / { r = r (r == "" ? "" : ", ") substr($2, 7) }
838+
END { flush() }'
839+
fi
840+
echo "drop one, with its Kafka topics and Redis index: scripts/dev-branch-servers.sh drop <name>"
701841
fi
702-
echo "branch databases whose worktree is gone:"
703-
printf '%s\n' "$list" | while IFS=$'\t' read -r name path; do
704-
echo " $name (was $path)"
705-
done
706-
echo "drop one, with its Kafka topics and Redis index: scripts/dev-branch-servers.sh drop <name>"
842+
down=$(container_down_reason codebite-kafka)
843+
if [[ -n "$down" ]]; then echo "Kafka topics and groups not looked at: $down"; fi
844+
down=$(container_down_reason codebite-redis)
845+
if [[ -n "$down" ]]; then echo "Redis indexes not looked at: $down"; fi
707846
}
708847

709-
# Drops branch database $1 by name, from any checkout, when its worktree is gone; a live
710-
# worktree's is dropped from that worktree, and any other database is never touched.
848+
# Drops branch database $1 by name, from any checkout, when its worktree is gone, or what it left
849+
# in Kafka and Redis when it is gone too; a live worktree's is dropped from that worktree, and
850+
# any other database or name is never touched.
711851
cmd_drop_orphan() {
712-
local name=$1
713-
if ! orphan_databases | cut -f1 | grep -qx -- "$name"; then
714-
echo "$name isn't a branch database whose worktree is gone (scripts/dev-branch-servers.sh orphans lists those); dropping nothing" >&2
852+
local name=$1 listed=false gone=false leftovers
853+
require_postgres
854+
# The shape comes first: grep reads a name of several lines as several names, any one of which
855+
# may match. Lists are captured, not piped to grep -q: under pipefail its early exit can fail
856+
# the pipe it ends, and a lookup that exits inside a pipe wouldn't end the drop.
857+
if is_branch_db_name "$name"; then
858+
if grep -qxF -- "$name" <<<"$(orphan_databases | cut -f1)"; then
859+
listed=true
860+
else
861+
leftovers=$(orphan_leftovers) || exit 1
862+
if grep -qxF -- "$name" <<<"$(cut -f1 <<<"$leftovers")"; then listed=true gone=true; fi
863+
fi
864+
fi
865+
if ! $listed; then
866+
echo "$name isn't a branch database whose worktree is gone, nor one already dropped that left Kafka topics or a Redis index" \
867+
"(scripts/dev-branch-servers.sh orphans lists both); dropping nothing" >&2
715868
exit 1
716869
fi
717870
use_database "$name"
@@ -721,8 +874,8 @@ cmd_drop_orphan() {
721874
echo "servers of $db still run (pids $(branch_pids | paste -sd' ' -)); dropping nothing" >&2
722875
exit 1
723876
fi
724-
drop_database
725-
drop_branch_data
877+
if $gone; then echo "database $db: none (dropped before)"; else drop_database; fi
878+
drop_branch_data "scripts/dev-branch-servers.sh drop $db"
726879
}
727880

728881
usage() {

0 commit comments

Comments
 (0)