Skip to content

add a psql subcommand to cncluster script for convenient DB inspection #2672

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mblaze-da merged 4 commits into from
Oct 16, 2025
Merged

add a psql subcommand to cncluster script for convenient DB inspection #2672

Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
031e074
add a psql subcommand to cncluster script for convenient DB inspection
mblaze-da Oct 13, 2025
b3f8ac9
fix a hardcoded namespace in cncluster psql
mblaze-da Oct 13, 2025
7a1651b
correct the DB name resolution in the cncluster psql command
mblaze-da Oct 14, 2025
be7010b
infer the schema name for search_path in cncluster psql subcommand ba…
mblaze-da Oct 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 135 additions & 1 deletion build-tools/cncluster
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1458,6 +1458,8 @@ function subcmd_debug_shell() {
fi
repo="ghcr.io/digital-asset/decentralized-canton-sync-dev/docker"

local SHELL_COMMAND="${1:-/bin/bash}"

_info "Deploying debug pod on version $VERSION_NUMBER"
kubectl apply --wait -f - <<EOF
apiVersion: v1
Expand All @@ -1479,10 +1481,11 @@ spec:
effect: "NoSchedule"
EOF

# shellcheck disable=SC2086
Copy link
Contributor

@julientinguely-da julientinguely-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's try to remove the shellcheck disable=SC2086 by adding the parentheses (like "$SHELL_COMMAND")

Copy link
Contributor Author

@mblaze-da mblaze-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then it would not work. We need this to expand.

( _info "Waiting for debug pod to become ready" && \
kubectl wait --for=condition=Ready pod splice-debug --timeout=30s && \
_info "Opening terminal on debug pod" && \
kubectl exec -it splice-debug -- /bin/bash ) || true
kubectl exec -it splice-debug -- $SHELL_COMMAND ) || true

_info "Deleting debug pod"
kubectl delete pod splice-debug
Expand Down Expand Up @@ -2172,6 +2175,137 @@ EOF
_update_cluster_config "$deployment_config" "operatorDeployment"
}

###

subcommand_whitelist[psql]=""
Copy link
Contributor

@julientinguely-da julientinguely-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: could we add the ### on top to match with the other methods

function subcmd_psql() {
_cluster_must_exist

if [ $# -lt 2 ] || [ $# -gt 3 ]; then
_error "Usage: $SCRIPTNAME psql <namespace> <application> [search_path]"
exit 1
fi

local NAMESPACE="$1"
local APPLICATION="$2"
local SEARCH_PATH=${3:-}

_info "Retrieving DB connection info from pod description..."
local DB_INIT_COMMAND
DB_INIT_COMMAND=$(
kubectl describe pod \
Copy link
Contributor

@julientinguely-da julientinguely-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: couldn't we use kubectl get pod instead of describe here, would seem less brittle?

Copy link
Contributor Author

@mblaze-da mblaze-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it would make it much more reliable. We'd still need to sed the psql command out of a larger script in the "command" field. We'd just exchange a grep -i psql for a more complicated JSON path expression.

--namespace "${NAMESPACE}" \
--selector "app=${APPLICATION}" \
| grep -i psql \
| sed -nE 's/^.*(psql.?*) 2>&1.*$/\1/p'
)

if [ -z "${DB_INIT_COMMAND}" ]; then
_error "Application ${APPLICATION} in namespace ${NAMESPACE} does not have an associated database."
exit 1
fi

# shellcheck disable=SC2086
set -- $DB_INIT_COMMAND
while [ $# -gt 0 ]; do
case "$1" in
-h|--host)
local DB_HOST="$2"
shift 2
;;
-h=*|--host=*)
local DB_HOST="${1#*=}"
shift
;;
-p|--port)
local DB_PORT="$2"
shift 2
;;
-p=*|--port=*)
local DB_PORT="${1#*=}"
shift
;;
-U|--username)
local DB_USERNAME="$2"
shift 2
;;
-U=*|--username=*)
local DB_USERNAME="${1#*=}"
shift
;;
-d|--dbname)
local DB_NAME="$2"
shift 2
;;
-d=*|--dbname=*)
local DB_NAME="${1#*=}"
Copy link
Contributor

@martinflorian-da martinflorian-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK so at least on the deployment I tested on (sv-app) this is wrong: we don't want the default db (cantonnet), but the one that the init container is creating sv_sv_1:

until errmsg=$(psql -h 10.60.65.197 -p 5432 --username=cnadmin --dbname=cantonnet -c 'create database sv_sv_1' 2\u003e\u00261); do\n if [[ $errmsg == *\"already exists\"* ]]; then\n echo \"Database sv_sv_1 already exists. Done.\"\n break\n fi\n\n echo \"trying to create postgres database sv_sv_1, last error: $errmsg\";\n sleep 2;\ndone\n"

Copy link
Contributor

@martinflorian-da martinflorian-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(sorry for the horrible paste, but I hope the point comes across; create database sv_sv_1 is the interesting one)

Copy link
Contributor Author

@mblaze-da mblaze-da Oct 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

shift
;;
*)
shift
;;
esac
done

if [ -z "${DB_HOST:-}" ]; then
_error "Failed to retrieve DB hostname from pod description."
exit 1
fi

if [ -z "${DB_PORT:-}" ]; then
_error "Failed to retrieve DB port from pod description."
exit 1
fi

if [ -z "${DB_USERNAME:-}" ]; then
_error "Failed to retrieve DB username from pod description."
exit 1
fi

if [ -z "${DB_NAME:-}" ]; then
_error "Failed to retrieve DB name from pod description."
exit 1
fi

local PASSWORD_REF
PASSWORD_REF=$(
kubectl get deployment \
--namespace "${NAMESPACE}" \
--output 'jsonpath={.spec.template.spec.initContainers[].env[?(@.name == "PGPASSWORD")].valueFrom.secretKeyRef}' \
"${APPLICATION}"
)
local SECRET_NAME
SECRET_NAME=$(jq -r '.name' <<< "$PASSWORD_REF")
local SECRET_KEY
SECRET_KEY=$(jq -r '.key' <<< "$PASSWORD_REF")

_info "Retrieving DB password from secret [${SECRET_NAME}]..."
local DB_PASSWORD
DB_PASSWORD=$(
kubectl get secret \
--namespace "${NAMESPACE}" \
--output "jsonpath={.data.${SECRET_KEY}}" \
"${SECRET_NAME}" \
| base64 --decode
)

if [ -z "${DB_PASSWORD}" ]; then
_error "Failed to retrieve DB password."
exit 1
fi

subcmd_debug_shell " \
/bin/env
PGPASSWORD=$DB_PASSWORD \
${SEARCH_PATH:+PGOPTIONS=--search_path=$SEARCH_PATH} \
Copy link
Contributor

@martinflorian-da martinflorian-da Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there is a nice way to infer the search path...

I often just guess/know it (is it perhaps always the same as database name for our deployments?).

But maybe there is some nice postgres way to just add all existing schemas to the search path or something...

Copy link
Contributor Author

@mblaze-da mblaze-da Oct 16, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I tried to set the search path to include all schemas but I couldn't get it to work. Listing all schemas is simple:

PGPASSWORD=$DB_PASSWORD psql \
  --host=$DB_HOST \
  --port=$DB_PORT \
  --username=$DB_USERNAME \
  --dbname=$DB_NAME \
  --tuples-only \
  --pset format=unaligned \
  --command "SELECT string_agg(schema_name, ',') FROM information_schema.schemata"

as well as setting the search path:

PGOPTIONS=--search_path=$(...)

What I couldn't figure out was how to delay the evaluation of the schema listing till pod runtime while at the same time keeping the session interactive. I could get either one or the other to work. Not both at the same time. I gave up and implemented a simple mapping that uses DB_NAME with exceptions for participant, sequencer and mediator, as we've discussed on Slack. TBH I don't want to invest much more time into improving this as I don't think it is that important.

Perhaps this could be achieved by modifying the pod configuration but I did not want to duplicate any code from the debug_shell subcommand.

I tested it with participant and sv-app and it seems to be working.

Copy link
Contributor

@martinflorian-da martinflorian-da Oct 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like a perfectly reasonable decision to me!

psql \
--host=$DB_HOST \
--port=$DB_PORT \
--username=$DB_USERNAME \
--dbname=$DB_NAME \
"
}

################################
### Main
################################
Expand Down