Setup Matrix Synapse (continues #153) - #202
Draft
miharp wants to merge 21 commits into
Draft
Conversation
This sets up a role and various profiles to run a Synapse server via docker compose. The setup defines multiple workers so that it is scalable and accounts for heavier than standard federation traffic seeing as most accounts will originate on matrix.org instead of on this Matrix servier. Significant parts of this code were generated by Claude and via Google's Antigravity IDE. In Claude's web interface, the Sonnet 4.6 model was used. In Antigravity, the Gemini Pro 3.1 (High) model was used. Attribution to both is provided both in this text and via Co-authored-by entries in the commit. Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: Antigravity <antigravity@google.com>
The letsencrypt module (via profiles::lets_encrypt) already manages the certbot package (aliased Package[letsencrypt] -> certbot) and renewal via manage_cron, so including profiles::certbot alongside it fails catalog compilation with a duplicate Package[certbot] declaration. No other role combines the two profiles. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
The role never included profiles::docker, so nothing installed the engine the compose stack needs. And docker::compose is a class in puppetlabs-docker 10.x (it installs the compose plugin); the resource for managing a stack is the docker_compose type. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
The nginx module types the parameter Optional[String[1]]; an empty string fails compilation. The default behavior is what was wanted. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
The nginx module types members as Hash[String, Nginx::UpstreamMember]; plain address arrays fail compilation. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
…rings The nginx module types proxy_set_header as Array[String]; a hash fails compilation. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
The official image runs Synapse as UID 991 with /config mounted read-only; a root-owned 0600 file gives PermissionError at startup and every container crash-loops. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
homeserver.yaml sets log_config: /config/log.yaml, but nothing creates it and /config is mounted read-only, so Synapse dies trying to generate it. Console logging suits containerized workers. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
… UID Docker auto-creates missing bind-mount sources root-owned; Synapse (UID 991) then cannot write its signing key, media, or logs and every container crash-loops on startup. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
nginx permits listen options on a single [::]:80 listener, and puppet-nginx v8 rejects the '' workaround the PR used — a single-space option string renders as none. Every Synapse worker parses the shared homeserver.yaml and creates media_store_path at startup, so the media bind mount is required on all synapse containers, not just main and the media worker. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
profiles: :docker disables Docker's own iptables management; the nftables: :rules::docker_ce parameters default to docker0/172.17.0.0/16, while compose creates a hash-named bridge on 172.18.0.0/16 — so all traffic to the containers (including nginx proxying to the published 127.0.0.1 ports) is dropped. Pin the network name and subnet in the compose file and point the firewall parameters at them. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
Docker's embedded DNS resolves container names reliably; service-alias registration proved flaky under docker 29.7/compose 5.4 with a custom network (endpoints came up with Aliases: null and service names returned NXDOMAIN). homeserver.yaml and the workers' instance_map dial postgres/redis/synapse-* by exactly these names, so pinning container_name makes resolution deterministic either way. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
docker_ce rules cover forwarded traffic; nginx and docker-proxy dial the published 127.0.0.1 ports from the host, and that egress into the container subnet hits the default-drop output chain — every proxied request times out (nginx 504). Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
The official Synapse image ignores SYNAPSE_WORKER_LISTENERS-style variables, so every worker ran without listeners (connection refused on its instance_map port, nginx 502). Workers now get a second --config-path with a Puppet-managed per-worker file; each worker's main listener also carries the replication resource that instance_map routing requires. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
Only synapse-main carried the build stanza; the other services reference synapse-s3-local:latest image-only, so a clean host races compose into pulling a nonexistent Hub image before (or instead of) building — bring-up aborts nondeterministically. Declaring the same build everywhere lets compose dedupe it and never pull. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
Build-time containers run on Docker's default bridge, which the nftables docker_ce rules no longer cover once they are pointed at the compose bridge — image builds cannot reach the network (pip fails resolving pypi.org). Host networking for builds sidesteps the seam. Pinning the base image matters because Synapse migrates the database schema at startup: an innocent rebuild from :latest can silently jump major versions with no rollback path. Renovate bumps pinned tags as a matter of course. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
…ped Sensitive regex - The delegation vhost served empty well-known files, breaking federation and client discovery for the whole server_name; they now carry the JSON documented in homeserver.yaml.epp, with application/json and CORS (client discovery is cross-origin). - Media locations gain the Matrix 1.11+ authenticated paths (/_matrix/client/v1/media, /_matrix/federation/v1/media) routed to the media worker; the legacy path stays for compatibility. - The sync location regex is replaced with the canonical worker-docs endpoints (the old one matched account-data/filters/OpenID instead), and its read timeout rises 60s -> 600s: sync long-poll timeouts are client-chosen and nginx was severing healthy polls. - The Sensitive lookup_options regex said 'profile::' (singular), so it never matched and plain-string secrets failed the Sensitive[String] parameter types; fixed and deliberately scoped to profiles::matrix rather than fleet-wide. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
The base image's built-in HEALTHCHECK probes localhost:8008, which only the main process serves — every worker reported unhealthy forever while working fine. Workers now probe /health on their own listener port; media-cron (a cron loop, nothing listens) disables the inherited check. Signed-off-by: Michael Harp <mike@mikeharp.com> Co-authored-by: Claude <noreply@anthropic.com>
Both registries are official per the Synapse install docs, but Docker Hub rate-limits anonymous pulls per source IP, which bites cloud hosts behind shared NAT, and element-hq is the canonical org since the Element split. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Michael Harp <mike@mikeharp.com>
Compile-checks the profile with the role's composition on both supported Ubuntu releases and pins the load-bearing details: the delegation payloads, homeserver.yaml ownership for the container UID, one config file per worker, and the egress nftables rule. Eight of the fixes in this branch were compile errors that a spec like this would have caught in CI. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Michael Harp <mike@mikeharp.com>
CI validates generated docs against the manifests; the matrix profiles were never added to the reference. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Michael Harp <mike@mikeharp.com>
Author
|
CI note: the one red job here is pre-existing. All 14 unit-test failures are profiles::grafana examples hitting a duplicate Package[certbot] that has failed every PR since March. Details and root cause in #203. Everything this branch adds is green, including the new profiles::matrix spec (13 examples, 0 failures). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Continues #153 by @genebean, who handed it over in #sig-matrix-ops. His commit is the base of this branch and the design decisions (docker-compose, the worker topology, S3 media) are his and unchanged. This adds the fixes needed to make the role deploy, found by running the controlrepo masterless in a throwaway Vagrant lab (miharp/matrix-lab) on a fresh Ubuntu 24.04 VM until Synapse served traffic.
Context: the puppet-nginx v7 to v8 major bump (#150) merged the day before the original branch was cut, and CI compiles no catalogs, so the typed-API breakage was invisible at review time.
19 fixes in 18 commits. The commit messages carry the per-fix reasoning; the short version:
Verified on a fresh Ubuntu 24.04 VM with zero manual steps (console log): full catalog applies, postgres/redis/main/4 workers/2 stream writers all healthy, /_matrix/client/versions serves through nginx. Lab caveats: ACME fails against fake domains, and S3 creds were dummies.
Also adds an rspec-puppet spec for profiles::matrix, closing the coverage gap that let the compile errors through.
Per the AI usage policy: I worked on this with Claude (Claude Code). Each commit carries the Co-authored-by trailer, and I have run and reviewed all of it myself in the lab above.