Skip to content

Commit c4b61b1

Browse files
stedsted
authored
Merge pull request #3 from coroluca/graceful-shutdown
Graceful shutdown: SIGTERM, in-flight request draining, /ready health semantics, clean pool close
2 parents 1ee7b9d + bc35a71 commit c4b61b1

12 files changed

Lines changed: 493 additions & 122 deletions

‎CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,10 @@
66
* **`::cast` injection** — cast targets in `select` are validated at parse time and rejected with 400 otherwise; a cast can't be a bind parameter, so it was interpolated verbatim and could alter the SELECT (reachable by any role that can read).
77
* **DDL quoting** — quote the remaining `CREATE`/`ALTER DATABASE` identifiers, and whitelist grant/revoke verbs and object types instead of interpolating them.
88

9+
### Added
10+
* Shutdown now waits for in-flight requests to complete; the wait was previously hardcoded to 1 second, so every restart killed any request slower than that. The new `GracefulShutdownTimeout` config key (seconds, default 0 = wait until done) bounds the wait for deployments that want a hard cap below their supervisor's stop grace period. A second signal during the wait forces an immediate exit, and `Shutdown()` is now idempotent.
11+
* `/ready` now reports `503 {"status":"draining"}` as soon as a graceful shutdown begins, while `/live` keeps answering 200 until the process exits — the standard probe contract for zero-downtime rolling deploys. The new `DrainDelay` config key (seconds, default 0 = disabled) keeps the listener serving for that long after readiness flips, giving load balancers time to deregister the instance before it stops accepting connections. The delay applies to SIGTERM only; an interactive Ctrl-C (SIGINT) shuts down immediately, and a second signal during the window skips it.
12+
913
### Fixed
1014
* Schema cache held in an atomic pointer (was a data race on reload).
1115
* Serializers return an error on a malformed wire buffer or a type/dimension mismatch instead of panicking or silently misparsing.
@@ -14,6 +18,9 @@
1418
* Boolean-filter nesting capped at 100 levels (stack-overflow DoS).
1519
* Invalid grant input returns 400 instead of 500.
1620
* Session watcher no longer deadlocks when paused.
21+
* **SIGTERM** — the server now shuts down gracefully on SIGTERM (the default stop signal of Docker, Kubernetes and systemd), not only on SIGINT/Ctrl-C. A raw SIGTERM previously terminated the process immediately, cutting in-flight requests and resetting database connections.
22+
* **Exit status** — a graceful shutdown now exits with status 0; it previously exited 1, which supervisors interpret as a crash.
23+
* **Database connections on shutdown** — connection pools and the notification listener's dedicated connection are now closed after the HTTP server drains, so Postgres sees clean disconnects instead of connection resets on every stop. `NotificationListener.Stop()` now interrupts a pending `WaitForNotification` and waits for the listener goroutine to exit before returning.
1724

1825
## 0.8.1 - 2026-07-08
1926

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ update-ui:
3131

3232
test:
3333
$(GO) test $(TEST_FLAGS) ./database
34+
$(GO) test $(TEST_FLAGS) ./server
3435
$(GO) test $(TEST_FLAGS) ./test/api
3536
$(GO) test $(TEST_FLAGS) ./test/postgrest
3637
$(GO) test $(TEST_FLAGS) ./test/recursive

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -772,6 +772,8 @@ The configuration file *config.jsonc* (JSON with Comments) is created automatica
772772
| Plugins | Ordered list of plugins | [] |
773773
| ReadTimeout | The maximum duration for reading the entire request, including the body (seconds) | 60 |
774774
| WriteTimeout | The maximum duration before timing out writes of the response (seconds) | 60 |
775+
| GracefulShutdownTimeout | The maximum duration to wait for in-flight requests to complete on shutdown (seconds, 0 to wait until done) | 0 |
776+
| DrainDelay | Seconds to keep serving after /ready starts reporting 503 on a SIGTERM shutdown, so load balancers can deregister the instance; SIGINT skips the delay (0 to disable) | 0 |
775777
| RequestMaxBytes | Max bytes allowed in requests, to limit the size of incoming request bodies (0 for unlimited) | 1048576 (1MB) |
776778
| Database.URL | Database URL as postgresql://user:pwd@host:port/database | "" |
777779
| Database.MinPoolConnections | Miminum connections per pool | 10 |

‎api/health.go‎

Lines changed: 18 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,13 +11,25 @@ import (
1111
func InitHealthRoutes(apiHelper Helper) {
1212
router := apiHelper.GetRouter()
1313

14-
// Register both health check endpoints with the same handler
15-
router.Handle("GET", "/live", HealthHandler)
16-
router.Handle("GET", "/ready", HealthHandler)
14+
router.Handle("GET", "/live", LiveHandler)
15+
router.Handle("GET", "/ready", ReadyHandler(apiHelper))
1716
}
1817

19-
// HealthHandler handles both /live and /ready endpoints
20-
func HealthHandler(c context.Context, w http.ResponseWriter, r heligo.Request) (int, error) {
21-
// If this handler is executing, the server is up and ready
18+
// LiveHandler handles the /live endpoint: the process is up. It deliberately
19+
// keeps answering 200 while the server drains — a graceful shutdown must not
20+
// look like a hung process.
21+
func LiveHandler(c context.Context, w http.ResponseWriter, r heligo.Request) (int, error) {
2222
return heligo.WriteJSON(w, http.StatusOK, map[string]string{"status": "ok"})
2323
}
24+
25+
// ReadyHandler returns the handler for the /ready endpoint: route traffic to
26+
// this instance. It flips to 503 as soon as a shutdown begins, so load
27+
// balancers deregister the instance while it is still serving.
28+
func ReadyHandler(apiHelper Helper) heligo.Handler {
29+
return func(c context.Context, w http.ResponseWriter, r heligo.Request) (int, error) {
30+
if apiHelper.IsDraining() {
31+
return heligo.WriteJSON(w, http.StatusServiceUnavailable, map[string]string{"status": "draining"})
32+
}
33+
return heligo.WriteJSON(w, http.StatusOK, map[string]string{"status": "ok"})
34+
}
35+
}

‎api/helper.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ type Helper interface {
2121
BaseAdminURL() string
2222
BaseAPIURL() string
2323
HasShortAPIURL() bool
24+
IsDraining() bool
2425

2526
SessionStatistics() authn.SessionStatistics
2627
}

‎database/database_test.go‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ func TestMain(m *testing.M) {
1717
var err error
1818
config := DefaultConfig()
1919
config.URL = "postgresql://postgres:postgres@0.0.0.0:5432/postgres"
20+
// Same override the server honors, already exported by CI
21+
if url := os.Getenv("SMOOTHDB_DATABASE_URL"); url != "" {
22+
config.URL = url
23+
}
2024
dbe, err = InitDbEngine(config, nil)
2125
if err != nil {
2226
fmt.Println(err.Error())

‎database/notification_listener.go‎

Lines changed: 45 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,11 @@ type NotificationListener struct {
3535
config *ListenerConfig
3636
logger *logging.Logger
3737
connString string
38-
conn *pgx.Conn
3938
handlers map[string]NotificationHandler
4039
mu sync.RWMutex
4140
running bool
42-
stop chan struct{}
41+
cancel context.CancelFunc
42+
stopped chan struct{}
4343
}
4444

4545
// NewNotificationListener creates a new notification listener
@@ -52,7 +52,6 @@ func NewNotificationListener(connString string, config *ListenerConfig, logger *
5252
logger: logger,
5353
connString: connString,
5454
handlers: make(map[string]NotificationHandler),
55-
stop: make(chan struct{}),
5655
}
5756
}
5857

@@ -78,44 +77,52 @@ func (l *NotificationListener) Start(ctx context.Context) error {
7877
return fmt.Errorf("listener is already running")
7978
}
8079
l.running = true
80+
ctx, l.cancel = context.WithCancel(ctx)
81+
l.stopped = make(chan struct{})
82+
stopped := l.stopped
8183
l.mu.Unlock()
8284

83-
go l.listenLoop(ctx)
85+
go func() {
86+
defer close(stopped)
87+
l.listenLoop(ctx)
88+
}()
8489
return nil
8590
}
8691

87-
// Stop stops the listener
92+
// Stop stops the listener, interrupting a pending WaitForNotification, and
93+
// waits for the loop to exit, so its dedicated connection is closed when
94+
// Stop returns — also when racing another Stop. It must not be called from
95+
// a notification handler, which runs on the loop it would wait for.
8896
func (l *NotificationListener) Stop() {
8997
l.mu.Lock()
90-
defer l.mu.Unlock()
9198
if l.running {
92-
close(l.stop)
9399
l.running = false
100+
l.cancel()
101+
}
102+
stopped := l.stopped
103+
l.mu.Unlock()
104+
if stopped != nil {
105+
<-stopped
94106
}
95107
}
96108

97109
// listenLoop maintains the connection and processes notifications
98110
func (l *NotificationListener) listenLoop(ctx context.Context) {
99111
for {
100-
select {
101-
case <-ctx.Done():
102-
return
103-
case <-l.stop:
112+
err := l.connectAndListen(ctx)
113+
if ctx.Err() != nil {
114+
// Stopped: WaitForNotification reports the canceled
115+
// context as an error, not worth logging
104116
return
105-
default:
106-
err := l.connectAndListen(ctx)
107-
if err != nil {
108-
if l.logger != nil {
109-
l.logger.Err(err).Msg("Notification listener error")
110-
}
111-
select {
112-
case <-time.After(l.config.ReconnectDelay):
113-
continue
114-
case <-ctx.Done():
115-
return
116-
case <-l.stop:
117-
return
118-
}
117+
}
118+
if err != nil {
119+
if l.logger != nil {
120+
l.logger.Err(err).Msg("Notification listener error")
121+
}
122+
select {
123+
case <-time.After(l.config.ReconnectDelay):
124+
case <-ctx.Done():
125+
return
119126
}
120127
}
121128
}
@@ -127,9 +134,13 @@ func (l *NotificationListener) connectAndListen(ctx context.Context) error {
127134
if err != nil {
128135
return fmt.Errorf("failed to connect: %w", err)
129136
}
130-
defer conn.Close(ctx)
131-
132-
l.conn = conn
137+
defer func() {
138+
// Close cleanly even when ctx is already canceled (i.e. during
139+
// Stop), so Postgres sees a Terminate message instead of a reset
140+
closeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Second)
141+
defer cancel()
142+
conn.Close(closeCtx)
143+
}()
133144

134145
_, err = conn.Exec(ctx, fmt.Sprintf("LISTEN %s", l.config.Channel))
135146
if err != nil {
@@ -141,20 +152,12 @@ func (l *NotificationListener) connectAndListen(ctx context.Context) error {
141152
}
142153

143154
for {
144-
select {
145-
case <-ctx.Done():
146-
return ctx.Err()
147-
case <-l.stop:
148-
return nil
149-
default:
150-
notification, err := conn.WaitForNotification(ctx)
151-
if err != nil {
152-
return fmt.Errorf("error waiting for notification: %w", err)
153-
}
154-
155-
if notification != nil {
156-
l.handleNotification(ctx, notification.Payload)
157-
}
155+
notification, err := conn.WaitForNotification(ctx)
156+
if err != nil {
157+
return fmt.Errorf("error waiting for notification: %w", err)
158+
}
159+
if notification != nil {
160+
l.handleNotification(ctx, notification.Payload)
158161
}
159162
}
160163
}

‎database/notification_listener_test.go‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
package database
2+
3+
import (
4+
"context"
5+
"testing"
6+
"time"
7+
)
8+
9+
// Stop must interrupt the listener even while it is blocked in
10+
// WaitForNotification, and return only once the loop has exited and its
11+
// dedicated connection is closed. It used to leave both behind: the stop
12+
// channel was only checked between notifications.
13+
func TestListenerStopInterruptsWait(t *testing.T) {
14+
l := NewNotificationListener(dbe.config.URL, nil, nil)
15+
if err := l.Start(context.Background()); err != nil {
16+
t.Fatal(err)
17+
}
18+
// Let the listener connect and block waiting for notifications
19+
time.Sleep(200 * time.Millisecond)
20+
21+
done := make(chan struct{})
22+
go func() {
23+
l.Stop()
24+
close(done)
25+
}()
26+
select {
27+
case <-done:
28+
case <-time.After(3 * time.Second):
29+
t.Fatal("Stop did not return: listener stuck in WaitForNotification")
30+
}
31+
32+
// The listener must be restartable after a stop
33+
if err := l.Start(context.Background()); err != nil {
34+
t.Fatalf("restart after Stop failed: %v", err)
35+
}
36+
l.Stop()
37+
}

‎server/config.go‎

Lines changed: 59 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -24,64 +24,68 @@ type ConfigOptions struct {
2424

2525
// Config holds the current configuration
2626
type Config struct {
27-
Address string `comment:"Server address and port (default: 0.0.0.0:4000)"`
28-
CertFile string `comment:"TLS certificate file (default: '')"`
29-
KeyFile string `comment:"TLS certificate key file (default: '')"`
30-
LoginMode string `comment:"Login mode: none, db, gotrue (default: none)"`
31-
AuthURL string `comment:"URL of the external AuthN service (default: '')"`
32-
AllowAnon bool `comment:"Allow unauthenticated connections (default: false)"`
33-
JWTSecret string `comment:"Secret for JWT tokens"`
34-
SessionMode string `comment:"Session mode: none, role (default: role)"`
35-
EnableAdminRoute bool `comment:"Enable administration of databases and tables (default: false)"`
36-
EnableAdminUI bool `comment:"Enable Admin dashboard (default: false)"`
37-
EnableAPIRoute bool `comment:"Enable API access (default: true)"`
38-
BaseAPIURL string `comment:"Base URL for the API (default: /api)"`
39-
ShortAPIURL bool `comment:"Avoid database name in API URL (needs a single allowed database)"`
40-
BaseAdminURL string `comment:"Base URL for the Admin API (default: /admin)"`
41-
CORSAllowedOrigins []string `comment:"CORS Access-Control-Allow-Origin (default: [*] for all)"`
42-
CORSAllowCredentials bool `comment:"CORS Access-Control-Allow-Credentials (default: false)"`
43-
EnableDebugRoute bool `comment:"Enable debug access (default: false)"`
44-
PluginDir string `comment:"Plugins' directory (default: ./_plugins)"`
45-
Plugins []string `comment:"Ordered list of plugins (default: [])"`
46-
ReadTimeout int64 `comment:"The maximum duration (seconds) for reading the entire request, including the body (default: 60)"`
47-
WriteTimeout int64 `comment:"The maximum duration before timing out writes of the response (default: 60)"`
48-
TokenExpiry int64 `comment:"JWT token expiry in seconds (default: 86400 = 24h, 0 for no expiry)"`
49-
VerboseErrors bool `comment:"Return full database error details (hint, detail) to clients (default: true)"`
50-
RequestMaxBytes int64 `comment:"Max bytes allowed in requests, to limit the size of incoming request bodies (default: 1M, 0 for unlimited)"`
51-
Database database.Config `comment:"Database configuration"`
52-
JQ jqeval.Config `comment:"jq evaluation configuration"`
53-
Logging logging.Config `comment:"Logging configuration"`
27+
Address string `comment:"Server address and port (default: 0.0.0.0:4000)"`
28+
CertFile string `comment:"TLS certificate file (default: '')"`
29+
KeyFile string `comment:"TLS certificate key file (default: '')"`
30+
LoginMode string `comment:"Login mode: none, db, gotrue (default: none)"`
31+
AuthURL string `comment:"URL of the external AuthN service (default: '')"`
32+
AllowAnon bool `comment:"Allow unauthenticated connections (default: false)"`
33+
JWTSecret string `comment:"Secret for JWT tokens"`
34+
SessionMode string `comment:"Session mode: none, role (default: role)"`
35+
EnableAdminRoute bool `comment:"Enable administration of databases and tables (default: false)"`
36+
EnableAdminUI bool `comment:"Enable Admin dashboard (default: false)"`
37+
EnableAPIRoute bool `comment:"Enable API access (default: true)"`
38+
BaseAPIURL string `comment:"Base URL for the API (default: /api)"`
39+
ShortAPIURL bool `comment:"Avoid database name in API URL (needs a single allowed database)"`
40+
BaseAdminURL string `comment:"Base URL for the Admin API (default: /admin)"`
41+
CORSAllowedOrigins []string `comment:"CORS Access-Control-Allow-Origin (default: [*] for all)"`
42+
CORSAllowCredentials bool `comment:"CORS Access-Control-Allow-Credentials (default: false)"`
43+
EnableDebugRoute bool `comment:"Enable debug access (default: false)"`
44+
PluginDir string `comment:"Plugins' directory (default: ./_plugins)"`
45+
Plugins []string `comment:"Ordered list of plugins (default: [])"`
46+
ReadTimeout int64 `comment:"The maximum duration (seconds) for reading the entire request, including the body (default: 60)"`
47+
WriteTimeout int64 `comment:"The maximum duration before timing out writes of the response (default: 60)"`
48+
GracefulShutdownTimeout int64 `comment:"The maximum duration (seconds) to wait for in-flight requests to complete on shutdown (default: 0, wait until done)"`
49+
DrainDelay int64 `comment:"Seconds to keep serving after /ready starts reporting 503 on a SIGTERM shutdown, so load balancers can deregister the instance; SIGINT skips the delay (default: 0, disabled)"`
50+
TokenExpiry int64 `comment:"JWT token expiry in seconds (default: 86400 = 24h, 0 for no expiry)"`
51+
VerboseErrors bool `comment:"Return full database error details (hint, detail) to clients (default: true)"`
52+
RequestMaxBytes int64 `comment:"Max bytes allowed in requests, to limit the size of incoming request bodies (default: 1M, 0 for unlimited)"`
53+
Database database.Config `comment:"Database configuration"`
54+
JQ jqeval.Config `comment:"jq evaluation configuration"`
55+
Logging logging.Config `comment:"Logging configuration"`
5456
}
5557

5658
func defaultConfig() *Config {
5759
return &Config{
58-
Address: "0.0.0.0:4000",
59-
CertFile: "",
60-
KeyFile: "",
61-
LoginMode: "none",
62-
AuthURL: "",
63-
AllowAnon: false,
64-
JWTSecret: "",
65-
SessionMode: "role",
66-
EnableAdminRoute: false,
67-
EnableAdminUI: false,
68-
EnableAPIRoute: true,
69-
BaseAPIURL: "/api",
70-
ShortAPIURL: false,
71-
BaseAdminURL: "/admin",
72-
CORSAllowedOrigins: []string{},
73-
CORSAllowCredentials: false,
74-
EnableDebugRoute: false,
75-
PluginDir: "./_plugins",
76-
Plugins: []string{},
77-
ReadTimeout: 60,
78-
WriteTimeout: 60,
79-
TokenExpiry: 86400,
80-
VerboseErrors: true,
81-
RequestMaxBytes: 1024 * 1024,
82-
Database: *database.DefaultConfig(),
83-
JQ: *jqeval.DefaultConfig(),
84-
Logging: *logging.DefaultConfig(),
60+
Address: "0.0.0.0:4000",
61+
CertFile: "",
62+
KeyFile: "",
63+
LoginMode: "none",
64+
AuthURL: "",
65+
AllowAnon: false,
66+
JWTSecret: "",
67+
SessionMode: "role",
68+
EnableAdminRoute: false,
69+
EnableAdminUI: false,
70+
EnableAPIRoute: true,
71+
BaseAPIURL: "/api",
72+
ShortAPIURL: false,
73+
BaseAdminURL: "/admin",
74+
CORSAllowedOrigins: []string{},
75+
CORSAllowCredentials: false,
76+
EnableDebugRoute: false,
77+
PluginDir: "./_plugins",
78+
Plugins: []string{},
79+
ReadTimeout: 60,
80+
WriteTimeout: 60,
81+
GracefulShutdownTimeout: 0,
82+
DrainDelay: 0,
83+
TokenExpiry: 86400,
84+
VerboseErrors: true,
85+
RequestMaxBytes: 1024 * 1024,
86+
Database: *database.DefaultConfig(),
87+
JQ: *jqeval.DefaultConfig(),
88+
Logging: *logging.DefaultConfig(),
8589
}
8690
}
8791

@@ -122,7 +126,7 @@ func getEnvironment(c *Config) {
122126
if enableAdminRoute != "" {
123127
c.EnableAdminRoute = strings.ToLower(enableAdminRoute) == "true"
124128
}
125-
129+
126130
// CORS configuration
127131
corsAllowedOrigins := os.Getenv("SMOOTHDB_CORS_ALLOWED_ORIGINS")
128132
if corsAllowedOrigins != "" {

0 commit comments

Comments
 (0)