[Feature 10] Graceful Shutdown

[onlyhyde]

Overview

Implement graceful shutdown for safe server termination - complete in-flight requests and cleanup resources.

Implementation Details

Files:

• internal/shutdown/manager.go - Shutdown manager
• cmd/facilitator/main.go - Integration

Shutdown Manager

type Manager struct {
    timeout    time.Duration
    callbacks  []func(context.Context) error
    mu         sync.Mutex
    shutdownCh chan struct{}
    doneCh     chan struct{}
}

func NewManager(timeout time.Duration) *Manager
func (m *Manager) RegisterCallback(fn func(context.Context) error)
func (m *Manager) Wait()      // Blocks until SIGINT/SIGTERM
func (m *Manager) Shutdown()  // Programmatic shutdown

Shutdown Sequence

1. Receive SIGINT/SIGTERM
2. Reject new requests (503 Service Unavailable)
3. Health check returns 503 (load balancer removes instance)
4. Wait for in-flight requests to complete
5. Execute callbacks in LIFO order:
   • Stop API server
   • Close storage connections
   • Close RPC connections
6. Exit process

Request Tracking Middleware

func (s *Server) RequestTrackingMiddleware() echo.MiddlewareFunc

• Track active requests with atomic counter
• Reject new requests during shutdown
• Wait for counter to reach zero

Health Check Integration

func (s *Server) healthHandler(c echo.Context) error {
    if atomic.LoadInt32(&s.shuttingDown) == 1 {
        return c.JSON(503, ...)
    }
    return c.JSON(200, ...)
}

Acceptance Criteria

• Shutdown manager with signal handling
• LIFO callback execution
• Request tracking middleware
• In-flight request completion
• Health check returns 503 during shutdown
• Configurable shutdown timeout
• Unit tests
• Integration tests

Dependencies

• [Feature 03] API Server #22 API Server
• [Feature 07] Storage & Idempotency #26 Storage & Idempotency

Branch

feature/graceful-shutdown