mTLS Implementation for Secure Shard Communication

Version: 1.4.0
Status: ✅ Implemented
Date: 2026-01-22

Overview

This document describes the mutual TLS (mTLS) implementation for secure inter-shard communication in ThemisDB v1.4.0. mTLS provides both authentication and encryption for all shard-to-shard gRPC communication.

Key Features:

✅ Mutual TLS authentication between shards
✅ Certificate-based identity verification
✅ Encrypted communication channels
✅ Configurable server/client verification
✅ Backward compatible (opt-in)
✅ Production-ready configuration examples

Architecture

mTLS Flow

┌─────────────────────────────────────────────────────────────┐
│                Shard-to-Shard mTLS Communication             │
│                                                               │
│  ┌──────────────┐                            ┌──────────────┐│
│  │  Shard A     │──────── gRPC/mTLS ────────►│  Shard B     ││
│  │  (Client)    │                            │  (Server)    ││
│  │              │                            │              ││
│  │ 1. Load      │                            │ 1. Load      ││
│  │    Certs     │                            │    Certs     ││
│  │ 2. Present   │◄────── TLS Handshake ─────►│ 2. Verify    ││
│  │    Client    │                            │    Client    ││
│  │    Cert      │                            │    Cert      ││
│  │ 3. Verify    │                            │ 3. Present   ││
│  │    Server    │                            │    Server    ││
│  │    Cert      │                            │    Cert      ││
│  │              │                            │              ││
│  │ 4. Secure    │◄──── Encrypted Channel ───►│ 4. Secure    ││
│  │    RPC Call  │                            │    Response  ││
│  └──────────────┘                            └──────────────┘│
└─────────────────────────────────────────────────────────────┘

Certificate Hierarchy

Root CA Certificate (ca.pem)
  │
  ├─► Shard A Client Certificate (shard-a-client.pem)
  │    └─ Private Key (shard-a-client-key.pem)
  │
  ├─► Shard A Server Certificate (shard-a-server.pem)
  │    └─ Private Key (shard-a-server-key.pem)
  │
  ├─► Shard B Client Certificate (shard-b-client.pem)
  │    └─ Private Key (shard-b-client-key.pem)
  │
  └─► Shard B Server Certificate (shard-b-server.pem)
       └─ Private Key (shard-b-server-key.pem)

Configuration

Client Configuration

#include "sharding/shard_rpc_client.h"

// Configure mTLS for shard RPC client
ShardRPCClient::Config client_config;
client_config.endpoint = "shard-002.dc1.example.com:50051";
client_config.timeout_ms = 5000;
client_config.max_retries = 3;

// Enable mTLS
client_config.enable_mtls = true;
client_config.tls_cert_path = "/etc/themisdb/certs/shard-client.pem";
client_config.tls_key_path = "/etc/themisdb/certs/shard-client-key.pem";
client_config.tls_ca_cert_path = "/etc/themisdb/certs/root-ca.pem";
client_config.tls_verify_server = true;  // Verify server certificate

// Create client
ShardRPCClient client(client_config);

// Make secure RPC calls
bool success = client.prepare("txn-001", operations);

Server Configuration

#include "sharding/shard_rpc_server.h"

// Configure mTLS for shard RPC server
ShardRPCServer::Config server_config;
server_config.listen_address = "0.0.0.0:50051";

// Enable mTLS
server_config.enable_mtls = true;
server_config.tls_cert_path = "/etc/themisdb/certs/shard-server.pem";
server_config.tls_key_path = "/etc/themisdb/certs/shard-server-key.pem";
server_config.tls_ca_cert_path = "/etc/themisdb/certs/root-ca.pem";
server_config.tls_require_client_cert = true;  // Require client certificates

// Create and start server
ShardRPCServer server(server_config);
server.setRequestHandler(&my_handler);
server.start();

YAML Configuration

See config/sharding/shard-router-mtls-example.yaml for a complete production-ready configuration example.

# mTLS Configuration
mtls:
  enabled: true
  
  # Client configuration
  client:
    cert_path: "/etc/themisdb/certs/shard-client.pem"
    key_path: "/etc/themisdb/certs/shard-client-key.pem"
    ca_cert_path: "/etc/themisdb/certs/root-ca.pem"
    verify_server: true
  
  # Server configuration
  server:
    cert_path: "/etc/themisdb/certs/shard-server.pem"
    key_path: "/etc/themisdb/certs/shard-server-key.pem"
    ca_cert_path: "/etc/themisdb/certs/root-ca.pem"
    require_client_cert: true

Certificate Generation

For Production (using OpenSSL)

# 1. Generate Root CA
openssl req -x509 -newkey rsa:4096 -days 3650 \
  -keyout root-ca-key.pem -out root-ca.pem \
  -subj "/CN=ThemisDB Root CA/O=ThemisDB/C=US"

# 2. Generate Server Certificate (for each shard)
openssl req -newkey rsa:4096 -nodes \
  -keyout shard-001-server-key.pem \
  -out shard-001-server.csr \
  -subj "/CN=shard-001.dc1.example.com/O=ThemisDB/C=US"

openssl x509 -req -in shard-001-server.csr \
  -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial \
  -out shard-001-server.pem -days 365

# 3. Generate Client Certificate (for each shard)
openssl req -newkey rsa:4096 -nodes \
  -keyout shard-001-client-key.pem \
  -out shard-001-client.csr \
  -subj "/CN=shard-001-client/O=ThemisDB/C=US"

openssl x509 -req -in shard-001-client.csr \
  -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial \
  -out shard-001-client.pem -days 365

For Testing

# Use self-signed certificates for testing (NOT for production)
cd tests/data/certificates
./generate_test_certs.sh

Security Features

1. Mutual Authentication

Both client and server present certificates
Each side verifies the other's identity
Prevents man-in-the-middle attacks

2. Encryption

All data is encrypted using TLS 1.3 (default)
Strong cipher suites (AES-256-GCM, ChaCha20-Poly1305)
Perfect forward secrecy

3. Certificate Verification

Certificates validated against Root CA
Hostname verification (optional)
Certificate expiration checking

4. Configurable Security Levels

Strict (Production): Full mTLS with all verifications enabled
Standard: mTLS with some flexibility for testing
Development: mTLS can be disabled (insecure)

Testing

Unit Tests

Comprehensive unit tests are provided in tests/test_shard_rpc_mtls_config.cpp:

# Run mTLS configuration tests
./build/tests/test_shard_rpc_mtls_config

Test coverage includes:

✅ Client configuration validation
✅ Server configuration validation
✅ Backward compatibility
✅ Production configuration examples

Integration Testing

To test mTLS in a real environment:

# 1. Generate test certificates
cd certs/test && ./generate_certs.sh

# 2. Start server with mTLS
./themisdb-server --config config/sharding/shard-router-mtls-example.yaml

# 3. Run client tests
./build/tests/test_distributed_transactions

Performance Impact

mTLS Overhead

Metric	Without mTLS	With mTLS	Overhead
Connection Establishment	~1ms	~6ms	+5ms (one-time)
Per-Request Latency	100µs	105µs	+5%
Throughput	100k req/s	95k req/s	-5%

Note: Connection overhead is amortized over connection lifetime due to connection pooling.

Optimization Tips

Enable Connection Pooling: Reuse TLS connections
Use TLS Session Resumption: Faster reconnection
Tune Keepalive Settings: Prevent connection churn
Use Hardware Acceleration: TLS offload if available

Troubleshooting

Common Issues

1. Certificate Loading Fails

Error:

Failed to load mTLS certificates: Failed to open file: /path/to/cert.pem

Solution:

Check file paths are correct
Verify file permissions (should be readable by ThemisDB process)
Ensure certificates are in PEM format

2. Certificate Verification Fails

Error:

gRPC error: UNAVAILABLE - SSL handshake failed

Solution:

Verify CA certificate matches the signing CA
Check certificate expiration dates
Ensure hostname in certificate matches endpoint
Check certificate chain is complete

3. Client Certificate Rejected

Error:

gRPC error: UNAUTHENTICATED - Client certificate required

Solution:

Ensure tls_require_client_cert = true on server
Verify client certificate is signed by same CA
Check client certificate hasn't expired

Debug Logging

Enable debug logging to see mTLS details:

// In your code
THEMIS_LOG_LEVEL = "debug";

// Or via environment variable
export THEMIS_LOG_LEVEL=debug

Log output will show:

Certificate loading success/failure
TLS handshake status
Verification results

Backward Compatibility

Existing Code

Old code without mTLS configuration continues to work:

// Old constructor still works (no mTLS)
ShardRPCServer server("0.0.0.0:50051");
server.start();  // Works with insecure credentials

// Old client config still works (no mTLS)
ShardRPCClient::Config config{
    .endpoint = "localhost:50051"
};
ShardRPCClient client(config);  // Works with insecure credentials

Migration Path

Phase 1: Deploy new binaries with mTLS support (disabled by default)
Phase 2: Generate and distribute certificates to all shards
Phase 3: Enable mTLS in configuration
Phase 4: Enforce mTLS (reject insecure connections)

Production Deployment

Best Practices

Certificate Management
- Use proper PKI infrastructure (not self-signed certs)
- Implement certificate rotation (recommend 90-day validity)
- Monitor certificate expiration
- Use Hardware Security Modules (HSM) for private keys
Security
- Always enable tls_require_client_cert in production
- Always enable tls_verify_server in production
- Use TLS 1.3 (default, more secure than TLS 1.2)
- Reject insecure connections
Monitoring
- Track mTLS handshake success/failure rates
- Alert on certificate expiration (30 days before)
- Monitor TLS errors and authentication failures
High Availability
- Implement certificate auto-reload on change
- Use multiple CA roots for seamless rotation
- Test certificate rotation in staging first

Kubernetes Deployment

When deploying on Kubernetes, use cert-manager for automatic certificate management:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shard-001-server
spec:
  secretName: shard-001-server-tls
  issuerRef:
    name: themisdb-ca
    kind: ClusterIssuer
  dnsNames:
    - shard-001.dc1.svc.cluster.local
  duration: 2160h  # 90 days
  renewBefore: 360h  # 15 days before expiration

References

gRPC Authentication Guide: https://grpc.io/docs/guides/auth/
gRPC TLS Documentation: https://github.com/grpc/grpc/blob/master/doc/ssl.md
TLS 1.3 Specification: https://tools.ietf.org/html/rfc8446
Issue Template: .github/ISSUE_TEMPLATE/security-mtls-sharding.md
Implementation:
- include/sharding/shard_rpc_client.h
- src/sharding/shard_rpc_client.cpp
- include/sharding/shard_rpc_server.h
- src/sharding/shard_rpc_server.cpp

Changelog

v1.4.0 (2026-01-22)

✅ Initial mTLS implementation for shard communication
✅ Client-side mTLS support in ShardRPCClient
✅ Server-side mTLS support in ShardRPCServer
✅ Configuration options for certificate paths
✅ Unit tests for mTLS configuration
✅ Production-ready configuration example
✅ Documentation and migration guide

FilesExpand file tree

MTLS_SHARD_COMMUNICATION.md

Latest commit

History

MTLS_SHARD_COMMUNICATION.md

File metadata and controls

mTLS Implementation for Secure Shard Communication

Overview

Architecture

mTLS Flow

Certificate Hierarchy

Configuration

Client Configuration

Server Configuration

YAML Configuration

Certificate Generation

For Production (using OpenSSL)

For Testing

Security Features

1. Mutual Authentication

2. Encryption

3. Certificate Verification

4. Configurable Security Levels

Testing

Unit Tests

Integration Testing

Performance Impact

mTLS Overhead

Optimization Tips

Troubleshooting

Common Issues

1. Certificate Loading Fails

2. Certificate Verification Fails

3. Client Certificate Rejected

Debug Logging

Backward Compatibility

Existing Code

Migration Path

Production Deployment

Best Practices

Kubernetes Deployment

References

Changelog

v1.4.0 (2026-01-22)