[Design Discussion] Decoupling encryption data storage

[hwupathum]

Related Feature Issue

#58

Problem Summary

Thunder currently encrypts sensitive data, but the responsibility for how encrypted data is stored is tightly coupled to individual services. This makes encryption key rotation difficult. To enable crypto agility, encryption data storage must be decoupled from the rest of the system.

Proposed Approach

• Move ownership of encrypted data handling to the crypto service so that key rotation can be introduced without data loss or service disruption.
• Design for future runtime key rotation, even if rotation is not enabled in the first iteration.
• Ensure the design works for both on-prem deployments and future Thunder Cloud requirements.

Technical Questions for Discussion

1. Should a single entity be responsible for storing all encrypted data?
2. If encrypted data is stored in multiple locations, how should a single entity track where that data exists?
3. How much coupling between storage layers and the crypto service is acceptable?

Alternative Approaches

Approach 1: Crypto Service Owns Encrypted Data Storage

Flow

encrypt(plaintext) -> encryptedDataId
decrypt(encryptedDataId) -> plaintext

Description

The crypto service encrypts the plaintext and persists the encrypted data in a storage location it owns. The calling service only stores a unique identifier returned by the crypto service. All future decrypt operations are performed by passing this identifier back to the crypto service.

Pros

• Encryption and encrypted data storage are fully decoupled from application services.
• Services have no knowledge of where or how encrypted data is stored.
• Enables centralized control for future encryption key rotation.
• Simplifies auditing and lifecycle management of encrypted data.

Cons

• Decryption requires an additional call to the crypto service.
• Introduces tighter runtime dependency on the crypto service.
• May impact read latency for high-throughput paths.

---

Approach 2: Service-Owned Storage with Centralized Encryption

Flow

encrypt(plaintext) -> encryptedData
decrypt(encryptedData) -> plaintext

Description

Each service calls the crypto service to encrypt plaintext, then stores the encrypted data itself in a dedicated encrypted-data table. For reads, the encrypted value is fetched by the service and passed back to the crypto service for decryption.

Pros

• Read and write operations can be performed atomically within the service.
• Fewer round trips during reads, since encrypted data is already available.
• Easier to fit into existing persistence models.

Cons

• Each service must implement logic to store encrypted data correctly.
• Storage patterns may diverge across services.
• Harder to guarantee full visibility of encrypted data locations during key rotation.

Architecture and Components

Components:

• Crypto Service

  • Performs encryption and decryption
  • Manages encryption key versions
  • Owns encryption metadata required for key rotation

• Encrypted Data Store(s)

  • One or more storage locations holding encrypted data
  • Storage locations are either directly managed by the crypto service or registered with it

• Consuming Services

  • Delegate encryption and decryption operations
  • Do not manage encryption keys or key versions

• The design assumes either:

• All encrypted data is stored through a single entity, or

• Encrypted data may live in multiple places, but one entity has complete knowledge of where it is stored.

Security Considerations

• Encryption keys are never exposed to application services.
• Clear separation between plaintext handling and encrypted storage.
• Support for decrypting data encrypted with older key versions.
• Audit logging for encryption operations and key changes.
• Minimized blast radius during encryption key rotation.

Implementation Complexity

Medium (3-6 weeks)

Areas of Thunder that will be impacted

Any area where encryption is used

Questions for Community Input

• Is centralized storage ownership required, or is centralized metadata sufficient?
• What guarantees are needed during encryption key rotation?
• Should runtime key rotation be mandatory or optional for on-prem deployments?
• What is the acceptable operational cost of re-encrypting existing data?

[senthalan]

This is something good to have in Thunder, so that we can provide support to external Key vaults as well. From a high level architecture perspective, what I see is Approach 1 and Approach 2 are the same. Only difference is what other services are going to store, whether it's the encrypted value or reference for that value. Because for the encryption or decryption those service have to talk to the crypto service. So I think we can start with Approach 2.

And could you please explain how the encryption data storage decoupling will help us key rotation in future? I couldn't able to correlate encryption data storage decoupling requirement with key rotation requirement.

[hwupathum]

To do an encryption-key rotation, we need to decrypt the existing encrypted data with the old key and encrypt with the new key.

A single service should be able to do the key rotation. For, that the crypto service should know where the encrypted data is stored in the product.
If the encrypted data is scattered around the db tables, implementing a key rotation service will be really hard. Also whenever a new encrypted data is added to a new table, the key rotation service also needed to be changed accordingly.
We faced this difficulties when we were trying to implement the key rotation tool for IS, and for these reasons we could not complete this.