feat(tokenization): introduce secure deterministic tokenization (#92)

Resolve the rainbow table attack vulnerability in deterministic tokenization by
introducing per-key version salts and HMAC-SHA256 keyed hashing. This ensures
that the same plaintext value produces different hashes across different key
versions and system installations.

Key changes:
- Added `Salt` field to `TokenizationKey` domain model.
- Refactored `HashService` to use HMAC-SHA256 with 32-byte salts.
- Updated `tokenizationKeyUseCase` to generate random salts for new key versions.
- Added database migrations (PostgreSQL/MySQL) for the `salt` column.
- Updated repository implementations to persist and retrieve the `salt` field.
- Updated application version to v0.24.0 and documented changes in CHANGELOG.md.

Author: allisson

CHANGELOG.md

@@ -5,6 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.24.0] - 2026-03-03
+
+### Changed
+- Refactored `tokenization` module to improve security for deterministic mode by adding per-key version salts and HMAC-SHA256 keyed hashing to prevent rainbow table attacks.
+
 ## [0.23.0] - 2026-03-02
 
 ### Added

cmd/app/main.go

@@ -12,7 +12,7 @@ import (
 
 // Build-time version information (injected via ldflags during build).
 var (
-	version   = "v0.23.0" // Semantic version with "v" prefix (e.g., "v0.12.0")
+	version   = "v0.24.0" // Semantic version with "v" prefix (e.g., "v0.12.0")
 	buildDate = "unknown" // ISO 8601 build timestamp
 	commitSHA = "unknown" // Git commit SHA
 )

docs/concepts/security-model.md

@@ -20,7 +20,8 @@ Secrets is designed for practical defense-in-depth around secret storage and cry
 ## 🎫 Tokenization security considerations
 
 - Metadata is not encrypted: do not place full PAN, credentials, or regulated payloads in token metadata.
-- Deterministic tokenization leaks equality patterns for identical plaintext under the same active key.
+- Deterministic tokenization now uses per-key version salts and HMAC-SHA256 keyed hashing to prevent rainbow table attacks.
+- Equality patterns for identical plaintext are still visible *under the same active key version* when deterministic mode is enabled.
 - TTL expiration and revocation both invalidate token usage, but neither should replace endpoint authorization.
 - Detokenization is plaintext exposure: isolate clients with `decrypt` capability and avoid shared broad policies.
 - Expired tokens should be cleaned on cadence (`clean-expired-tokens`) to reduce stale sensitive mappings.

docs/engines/tokenization.md

@@ -101,6 +101,14 @@ Example response (`200 OK`):
 - `GET /v1/tokenization/keys` (Capability: `read`)
 - `DELETE /v1/tokenization/keys/:id` (Capability: `delete`)
 
+## Deterministic Tokenization
+
+When `is_deterministic` is set to `true`, the engine ensures that the same plaintext value always produces the same token *under the same key version*.
+
+- **Security**: To prevent rainbow table attacks, each key version generates a unique random 32-byte salt. The engine uses HMAC-SHA256 with this salt to compute a unique hash for each plaintext.
+- **Equality Matching**: This mode allows for equality matching and duplicate detection within your application without exposing the sensitive plaintext.
+- **Rotation**: When a key is rotated, a new salt is generated. Identical plaintext tokenized under the new version will produce a different token than the previous version.
+
 ## Relevant CLI Commands
 
 - `rewrap-deks`: Rewraps tokenization key DEKs when rotating the KEK.

docs/examples/curl.md

@@ -171,8 +171,9 @@ curl -X POST "$BASE_URL/v1/tokenization/detokenize" \
 
 Deterministic caveat:
 
-- When `is_deterministic=true`, tokenizing the same plaintext with the same active key can return the same token
-- Prefer non-deterministic mode unless you explicitly need equality matching
+- When `is_deterministic=true`, the engine uses per-key version salts and HMAC-SHA256 to prevent rainbow table attacks.
+- Identical plaintext tokenized with the same active key version will return the same token.
+- Prefer non-deterministic mode unless you explicitly need equality matching.
 
 ## Common Mistakes
 

docs/examples/go.md

@@ -254,7 +254,8 @@ func tokenizationFlow(token string) error {
 
 Deterministic caveat:
 
-- Keys configured as deterministic can emit the same token for the same plaintext under the same active key.
+- Keys configured as deterministic use per-key version salts and HMAC-SHA256 to prevent rainbow table attacks.
+- They can emit the same token for the same plaintext under the same active key version.
 - Use deterministic mode only when your workflow requires equality matching.
 
 Rate-limit note:

docs/examples/javascript.md

@@ -165,7 +165,8 @@ async function tokenizationFlow(token) {
 
 Deterministic caveat:
 
-- With `is_deterministic: true`, tokenizing the same plaintext with the same active key can produce the same token.
+- With `is_deterministic: true`, the engine uses per-key version salts and HMAC-SHA256 to prevent rainbow table attacks.
+- Identical plaintext tokenized with the same active key version will produce the same token.
 - Prefer non-deterministic mode unless stable equality matching is required.
 
 Rate-limit note:

docs/examples/python.md

@@ -147,7 +147,8 @@ def tokenize_detokenize(token: str) -> None:
 
 Deterministic caveat:
 
-- If you create a key with `is_deterministic=True`, repeated tokenization of identical plaintext can return the same token.
+- If you create a key with `is_deterministic=True`, the engine uses per-key version salts and HMAC-SHA256 to prevent rainbow table attacks.
+- Repeated tokenization of identical plaintext under the same active key version will return the same token.
 - Use deterministic mode only when equality matching is a functional requirement.
 
 Rate-limit note:

internal/tokenization/domain/errors.go

@@ -41,7 +41,14 @@ var (
 	// ErrTokenizationKeyNameEmpty indicates the tokenization key name is empty.
 	ErrTokenizationKeyNameEmpty = errors.Wrap(errors.ErrInvalidInput, "tokenization key name cannot be empty")
 
-	// ErrTokenizationKeyVersionInvalid indicates the version is invalid (must be > 0).
+	// ErrTokenizationKeySaltInvalid indicates the tokenization key salt is invalid.
+	ErrTokenizationKeySaltInvalid = errors.Wrap(
+		errors.ErrInvalidInput,
+		"tokenization key salt cannot be empty for deterministic keys",
+	)
+
+	// ErrTokenizationKeyVersionInvalid indicates the tokenization key version is invalid.
+
 	ErrTokenizationKeyVersionInvalid = errors.Wrap(
 		errors.ErrInvalidInput,
 		"tokenization key version must be greater than 0",

internal/tokenization/domain/tokenization_key.go

@@ -27,6 +27,9 @@ type TokenizationKey struct {
 	// When true, enables efficient duplicate detection; when false, provides better privacy.
 	IsDeterministic bool
 
+	// Salt is used for deterministic tokenization to prevent rainbow table attacks.
+	Salt []byte
+
 	// DekID is the reference to the Data Encryption Key used to encrypt values for this version.
 	DekID uuid.UUID
 
@@ -49,6 +52,9 @@ func (tk *TokenizationKey) Validate() error {
 	if err := tk.FormatType.Validate(); err != nil {
 		return ErrInvalidFormatType
 	}
+	if tk.IsDeterministic && len(tk.Salt) == 0 {
+		return ErrTokenizationKeySaltInvalid
+	}
 	if tk.DekID == uuid.Nil {
 		return ErrTokenizationKeyDekIDInvalid
 	}