Implement Keystore Cleanup Service (#1801)

Signed-off-by: Angelo De Caro <adc@zurich.ibm.com>

Author: adecaro

cmd/tokengen/go.mod

@@ -122,6 +122,7 @@ require (
 	go.uber.org/zap v1.28.0 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
 	golang.org/x/crypto v0.52.0 // indirect
+	golang.org/x/exp v0.0.0-20260527015227-08cc5374adb3 // indirect
 	golang.org/x/mod v0.36.0 // indirect
 	golang.org/x/net v0.55.0 // indirect
 	golang.org/x/sync v0.20.0 // indirect

docs/configuration.md

@@ -42,7 +42,7 @@ token:
 
   # When we are interested in knowing when a transaction reaches finality, we subscribe to the Finality Listener Manager for the finality event of that transaction.
   # This configuration specifies the way the manager is instantiated (i.e., how it gets notified about the finality events, how often it checks).
- finality:
+  finality:
     # Only applicable for fabric networks.
     # The manager subscribes to the delivery service and receives all final transactions.
     #   This manager keeps two structures: an LRU cache of recently finalized transactions, and a list of listeners that are waiting for future transactions.
@@ -209,6 +209,56 @@ token:
               # on every sweep until it either resolves or an operator intervenes.
               notFoundGracePeriod: 30m
 
+        # storage service configuration
+        storage:
+          # cleanup config controls automatic deletion of cryptographic keys from the keystore
+          # for tokens that have been deleted (spent, expired, or invalidated).
+          # If omitted, the cleanup manager uses its built-in defaults (disabled by default).
+          cleanup:
+            # enabled determines whether keystore cleanup runs. Default: false.
+            # Must be explicitly enabled. This is a conservative default to prevent
+            # unexpected key deletion in existing deployments.
+            enabled: false
+            
+            # ttl is the minimum age of deleted tokens before their keys are eligible for cleanup. Default: 24h.
+            # This ensures tokens are truly finalized before key deletion.
+            # Increase this value for additional safety margin in high-latency networks.
+            # Relationship: Should be significantly greater than transaction finality time.
+            ttl: 24h
+            
+            # scanInterval is how often the cleanup manager scans for deleted tokens. Default: 1h.
+            # Lower values provide faster cleanup but increase database load.
+            # Higher values reduce overhead but delay key removal.
+            # Relationship: Should be less than ttl to ensure timely cleanup.
+            # Performance impact: Each scan queries the token database for deleted tokens.
+            scanInterval: 1h
+            
+            # batchSize is the maximum number of deleted tokens processed per scan. Default: 100.
+            # Limits the number of tokens processed in a single cleanup sweep.
+            # Increase for high-volume environments with many deleted tokens.
+            # Performance impact: Larger batches reduce scan overhead but increase memory usage and processing time per sweep.
+            batchSize: 100
+            
+            # workerCount is the number of local workers that process tokens in parallel. Default: 1.
+            # Increase to improve cleanup throughput in high-volume scenarios.
+            # Decrease to reduce resource consumption on constrained systems.
+            # Performance impact: More workers increase CPU utilization during cleanup sweeps.
+            workerCount: 1
+            
+            # advisoryLockID is the PostgreSQL advisory lock identifier used for cleanup leader election.
+            # This ensures only one replica performs cleanup sweeps at a time in multi-instance deployments.
+            # Default: 8389190333894887277 (hex: 0x74746b636c65616e, ASCII: "ttkclean")
+            # The default value is derived from the ASCII encoding of "ttkclean" (Token Transaction Keystore Cleanup).
+            # Only change this if you need to run multiple independent cleanup managers on the same database.
+            # Note: PostgreSQL advisory locks use 64-bit integers. This value must be unique across your application.
+            advisoryLockID: 8389190333894887277
+            
+            # instanceID identifies this replica in logs and monitoring.
+            # If empty, a unique identifier is generated automatically at startup.
+            # Set this explicitly in containerized environments for consistent identity across restarts.
+            # This helps with debugging and tracking which instance performed cleanup operations.
+            instanceID:
+
       # auditor-specific settings
       auditor:
         # locker configures the distributed locking strategy for the auditor's
@@ -445,6 +495,79 @@ Default values:
    - Increase `workerCount` to 8-16 to improve parallel processing
    - Decrease `scanInterval` to 2-3s for faster recovery detection
 
+
+### Optional: token.tms.<name>.services.storage.cleanup
+
+If not specified, the default configuration is:
+
+```yaml
+token:
+  tms:
+    <name>:
+      services:
+        storage:
+          cleanup:
+            enabled: false
+            ttl: 24h
+            scanInterval: 1h
+            batchSize: 100
+            workerCount: 1
+            advisoryLockID: 8389190333894887277
+            instanceID:
+```
+
+Default values:
+
+- enabled: false
+- ttl: 24h
+- scanInterval: 1h
+- batchSize: 100
+- workerCount: 1
+- advisoryLockID: 8389190333894887277 (`0x74746b636c65616e`)
+- instanceID: empty, auto-generated when the cleanup manager starts
+
+**Parameter Relationships and Tuning:**
+
+- **Cleanup is disabled by default** and must be explicitly enabled. This is a conservative default to prevent unexpected key deletion in existing deployments.
+- **Only deleted tokens older than `ttl` are considered for cleanup** to ensure tokens are truly finalized before key deletion.
+- **The manager validates** that `ttl`, `scanInterval`, `batchSize`, and `workerCount` are all greater than zero.
+- **`advisoryLockID`** is used to acquire PostgreSQL advisory-lock leadership so that only one replica performs a cleanup sweep at a time. The default value (8389190333894887277 or 0x74746b636c65616e) represents the ASCII string "ttkclean" (Token Transaction Keystore Cleanup) encoded as a 64-bit integer.
+- **`instanceID`** is used to identify this replica in logs and monitoring; if omitted, the manager generates a unique identifier automatically at startup.
+
+**Tuning Recommendations:**
+
+1. **For High-Volume Environments:**
+   - Increase `batchSize` to 200-500 to process more tokens per sweep
+   - Increase `workerCount` to 8-16 to improve parallel key deletion
+   - Decrease `scanInterval` to 30m for more frequent cleanup
+
+2. **For Resource-Constrained Systems:**
+   - Decrease `workerCount` to 2 to reduce CPU usage
+   - Increase `scanInterval` to 2-4h to reduce database load
+   - Keep default `batchSize` to limit memory usage
+
+3. **For Security-Sensitive Deployments:**
+   - Decrease `ttl` to 12h for faster key removal
+   - Decrease `scanInterval` to 30m for more frequent cleanup
+   - Monitor cleanup metrics to ensure timely processing
+
+4. **For Multi-Instance Deployments:**
+   - **PostgreSQL Required**: Multi-instance deployments require PostgreSQL for distributed coordination via advisory locks
+   - Keep default `advisoryLockID` unless running multiple independent cleanup systems
+   - Consider setting explicit `instanceID` values for easier debugging and monitoring
+
+5. **For Single-Node Deployments:**
+   - **SQLite Supported**: SQLite can be used for single-node deployments and handles node restarts gracefully
+   - Cleanup works automatically after node restarts by scanning for eligible tokens
+   - **Important**: Do not use SQLite with multiple replicas as it lacks the advisory lock mechanism for leader election
+
+**Performance Considerations:**
+- Each scan queries the token database, so `scanInterval` directly affects database load
+- `workerCount` affects CPU utilization during cleanup sweeps
+- `batchSize` affects memory usage and the duration of each cleanup sweep
+- The relationship `scanInterval < ttl` ensures timely cleanup without premature processing
+
+---
 ---
 
 ### Optional: token.tms.<name>.auditor.lock

docs/evolution_summary.md

@@ -52,7 +52,7 @@ The Token Transaction service, a core component for orchestrating token lifecycl
     - **Network Integration:** Instantiated by both Fabric and FabricX network services
     - **Dual Database Support:** Operates on TTXDB for regular transactions and AuditDB for auditor nodes
 
-*   *See also:* [**TTX Service Documentation**](services/ttx.md), [**Transaction Recovery Service**](services/recovery.md)
+*   *See also:* [**TTX Service Documentation**](services/ttx.md), [**Transaction Recovery Service**](services/storage/recovery.md)
 
 ## Core Service Enhancements
 The SDK's foundational services have been matured:
@@ -63,7 +63,8 @@ The SDK's foundational services have been matured:
     - **Database Indexing:** Added missing indexes for token queries, significantly improving query performance
     - **Query Enhancements:** New `SearchDirection` support in `QueryTransactionsParams` for flexible result ordering
     - **Recovery Service Integration:** Built-in transaction recovery capabilities for handling finality listener failures
-    *   *See also:* [**Storage Service**](services/storage.md)
+    - **Keystore Cleanup Service:** Automatic deletion of cryptographic keys for deleted tokens, improving security and reducing storage overhead
+    *   *See also:* [**Storage Service**](services/storage.md), [**Keystore Cleanup Service**](services/storage/keystore_cleanup.md)
 - **Network Service:** Expanded to handle more complex Fabric network interactions and better integration with the Fabric Smart Client. A major change was the **removal of the Orion-based implementation**, which has been replaced by the introduction of **FabricX support**, providing a more modern and integrated approach for advanced ledger interactions.
     *   *See also:* [**Network Service**](services/network.md)
 

docs/imgs/storage_db.png

@@ -115,9 +115,18 @@ package "Token Store (TokenDB)" {
         created_at : TIMESTAMP
     }
 
+    entity "TokenSKICleanups" as tkn_ski_cleanups {
+        * tx_id : TEXT <<PK, NOT NULL, FK>>
+        * idx : INT <<PK, NOT NULL, FK>>
+        --
+        cleaned_at : TIMESTAMP <<NOT NULL>>
+        cleaned_by : TEXT <<NOT NULL>>
+    }
+
     tkn_own }o--|| tokens : "(tx_id, idx)"
     tkn_crts }o--|| tokens : "(tx_id, idx)"
     tkn_locks |o--o| tokens : "(tx_id, idx)"
+    tkn_ski_cleanups }o--|| tokens : "(tx_id, idx)"
 }
 
 package "Wallet & Identity Store (WalletDB / IdentityDB)" {

docs/imgs/storage_db.puml

@@ -39,6 +39,7 @@ This store serves as the authoritative registry for all tokens (UTXOs) known to
 *   **PublicParameters**: A cache for the network's cryptographic public parameters and their hashes.
 *   **TokenCertifications**: Stores third-party certifications for tokens, often required by privacy-preserving drivers.
 *   **TokenLocks**: Manages short-lived pessimistic locks on tokens to prevent double-spending during transaction assembly.
+*   **TokenSKICleanups**: Tracks keystore cleanup operations for deleted tokens. Records when cryptographic keys were removed from the keystore and which instance performed the cleanup, preventing reprocessing and enabling audit trails in multi-instance deployments.
 
 ### Wallet & Identity Store (WalletDB / IdentityDB)
 Manages the cryptographic identities and logical wallet groupings used by the node.
@@ -49,7 +50,7 @@ Manages the cryptographic identities and logical wallet groupings used by the no
 *   **IdentitySigners**: Tracks which identities have locally available signing keys and their associated metadata.
 
 ### Generic Store
-*   **KeyStore**: A secure, generic key-value store used for persisting various cryptographic materials and small sensitive states.
+*   **KeyStore**: A secure, generic key-value store used for persisting various cryptographic materials and small sensitive states. The keystore uses identifiers that are expected to be the hexadecimal representation of the key's Subject Key Identifier (SKI). This convention is relied upon by other packages, particularly for operations like keystore cleanup where SKIs are derived from owner identities to locate and manage keys.
 
 ## Internal Databases
 
@@ -88,7 +89,7 @@ This ensures that the local view of the "Token Landscape" always reflects the gr
 
 The Storage Service includes a **Transaction Recovery Service** that provides the core recovery mechanism for handling pending transactions that may have lost their finality listeners due to node restarts, network interruptions, or other failures.
 
-For detailed documentation on the recovery service architecture, configuration, and usage, see [**Transaction Recovery Service**](recovery.md).
+For detailed documentation on the recovery service architecture, configuration, and usage, see [**Transaction Recovery Service**](storage/recovery.md).
 
 ### Architecture
 
@@ -153,4 +154,55 @@ The recovery service supports both PostgreSQL and SQLite backends, with differen
 ### Configuration
 
 Recovery behavior is controlled by the `token.tms.<name>.services.network.fabric.recovery` configuration section. 
+
+## Keystore Cleanup Service
+
+The Storage Service includes a **Keystore Cleanup Service** that provides automatic deletion of cryptographic keys from the keystore for tokens that have been deleted (spent, expired, or invalidated). This ensures that the keystore doesn't accumulate stale keys indefinitely, improving security and reducing storage overhead.
+
+For detailed documentation on the cleanup service architecture, configuration, and usage, see [**Keystore Cleanup Service**](storage/keystore_cleanup.md).
+
+### Architecture
+
+The cleanup service operates on the token database and keystore, scanning for deleted tokens that are eligible for key cleanup. It uses a distributed locking mechanism (PostgreSQL advisory locks) to ensure only one replica in a multi-instance deployment performs cleanup at a time.
+
+### Cleanup Manager
+
+The cleanup manager runs in the background and periodically scans for deleted tokens whose cryptographic keys can be safely removed from the keystore.
+
+**Key Features:**
+- **Automatic Key Deletion**: Removes keys for deleted tokens after a configurable TTL period
+- **SKI Derivation**: Derives Subject Key Identifiers (SKIs) from owner identities to locate keys
+- **Multi-Database Support**:
+  - **PostgreSQL**: Recommended for production multi-instance deployments. Uses advisory locks for distributed coordination and leader election
+  - **SQLite**: Supported for single-node deployments and development. Handles node restarts gracefully but is not designed for multi-replica scenarios
+- **Configurable Behavior**: Cleanup parameters can be tuned via configuration (see [Configuration](../configuration.md))
+
+### Cleanup Process
+
+The cleanup service follows this workflow:
+
+1. **Leadership Acquisition**: The cleanup manager acquires an advisory lock to become the leader
+2. **Scan Phase**: Scans the token database for deleted tokens older than the configured TTL (default: 24 hours) that haven't had their keys cleaned
+3. **SKI Derivation**: For each eligible token, derives SKIs from the owner identity
+4. **Key Deletion**: Deletes the derived keys from the keystore using parallel workers
+5. **Tracking**: Marks tokens as cleaned in the database to prevent reprocessing
+
+### Database Backend Considerations
+
+The cleanup service supports both PostgreSQL and SQLite backends, with different characteristics:
+
+**PostgreSQL:**
+- **Multi-Instance Support**: Uses advisory locks for distributed coordination
+- **Leader Election**: Only one replica performs cleanup sweeps at a time
+- **High Availability**: Multiple replicas can share the same database
+
+**SQLite:**
+- **Single-Node Only**: Suitable for development and single-node deployments
+- **Node Restart Support**: Cleanup resumes automatically after restart
+- **No Multi-Replica Support**: Lacks advisory lock mechanism for leader election
+
+### Configuration
+
+Cleanup behavior is controlled by the configuration section. See the [Configuration Guide](../configuration.md) for detailed parameter descriptions and tuning recommendations.
+
 See the [Configuration Guide](../configuration.md), Section `Optional: token.tms.<name>.services.network.fabric.recovery`, for detailed parameter descriptions and tuning recommendations.