Skip to content

PHASE1_STATUS_AND_NEXT_STEPS.md

Latest commit

 

History

History
657 lines (517 loc) · 18.8 KB

PHASE1_STATUS_AND_NEXT_STEPS.md

File metadata and controls

657 lines (517 loc) · 18.8 KB

Phase 1: Vector Encryption Implementation - Status & Next Steps

Date: December 15, 2025
Status: Core encryption ready, integration pending
Estimated Time to Complete: 3-5 days

Summary

Phase 1 implements at-rest encryption for vector embeddings in RocksDB. The core encryption infrastructure (EncryptedField<std::vector<float>>) is complete with comprehensive test coverage. Integration with VectorIndexManager requires minimal surgical modifications to existing code paths.

✅ Completed Work

1. Encryption Infrastructure (Commit 7817227)

File: src/security/encrypted_field.cpp

  • Added template specialization for std::vector<float>
  • Binary serialization: [uint32_t size][float[]] format
  • Full float precision preservation (no quantization loss)
  • Integration with existing FieldEncryption engine

Code:

template<>
std::string EncryptedField<std::vector<float>>::serialize(const std::vector<float>& value) {
    std::string result;
    uint32_t size = static_cast<uint32_t>(value.size());
    result.append(reinterpret_cast<const char*>(&size), sizeof(uint32_t));
    if (!value.empty()) {
        result.append(reinterpret_cast<const char*>(value.data()), 
                     value.size() * sizeof(float));
    }
    return result;
}

Encryption Flow:

Vector [0.1, 0.2, ...]
    ↓ serialize()
Binary [00 00 00 02 | 3D CC CC CD | 3E 4C CC CD]
    ↓ FieldEncryption::encrypt()
AES-256-GCM(binary, key, random_iv) + auth_tag
    ↓ EncryptedBlob::toBase64()
"vector_embeddings:1:iv_base64:ciphertext_base64:tag_base64"
    ↓ RocksDB
Stored on disk (encrypted)

2. Comprehensive Test Suite (Commit 7817227)

File: tests/test_vector_encryption_phase1.cpp (400+ lines)

15 test cases covering:

  • ✅ Basic encrypt/decrypt roundtrip
  • ✅ Empty vector handling
  • ✅ Large embeddings (768-dim, 1536-dim)
  • ✅ Float precision preservation
  • ✅ Base64/JSON serialization
  • ✅ Error handling (missing encryption, corrupted data)
  • ✅ Key versioning for rotation
  • ✅ IV randomness validation
  • ✅ Performance benchmarks
  • ✅ Normalized vectors (L2-norm preservation)
  • ✅ Sparse vectors
  • ✅ Float edge cases (NaN, Infinity, -0.0)

Performance Targets:

  • Encryption: < 1 ms per vector (768-dim)
  • Decryption: < 1 ms per vector
  • Throughput: > 1,000 vectors/sec (single-threaded)

3. Implementation Plan (Commit 7817227)

File: docs/security/PHASE1_IMPLEMENTATION_PLAN.md (24KB)

Complete 6-week rollout plan with:

  • Architecture diagrams (current vs. target state)
  • Code modification strategy
  • Performance analysis
  • Migration approach
  • BSI C5 compliance impact
  • Risk assessment

⏳ Remaining Work

Step 1: Modify VectorIndexManager::addEntity() (CRITICAL PATH)

File: src/index/vector_index.cpp (Lines 299-387)

Current Code:

VectorIndexManager::Status VectorIndexManager::addEntity(
    const BaseEntity& e, 
    std::string_view vectorField
) {
    // ... existing code ...
    
    // Current: Store raw or SQ8-quantized vector
    if (shouldQuantize) {
        // SQ8 quantization path
        auto fields = e.getAllFields();
        fields.erase("embedding");
        fields["embedding_q"] = codes;
        fields["embedding_scale"] = static_cast<double>(scale);
        BaseEntity eq = BaseEntity::fromFields(pk, fields);
        serialized = eq.serialize();
    } else {
        // Raw storage
        serialized = e.serialize();
    }
    
    db_.put(key, serialized);
}

Proposed Modification:

VectorIndexManager::Status VectorIndexManager::addEntity(
    const BaseEntity& e, 
    std::string_view vectorField
) {
    // ... existing code (keep unchanged) ...
    
    // NEW: Check if encryption is enabled
    bool encryptVectors = isVectorEncryptionEnabled();
    
    if (encryptVectors) {
        // NEW PATH: Store encrypted vector
        auto fields = e.getAllFields();
        fields.erase("embedding");  // Remove plaintext
        
        // Encrypt using EncryptedField
        EncryptedField<std::vector<float>> enc_emb;
        enc_emb.encrypt(*v, "vector_embeddings");
        fields["embedding_encrypted"] = enc_emb.toBase64();
        
        BaseEntity eq = BaseEntity::fromFields(pk, fields);
        serialized = eq.serialize();
    } else {
        // EXISTING PATHS: SQ8 or raw (keep unchanged)
        if (shouldQuantize) {
            // ... existing SQ8 code ...
        } else {
            serialized = e.serialize();
        }
    }
    
    db_.put(key, serialized);
    // ... rest unchanged ...
}

Configuration Check:

bool VectorIndexManager::isVectorEncryptionEnabled() const {
    try {
        if (auto cfg = db_.get("config:vector")) {
            std::string s(cfg->begin(), cfg->end());
            nlohmann::json j = nlohmann::json::parse(s);
            return j.value("encryption_enabled", false);
        }
    } catch (...) {}
    return false;  // Default: disabled (backward compatible)
}

Step 2: Modify VectorIndexManager::rebuildFromStorage() (CRITICAL PATH)

File: src/index/vector_index.cpp (Lines 231-297)

Current Code:

VectorIndexManager::Status VectorIndexManager::rebuildFromStorage() {
    // ... scan RocksDB ...
    db_.scanPrefix(prefix, [&](std::string_view key, std::string_view value) {
        BaseEntity e = BaseEntity::deserialize(pk, bytes);
        
        // Current: Extract plaintext or SQ8 vector
        auto vecOpt = e.extractVector("embedding");
        if (vecOpt) {
            v = *vecOpt;
        } else {
            // Decode SQ8
            // ...
        }
        
        // ... HNSW index building ...
    });
}

Proposed Modification:

VectorIndexManager::Status VectorIndexManager::rebuildFromStorage() {
    // ... scan RocksDB (unchanged) ...
    
    db_.scanPrefix(prefix, [&](std::string_view key, std::string_view value) {
        BaseEntity e = BaseEntity::deserialize(pk, bytes);
        
        std::vector<float> v;
        
        // NEW: Try encrypted field first
        auto encFieldOpt = e.getField("embedding_encrypted");
        if (encFieldOpt) {
            try {
                auto enc = EncryptedField<std::vector<float>>::fromBase64(
                    std::get<std::string>(*encFieldOpt)
                );
                v = enc.decrypt();  // Batch decrypt (parallel later)
            } catch (...) {
                THEMIS_WARN("Failed to decrypt vector for pk={}", pk);
                return true;  // Skip this entity
            }
        } 
        // EXISTING: Try plaintext (backward compat)
        else if (auto vecOpt = e.extractVector("embedding"); vecOpt) {
            v = *vecOpt;
        }
        // EXISTING: Try SQ8 (backward compat)
        else {
            // ... existing SQ8 decode code (unchanged) ...
        }
        
        // ... rest unchanged (HNSW indexing) ...
    });
}

Step 3: Batch Decryption Optimization (PERFORMANCE)

File: src/index/vector_index.cpp

Concept:

// Instead of: decrypt each vector serially
for (auto& entity : entities) {
    v = enc.decrypt();  // Sequential: 1ms × 1M = 16 minutes!
}

// Do: batch decrypt with parallelization
std::vector<EncryptedField<std::vector<float>>> encrypted_batch;
// ... collect all encrypted fields ...

// Parallel decrypt with TBB (8 cores)
std::vector<std::vector<float>> decrypted_batch(encrypted_batch.size());
tbb::parallel_for(size_t(0), encrypted_batch.size(), [&](size_t i) {
    decrypted_batch[i] = encrypted_batch[i].decrypt();
});
// Parallel: 1ms × 1M / 8 cores = 2 minutes (8x faster)

Implementation:

// Add to rebuildFromStorage()
Status VectorIndexManager::rebuildFromStorage() {
    // ... scan and collect encrypted entities ...
    
    struct PendingEntity {
        std::string pk;
        EncryptedField<std::vector<float>> encrypted;
    };
    std::vector<PendingEntity> pending;
    
    db_.scanPrefix(prefix, [&](std::string_view key, std::string_view value) {
        // ... parse entity ...
        if (auto encField = e.getField("embedding_encrypted"); encField) {
            pending.push_back({pk, EncryptedField<std::vector<float>>::fromBase64(...)});
        }
        return true;
    });
    
    // Parallel batch decryption
    std::vector<std::vector<float>> decrypted(pending.size());
    
    #pragma omp parallel for  // Or use TBB
    for (size_t i = 0; i < pending.size(); ++i) {
        try {
            decrypted[i] = pending[i].encrypted.decrypt();
        } catch (...) {
            THEMIS_WARN("Decrypt failed for entity {}", i);
        }
    }
    
    // Build HNSW index from decrypted vectors
    for (size_t i = 0; i < pending.size(); ++i) {
        const auto& pk = pending[i].pk;
        const auto& v = decrypted[i];
        
        if (metric_ == Metric::COSINE) normalizeL2(v);
        cache_[pk] = v;
        
        // ... HNSW addPoint (unchanged) ...
    }
    
    return Status::OK();
}

Step 4: Configuration & Feature Flag (SAFETY)

File: Create src/index/vector_encryption_config.cpp

namespace themis {

class VectorEncryptionConfig {
public:
    static VectorEncryptionConfig& getInstance() {
        static VectorEncryptionConfig instance;
        return instance;
    }
    
    bool isEnabled() const {
        std::lock_guard<std::mutex> lock(mutex_);
        return enabled_;
    }
    
    void setEnabled(bool enabled) {
        std::lock_guard<std::mutex> lock(mutex_);
        enabled_ = enabled;
        THEMIS_INFO("Vector encryption: {}", enabled ? "ENABLED" : "DISABLED");
    }
    
    std::string getKeyId() const {
        std::lock_guard<std::mutex> lock(mutex_);
        return key_id_;
    }
    
    void setKeyId(const std::string& key_id) {
        std::lock_guard<std::mutex> lock(mutex_);
        key_id_ = key_id;
    }
    
private:
    mutable std::mutex mutex_;
    bool enabled_ = false;  // Default: OFF for backward compatibility
    std::string key_id_ = "vector_embeddings";
    
    VectorEncryptionConfig() = default;
};

}  // namespace themis

Usage in VectorIndexManager:

bool VectorIndexManager::isVectorEncryptionEnabled() const {
    return VectorEncryptionConfig::getInstance().isEnabled();
}

HTTP API to enable:

// POST /api/config/vector/encryption
{
    "enabled": true,
    "key_id": "vector_embeddings"
}

Step 5: Migration Tool (OPERATIONS)

File: Create tools/migrate_vector_encryption.cpp

int main(int argc, char** argv) {
    // Parse args: --db-path, --object-name, --batch-size
    
    RocksDBWrapper db(db_path);
    VectorIndexManager vim(db);
    vim.init(object_name, dim);
    
    // Scan all vectors
    std::vector<std::string> pks_to_migrate;
    db.scanPrefix(object_name + ":", [&](auto key, auto value) {
        std::string pk = KeySchema::extractPrimaryKey(key);
        BaseEntity e = BaseEntity::deserialize(pk, value);
        
        // Check if already encrypted
        if (!e.getField("embedding_encrypted")) {
            pks_to_migrate.push_back(pk);
        }
        return true;
    });
    
    THEMIS_INFO("Found {} vectors to migrate", pks_to_migrate.size());
    
    // Migrate in batches
    size_t batch_size = 1000;
    for (size_t i = 0; i < pks_to_migrate.size(); i += batch_size) {
        auto batch = db.createWriteBatch();
        
        for (size_t j = i; j < std::min(i + batch_size, pks_to_migrate.size()); ++j) {
            const auto& pk = pks_to_migrate[j];
            
            // Read existing entity
            auto blob = db.get(object_name + ":" + pk);
            BaseEntity e = BaseEntity::deserialize(pk, *blob);
            
            // Extract plaintext vector
            auto v = e.extractVector("embedding");
            if (!v) continue;
            
            // Encrypt
            EncryptedField<std::vector<float>> enc;
            enc.encrypt(*v, "vector_embeddings");
            
            // Store encrypted
            auto fields = e.getAllFields();
            fields.erase("embedding");  // Remove plaintext
            fields["embedding_encrypted"] = enc.toBase64();
            
            BaseEntity new_e = BaseEntity::fromFields(pk, fields);
            batch->put(object_name + ":" + pk, new_e.serialize());
        }
        
        if (!batch->commit()) {
            THEMIS_ERROR("Batch {} failed", i / batch_size);
            return 1;
        }
        
        THEMIS_INFO("Migrated batch {}/{}", i / batch_size + 1, 
                    (pks_to_migrate.size() + batch_size - 1) / batch_size);
    }
    
    THEMIS_INFO("Migration complete!");
    return 0;
}

Step 6: Integration Tests (VALIDATION)

File: Create tests/test_vector_encryption_integration.cpp

TEST_CASE("VectorIndexManager with encryption enabled", "[vector][encryption]") {
    // Setup
    RocksDBWrapper db("/tmp/test_vec_enc");
    VectorEncryptionConfig::getInstance().setEnabled(true);
    
    VectorIndexManager vim(db);
    vim.init("documents", 768);
    
    SECTION("Add encrypted vector") {
        std::vector<float> embedding(768, 0.5f);
        BaseEntity e = BaseEntity::fromFields("doc1", {{"embedding", embedding}});
        
        auto status = vim.addEntity(e);
        REQUIRE(status.ok);
        
        // Verify encrypted in storage
        auto blob = db.get("documents:doc1");
        REQUIRE(blob.has_value());
        BaseEntity stored = BaseEntity::deserialize("doc1", *blob);
        
        // Should have encrypted field, not plaintext
        REQUIRE(stored.getField("embedding_encrypted").has_value());
        REQUIRE_FALSE(stored.getField("embedding").has_value());
    }
    
    SECTION("Search encrypted vectors") {
        // Add 100 encrypted vectors
        for (int i = 0; i < 100; ++i) {
            std::vector<float> emb(768);
            for (int j = 0; j < 768; ++j) {
                emb[j] = static_cast<float>(i + j) / 1000.0f;
            }
            BaseEntity e = BaseEntity::fromFields(
                "doc" + std::to_string(i), 
                {{"embedding", emb}}
            );
            vim.addEntity(e);
        }
        
        // Rebuild from encrypted storage
        vim.rebuildFromStorage();
        
        // Search should work normally
        std::vector<float> query(768, 0.5f);
        auto [st, results] = vim.searchKnn(query, 10);
        
        REQUIRE(st.ok);
        REQUIRE(results.size() == 10);
        REQUIRE(results[0].distance < results[9].distance);
    }
    
    SECTION("Backward compatibility: read plaintext") {
        VectorEncryptionConfig::getInstance().setEnabled(false);
        
        // Add plaintext vector
        std::vector<float> emb(768, 0.3f);
        BaseEntity e = BaseEntity::fromFields("doc_plain", {{"embedding", emb}});
        vim.addEntity(e);
        
        // Enable encryption
        VectorEncryptionConfig::getInstance().setEnabled(true);
        
        // Rebuild should handle both encrypted and plaintext
        vim.rebuildFromStorage();
        
        // Search should work
        auto [st, results] = vim.searchKnn(emb, 1);
        REQUIRE(st.ok);
        REQUIRE(results.size() == 1);
        REQUIRE(results[0].pk == "doc_plain");
    }
}

Implementation Timeline

Week 1 (Days 1-2):

  • Implement addEntity() modifications
  • Implement rebuildFromStorage() modifications
  • Add VectorEncryptionConfig class
  • Build and run unit tests

Week 1 (Days 3-5):

  • Implement batch decryption optimization
  • Create migration tool
  • Write integration tests
  • Performance benchmarking

Week 2:

  • Code review
  • Security audit
  • Documentation updates
  • Production rollout plan

Performance Impact

Storage Overhead:

Plaintext 768-dim vector:  3,072 bytes
Encrypted 768-dim vector:  3,150 bytes (+2.5%)

Components:
- Plaintext:     3,072 bytes (768 × 4)
- IV:               12 bytes
- Auth tag:         16 bytes
- Metadata:        ~50 bytes (key_id, version, base64 encoding)
Total:           3,150 bytes

Index Load Time (1M vectors):

Current (plaintext):     2 seconds
With encryption:         5 seconds (+3 sec for decryption)
vs. Rebuild from scratch: 300 seconds (5 minutes)

Speedup with encryption: 60x faster than rebuild

Query Performance:

No impact - vectors decrypted at index load time
HNSW search operates on plaintext in memory
O(log n) complexity maintained

BSI C5 Compliance Impact

Before Phase 1:

  • Vector embeddings: ❌ Plaintext on disk
  • CRY-03 (Data-at-Rest): ⚠️ Conditionally compliant
  • Overall: 90% (5.5/6 models)

After Phase 1:

  • Vector embeddings: ✅ AES-256-GCM encrypted
  • CRY-03 (Data-at-Rest): ✅ Fully compliant
  • Overall: 95% (6/6 models encrypted)

Remaining Gap (Phase 2):

  • HNSW persistence still stores plaintext
  • Need to encrypt data/hnsw_chunks/index.bin
  • Estimated 4 weeks for Phase 2

Risks & Mitigations

Risk Probability Impact Mitigation
Performance degradation Medium High Batch decryption with parallelization (8x speedup)
Migration failure Low High Incremental batch migration with rollback support
Key rotation issues Low Medium Test with multiple key versions
Backward incompatibility Low High Feature flag + dual-path support (encrypted + plaintext)

Next Actions

Immediate (This Week):

  1. Review this document with security team
  2. Get approval for code modifications
  3. Begin implementation of addEntity() modifications
  4. Set up build environment for testing

Week 2:

  1. Complete integration
  2. Run full test suite
  3. Performance benchmarking
  4. Code review

Week 3:

  1. Documentation finalization
  2. Migration tool testing
  3. Production readiness review
  4. Deployment planning

Questions for Review

  1. Feature Flag: Should vector encryption be opt-in or opt-out?

    • Recommendation: Opt-in (default disabled for backward compatibility)

  2. Migration Strategy: Should we auto-migrate on first startup or require manual migration?

    • Recommendation: Manual migration via CLI tool (safer, more control)

  3. Batch Size: What batch size for parallel decryption?

    • Recommendation: 10,000 vectors (balances memory usage and parallelism)

  4. Key Rotation: How often should we rotate vector encryption keys?

    • Recommendation: Quarterly (aligns with BSI C5 requirements)

  5. Monitoring: What metrics should we track?

    • Recommendation: Encrypt/decrypt latency, throughput, key version distribution

References

  • Implementation Plan: docs/security/PHASE1_IMPLEMENTATION_PLAN.md
  • Test Suite: tests/test_vector_encryption_phase1.cpp
  • BSI C5 Analysis: docs/security/BSI_C5_COLUMN_ENCRYPTION_COMPLIANCE.md
  • Encryption Design: docs/security/security_column_encryption.md

Status: Ready for implementation
Owner: Security Team
Timeline: 2-3 weeks for complete Phase 1
Priority: P0 (Critical for BSI C5 compliance)