-
Notifications
You must be signed in to change notification settings - Fork 142
WIP :Use key-value-storage/policy-engine crates to make Trustee stateless
#1129
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
Xynnn007
wants to merge
10
commits into
confidential-containers:main
Choose a base branch
from
Xynnn007:kbs/stateless
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
203f170 to
8f5ade0
Compare
key-value-storage/policy-engine crates to make Trustee statelesskey-value-storage/policy-engine crates to make Trustee stateless
8f5ade0 to
0b8198b
Compare
To consolidate all backend storage configurations (postgres, local_json, local_fs) into a single configuration structure. This change prepares for centralizing all backend storage configuration sections in the config file. Changes: - Adds KeyValueStorageStructConfig with optional backend-specific config fields - Introduce ShimConfig for each backend to hold shared configuration - Add KeyValueStorageType enum to specify backend type - Replace to_key_value_storage() with to_client_with_instance() to support multiple instances per backend - Remove storage field from PolicyEngineConfig, accept storage instance directly - Add InvalidConfiguration error variant This refactoring enables future work to unify all backend storage configuration sections in the configuration file. Signed-off-by: Xynnn007 <[email protected]>
- Add StorageBackendConfig struct to support unified configuration across multiple components with instance-based data separation - Replace KeyValueStorageConfig enum with StorageBackendConfig - Add serde aliases for lowercase storage type variants - Make KeyValueStorageStructConfig fields public for better access - Add comprehensive documentation for unified storage backend configuration including instance concept, configuration structure, and backend-specific properties for LocalFs, LocalJson, Postgres, and Memory backends Signed-off-by: Xynnn007 <[email protected]>
- Remove KeyValueStorageConfig export (no longer needed) - Make PolicyEngine::new synchronous (storage is pre-initialized) Signed-off-by: Xynnn007 <[email protected]>
We now have a key-value implementation crate `key-value-storage`, this patch deletes original ReferenceValueStorage implementation and use the crate. Signed-off-by: Xynnn007 <[email protected]>
To improve code reusability and maintainability, the policy engine implementation has been extracted from attestation-service into policy-engine crate. This allows both kbs and attestation-service to share the same policy engine implementation, reducing code duplication and ensuring consistent behavior across components. At the same time, key-value-storage crate is introduced to provide unified storage backend abstraction. The original work_dir-based configuration is replaced with storage_backend configuration that supports multiple storage backends (LocalFs, LocalJson, etc.). Changes: - Remove internal policy_engine module (mod.rs, opa/mod.rs) - Update config to use PolicyEngineConfig from policy-engine crate - Add storage_backend configuration for unified storage management - Update RVPS config to use unified storage backend - Update documentation to reflect new configuration structure - Remove work_dir dependency for policy storage Once we bring in the crates, original work_dir will not work anymore. Signed-off-by: Xynnn007 <[email protected]>
This commit refactors KBS to use the external policy-engine crate instead of maintaining its own internal policy_engine module. At the same time, uses key-value-storage crate to replace localfs implementations for both policy storage and resource storage. The main reasons for this change are: 1. Code reuse: The policy-engine crate can be shared across multiple components (KBS, Attestation Service, etc.), reducing code duplication and maintenance burden. 2. Stateless design: By using the key-value-storage abstraction, the policy engine becomes stateless and can support multiple storage backends (local filesystem, local JSON, etc.), making it more flexible and suitable for different deployment scenarios. 3. Unified storage: The original fs storage backend for KBS resource was replaced by the kvstorage abstraction, providing a consistent storage interface across the codebase. 4. Better abstraction: The external crate provides a cleaner API with support for multiple policies management, which aligns with the stateless architecture goals. Changes: - Remove internal policy_engine module (error.rs, mod.rs, opa/mod.rs) - Move policy file from policy_engine/opa/default_policy.rego to policy/resource-policy.rego - Replace LocalFs resource storage with kv_storage implementation - Update config to use PolicyEngineConfig from policy-engine crate with key-value-storage configuration - Update API calls to use new policy-engine API (set_policy, list_policies, evaluate_rego) - Update Cargo.toml to depend on policy-engine and key-value-storage crates - Update test configurations to use new config format - Update policy-engine tests to use new data format This change maintains backward compatibility at the API level while improving the internal architecture. Signed-off-by: Xynnn007 <[email protected]>
…age migration After refactoring AS and KBS with new policy-engine and key-value-storage crates, the configuration formats have been changed. This commit updates the integration test module to adapt to the new configuration structure. Changes: - Add key-value-storage and policy-engine dependencies to Cargo.toml - Update test harness to use new storage_backend configuration format - Replace work_dir-based policy configuration with storage_backend config - Update RVPS configuration to use unified storage backend - Update KBS configuration to use new policy-engine and kv-storage config - Remove deprecated policy_dir and work_dir configurations - Update resource storage configuration to use kvstorage backend The integration tests now properly configure both AS and KBS with the new unified storage backend and policy engine architecture. Signed-off-by: Xynnn007 <[email protected]>
With modular policy storage, probing backend for an existing policy is costly. Luckily usually the KBS deployers need to set the resource policy to deny-all at first and then change it. Thus we can have a unified policy between kbs and trustee-cli. Changes: - Set default policy to deny_all for both KBS and trustee-cli - Unify policy files between KBS and trustee-cli by reusing sample policies - Move path relocate logic from trustee-cli to key-value-storage crate (different backends may have different file locations) - Remove original policy directory as policies are now reused - Update sample policy files (affirming.rego, allow_all.rego, deny_all.rego) with comprehensive documentation about policy input format and rules - Update allow_all.rego to default to deny with explicit allow rule This change improves performance by avoiding costly policy probing and simplifies policy management by having a single source of truth for policies. Signed-off-by: Xynnn007 <[email protected]>
Update the quickstart documentation to reflect the change in default resource policy file location. The policy file has been moved from src/policy_engine/opa/default_policy.rego to sample_policies/deny_all.rego as part of the migration to external policy-engine crate. Signed-off-by: Xynnn007 <[email protected]>
Update all deployment configuration files to use the new key-value-storage backend configuration format for the resource plugin. This change aligns with the migration to the key-value-storage crate abstraction layer. Changes: - Replace direct type/dir_path configuration with backend = "kvstorage" - Add storage_backend section with storage_type = "LocalFs" - Add storage_backend.backends.local_fs section with dir_path - Update storage path from /opt/confidential-containers/kbs/repository to /opt/confidential-containers/kbs Affected files: - docker-compose/kbs-config.toml - kubernetes/base/kbs-config.toml - kubernetes/ita/kbs-config.toml Signed-off-by: Xynnn007 <[email protected]>
0b8198b to
5ad2fd1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This a second part of #1093. Including the following changes
key-value-storageto replace RVPS's storage backendpolicy-engineto allow supporting extensionspolicy-engineto replace AS's policy engine implementationpolicy-engineto replace KBS's policy engine implementation. Move the KBS policy unit tests topolicy-enginecrate.integration-testsandtrustee-clicode to follow the mentioned underlying crates.README.mds.Config Changes
Hi @lmilleri @bpradipt , the main changes of the user interface are two
For RVPS, config format does not change.
For AS, config format would be changed to
{ "work_dir": "/opt/confidential-containers/attestation-service", "rvps_config": { "type": "GrpcRemote", "address": "http://127.0.0.1:50003" }, "attestation_token_broker": { "type": "Ear", "policy_engine": { "storage": { "type": "LocalFs", "dir_path": "/opt/confidential-containers/attestation-service/policies" } } }, "attestation_token_config": { "duration_min": 5 } }For KBS, config format would be changed to
And for KBS, the resource policy will not be a directly specified file. Instead, it will be the value that key
resource-policypoints to in the backend kv-storage. What does this mean? It means if we useLocalFslike in this example, the policy file should be the fileresource-policy.regounder/opt/confidential-containers/opapath, s.t. file/opt/confidential-containers/opa/resource-policy.rego.LocalJson.Before this PR, the reference values in the LocalJson backend file looks like
[ { "expiration": "1970-01-01T00:00:00Z", "name": "artifact1", "version": "1.0.0", "value": "abcd" }, { "expiration": "1970-01-01T00:00:00Z", "name": "artifact2", "version": "2.0.0", "value": "efgh" } ]But after this PR, it would like
{ "artifact1": "ewogICAgImV4cGlyYXRpb24iOiAiMTk3MC0wMS0wMVQwMDowMDowMFoiLAogICAgIm5hbWUiOiAiYXJ0aWZhY3QxIiwKICAgICJ2ZXJzaW9uIjogIjEuMC4wIiwKICAgICJ2YWx1ZSI6ICJhYmNkIgp9", ... }where
ewo..is base64-encoding of original{ "expiration": "1970-01-01T00:00:00Z", "name": "artifact1", "version": "1.0.0", "value": "abcd" }and the key is the
namefield.What's More
Hi @mkulke ,
The current PR still has over 1000 lines because each subcomponent (KBS/AS/RVPS) needs to update the global configuration README after replacing relevant parts. I want to be able to include all the README changes in a single PR to make it easier for TrusteeOperator folks a better view.
Not sure if it is acceptable - or do you have better idea to re-organize the PR for reviewing?