Skip to content

Conversation

@Xynnn007
Copy link
Member

@Xynnn007 Xynnn007 commented Dec 24, 2025

Summary

This a second part of #1093. Including the following changes

  • Use key-value-storage to replace RVPS's storage backend
  • A Refactoring upon policy-engine to allow supporting extensions
  • Use policy-engine to replace AS's policy engine implementation
  • Use policy-engine to replace KBS's policy engine implementation. Move the KBS policy unit tests to policy-engine crate.
  • Fix integration-tests and trustee-cli code to follow the mentioned underlying crates.
  • Update the README.mds.

Config Changes

Hi @lmilleri @bpradipt , the main changes of the user interface are two

  1. AS/KBS Config format is included in the commit ; RVPS Config format is included in the commit.

For RVPS, config format does not change.

For AS, config format would be changed to

{
        "work_dir": "/opt/confidential-containers/attestation-service",
        "rvps_config": {
          "type": "GrpcRemote",
          "address": "http://127.0.0.1:50003"
        },
        "attestation_token_broker": {
          "type": "Ear",
          "policy_engine": {
            "storage": {
              "type": "LocalFs",
              "dir_path": "/opt/confidential-containers/attestation-service/policies"
            }
          }
        },
        "attestation_token_config": {
          "duration_min": 5
        }
    }

For KBS, config format would be changed to

[http_server]
sockets = ["0.0.0.0:8080"]
insecure_http = true
worker_count = 4

[admin]
insecure_api = true
auth_public_key = "/etc/auth-secret/kbs.pem"

[attestation_token]
insecure_key = true

[attestation_service]
type = "coco_as_grpc"
as_addr = "http://127.0.0.1:50004"

[[plugins]]
name = "resource"
backend = "kvstorage"
type = "LocalFs"
dir_path = "/opt/confidential-containers/kbs/repository"

[policy_engine.storage]
type = "LocalFs"
dir_path = "/opt/confidential-containers/opa"

And for KBS, the resource policy will not be a directly specified file. Instead, it will be the value that key resource-policy points to in the backend kv-storage. What does this mean? It means if we use LocalFs like in this example, the policy file should be the file resource-policy.rego under /opt/confidential-containers/opa path, s.t. file /opt/confidential-containers/opa/resource-policy.rego.

  1. The format of LocalJson.

Before this PR, the reference values in the LocalJson backend file looks like

[
    {
        "expiration": "1970-01-01T00:00:00Z",
        "name": "artifact1",
        "version": "1.0.0",
        "value": "abcd"
    },
    {
        "expiration": "1970-01-01T00:00:00Z",
        "name": "artifact2",
        "version": "2.0.0",
        "value": "efgh"
    }
]

But after this PR, it would like

{
    "artifact1": "ewogICAgImV4cGlyYXRpb24iOiAiMTk3MC0wMS0wMVQwMDowMDowMFoiLAogICAgIm5hbWUiOiAiYXJ0aWZhY3QxIiwKICAgICJ2ZXJzaW9uIjogIjEuMC4wIiwKICAgICJ2YWx1ZSI6ICJhYmNkIgp9",
    ...
}

where ewo.. is base64-encoding of original

{
      "expiration": "1970-01-01T00:00:00Z",
      "name": "artifact1",
      "version": "1.0.0",
      "value": "abcd"
}

and the key is the name field.

What's More

Hi @mkulke ,

The current PR still has over 1000 lines because each subcomponent (KBS/AS/RVPS) needs to update the global configuration README after replacing relevant parts. I want to be able to include all the README changes in a single PR to make it easier for TrusteeOperator folks a better view.

Not sure if it is acceptable - or do you have better idea to re-organize the PR for reviewing?

@Xynnn007 Xynnn007 force-pushed the kbs/stateless branch 2 times, most recently from 203f170 to 8f5ade0 Compare December 24, 2025 08:29
@Xynnn007 Xynnn007 changed the title Use key-value-storage/policy-engine crates to make Trustee stateless WIP :Use key-value-storage/policy-engine crates to make Trustee stateless Dec 24, 2025
To consolidate all backend storage configurations (postgres,
local_json, local_fs) into a single configuration structure. This
change prepares for centralizing all backend storage configuration
sections in the config file.

Changes:
- Adds KeyValueStorageStructConfig with optional backend-specific
  config fields
- Introduce ShimConfig for each backend to hold shared configuration
- Add KeyValueStorageType enum to specify backend type
- Replace to_key_value_storage() with to_client_with_instance() to
  support multiple instances per backend
- Remove storage field from PolicyEngineConfig, accept storage instance
  directly
- Add InvalidConfiguration error variant

This refactoring enables future work to unify all backend storage
configuration sections in the configuration file.

Signed-off-by: Xynnn007 <[email protected]>
- Add StorageBackendConfig struct to support unified configuration
  across multiple components with instance-based data separation
- Replace KeyValueStorageConfig enum with StorageBackendConfig
- Add serde aliases for lowercase storage type variants
- Make KeyValueStorageStructConfig fields public for better access
- Add comprehensive documentation for unified storage backend
  configuration including instance concept, configuration structure,
  and backend-specific properties for LocalFs, LocalJson, Postgres,
  and Memory backends

Signed-off-by: Xynnn007 <[email protected]>
- Remove KeyValueStorageConfig export (no longer needed)
- Make PolicyEngine::new synchronous (storage is pre-initialized)

Signed-off-by: Xynnn007 <[email protected]>
We now have a key-value implementation crate `key-value-storage`, this
patch deletes original ReferenceValueStorage implementation and use the
crate.

Signed-off-by: Xynnn007 <[email protected]>
To improve code reusability and maintainability, the policy engine
implementation has been extracted from attestation-service into
policy-engine crate. This allows both kbs and attestation-service
to share the same policy engine implementation, reducing code
duplication and ensuring consistent behavior across components.

At the same time, key-value-storage crate is introduced to provide
unified storage backend abstraction. The original work_dir-based
configuration is replaced with storage_backend configuration that
supports multiple storage backends (LocalFs, LocalJson, etc.).

Changes:
- Remove internal policy_engine module (mod.rs, opa/mod.rs)
- Update config to use PolicyEngineConfig from policy-engine crate
- Add storage_backend configuration for unified storage management
- Update RVPS config to use unified storage backend
- Update documentation to reflect new configuration structure
- Remove work_dir dependency for policy storage

Once we bring in the crates, original work_dir will not work anymore.

Signed-off-by: Xynnn007 <[email protected]>
This commit refactors KBS to use the external policy-engine crate instead
of maintaining its own internal policy_engine module. At the same time,
uses key-value-storage crate to replace localfs implementations for both
policy storage and resource storage. The main reasons for this change are:

1. Code reuse: The policy-engine crate can be shared across multiple
   components (KBS, Attestation Service, etc.), reducing code duplication
   and maintenance burden.

2. Stateless design: By using the key-value-storage abstraction, the
   policy engine becomes stateless and can support multiple storage
   backends (local filesystem, local JSON, etc.), making it more flexible
   and suitable for different deployment scenarios.

3. Unified storage: The original fs storage backend for KBS resource was
   replaced by the kvstorage abstraction, providing a consistent storage
   interface across the codebase.

4. Better abstraction: The external crate provides a cleaner API with
   support for multiple policies management, which aligns with the
   stateless architecture goals.

Changes:
- Remove internal policy_engine module (error.rs, mod.rs, opa/mod.rs)
- Move policy file from policy_engine/opa/default_policy.rego to
  policy/resource-policy.rego
- Replace LocalFs resource storage with kv_storage implementation
- Update config to use PolicyEngineConfig from policy-engine crate with
  key-value-storage configuration
- Update API calls to use new policy-engine API (set_policy, list_policies,
  evaluate_rego)
- Update Cargo.toml to depend on policy-engine and key-value-storage crates
- Update test configurations to use new config format
- Update policy-engine tests to use new data format

This change maintains backward compatibility at the API level while
improving the internal architecture.

Signed-off-by: Xynnn007 <[email protected]>
…age migration

After refactoring AS and KBS with new policy-engine and key-value-storage
crates, the configuration formats have been changed. This commit updates
the integration test module to adapt to the new configuration structure.

Changes:
- Add key-value-storage and policy-engine dependencies to Cargo.toml
- Update test harness to use new storage_backend configuration format
- Replace work_dir-based policy configuration with storage_backend config
- Update RVPS configuration to use unified storage backend
- Update KBS configuration to use new policy-engine and kv-storage config
- Remove deprecated policy_dir and work_dir configurations
- Update resource storage configuration to use kvstorage backend

The integration tests now properly configure both AS and KBS with the new
unified storage backend and policy engine architecture.

Signed-off-by: Xynnn007 <[email protected]>
With modular policy storage, probing backend for an existing policy is
costly. Luckily usually the KBS deployers need to set the resource
policy to deny-all at first and then change it. Thus we can have a
unified policy between kbs and trustee-cli.

Changes:
- Set default policy to deny_all for both KBS and trustee-cli
- Unify policy files between KBS and trustee-cli by reusing sample policies
- Move path relocate logic from trustee-cli to key-value-storage crate
  (different backends may have different file locations)
- Remove original policy directory as policies are now reused
- Update sample policy files (affirming.rego, allow_all.rego, deny_all.rego)
  with comprehensive documentation about policy input format and rules
- Update allow_all.rego to default to deny with explicit allow rule

This change improves performance by avoiding costly policy probing and
simplifies policy management by having a single source of truth for
policies.

Signed-off-by: Xynnn007 <[email protected]>
Update the quickstart documentation to reflect the change in default
resource policy file location. The policy file has been moved from
src/policy_engine/opa/default_policy.rego to sample_policies/deny_all.rego
as part of the migration to external policy-engine crate.

Signed-off-by: Xynnn007 <[email protected]>
Update all deployment configuration files to use the new key-value-storage
backend configuration format for the resource plugin. This change aligns
with the migration to the key-value-storage crate abstraction layer.

Changes:
- Replace direct type/dir_path configuration with backend = "kvstorage"
- Add storage_backend section with storage_type = "LocalFs"
- Add storage_backend.backends.local_fs section with dir_path
- Update storage path from /opt/confidential-containers/kbs/repository
  to /opt/confidential-containers/kbs

Affected files:
- docker-compose/kbs-config.toml
- kubernetes/base/kbs-config.toml
- kubernetes/ita/kbs-config.toml

Signed-off-by: Xynnn007 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant