Skip to content

Latest commit

 

History

History
270 lines (215 loc) · 7.76 KB

docs.md

File metadata and controls

270 lines (215 loc) · 7.76 KB

Capabilities

This service can be used to:

  • create_dir
  • stat
  • read
  • write
  • delete
  • list
  • copy
  • rename
  • presign

Configuration

  • root: Set the work dir for backend.
  • bucket: Set the container name for backend.
  • endpoint: Set the endpoint for backend.
  • region: Set the region for backend.
  • access_key_id: Set the access_key_id for backend.
  • secret_access_key: Set the secret_access_key for backend.
  • session_token: Set the session_token for backend.
  • default_storage_class: Set the default storage_class for backend.
  • server_side_encryption: Set the server_side_encryption for backend.
  • server_side_encryption_aws_kms_key_id: Set the server_side_encryption_aws_kms_key_id for backend.
  • server_side_encryption_customer_algorithm: Set the server_side_encryption_customer_algorithm for backend.
  • server_side_encryption_customer_key: Set the server_side_encryption_customer_key for backend.
  • server_side_encryption_customer_key_md5: Set the server_side_encryption_customer_key_md5 for backend.
  • disable_config_load: Disable aws config load from env.
  • enable_virtual_host_style: Enable virtual host style.
  • disable_write_with_if_match: Disable write with if match.
  • enable_request_payer: Enable the request payer for backend.
  • default_acl: Define the default access control list (ACL) when creating a new object. Note that some s3 services like minio do not support this option.

Refer to [S3Builder]'s public API docs for more information.

Temporary security credentials

OpenDAL now provides support for S3 temporary security credentials in IAM.

The way to take advantage of this feature is to build your S3 backend with Builder::session_token.

But OpenDAL will not refresh the temporary security credentials, please keep in mind to refresh those credentials in time.

Server Side Encryption

OpenDAL provides full support of S3 Server Side Encryption(SSE) features.

The easiest way to configure them is to use helper functions like

  • SSE-KMS: server_side_encryption_with_aws_managed_kms_key
  • SSE-KMS: server_side_encryption_with_customer_managed_kms_key
  • SSE-S3: server_side_encryption_with_s3_key
  • SSE-C: server_side_encryption_with_customer_key

If those functions don't fulfill need, low-level options are also provided:

  • Use service managed kms key
    • server_side_encryption="aws:kms"
  • Use customer provided kms key
    • server_side_encryption="aws:kms"
    • server_side_encryption_aws_kms_key_id="your-kms-key"
  • Use S3 managed key
    • server_side_encryption="AES256"
  • Use customer key
    • server_side_encryption_customer_algorithm="AES256"
    • server_side_encryption_customer_key="base64-of-your-aes256-key"
    • server_side_encryption_customer_key_md5="base64-of-your-aes256-key-md5"

After SSE have been configured, all requests send by this backed will attach those headers.

Reference: Protecting data using server-side encryption

Example

Via Builder

Basic Setup

use opendal_core::Operator;
use opendal_core::Result;
use opendal_service_s3::S3;

#[tokio::main]
async fn main() -> Result<()> {
    // Create s3 backend builder.
    let mut builder = S3::default()
      // Set the root for s3, all operations will happen under this root.
      //
      // NOTE: the root must be absolute path.
      .root("/path/to/dir")
      // Set the bucket name. This is required.
      .bucket("test")
      // Set the region. This is required for some services, if you don't care about it, for example Minio service, just set it to "auto", it will be ignored.
      .region("us-east-1")
      // Set the endpoint.
      //
      // For examples:
      // - "https://s3.amazonaws.com"
      // - "http://127.0.0.1:9000"
      // - "https://oss-ap-northeast-1.aliyuncs.com"
      // - "https://cos.ap-seoul.myqcloud.com"
      //
      // Default to "https://s3.amazonaws.com"
      .endpoint("https://s3.amazonaws.com")
      // Set the access_key_id and secret_access_key.
      //
      // OpenDAL will try load credential from the env.
      // If credential not set and no valid credential in env, OpenDAL will
      // send request without signing like anonymous user.
      .access_key_id("access_key_id")
      .secret_access_key("secret_access_key");

    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

S3 with SSE-C

use log::info;
use opendal_core::Operator;
use opendal_core::Result;
use opendal_service_s3::S3;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default()
      .root("/path/to/dir")
      .bucket("test")
      .region("us-east-1")
      .endpoint("https://s3.amazonaws.com")
      .access_key_id("access_key_id")
      .secret_access_key("secret_access_key")
      // Enable SSE-C
      .server_side_encryption_with_customer_key("AES256", "customer_key".as_bytes());

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-KMS and aws managed kms key

use log::info;
use opendal_core::Operator;
use opendal_core::Result;
use opendal_service_s3::S3;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default()
      // Setup builders
      .root("/path/to/dir")
      .bucket("test")
      .region("us-east-1")
      .endpoint("https://s3.amazonaws.com")
      .access_key_id("access_key_id")
      .secret_access_key("secret_access_key")
      // Enable SSE-KMS with aws managed kms key
      .server_side_encryption_with_aws_managed_kms_key();

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-KMS and customer managed kms key

use log::info;
use opendal_core::Operator;
use opendal_core::Result;
use opendal_service_s3::S3;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default()
      // Setup builders
      .root("/path/to/dir")
      .bucket("test")
      .region("us-east-1")
      .endpoint("https://s3.amazonaws.com")
      .access_key_id("access_key_id")
      .secret_access_key("secret_access_key")
      // Enable SSE-KMS with customer managed kms key
      .server_side_encryption_with_customer_managed_kms_key("aws_kms_key_id");

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with SSE-S3

use log::info;
use opendal_core::Operator;
use opendal_core::Result;
use opendal_service_s3::S3;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default()
      // Setup builders
      .root("/path/to/dir")
      .bucket("test")
      .region("us-east-1")
      .endpoint("https://s3.amazonaws.com")
      .access_key_id("access_key_id")
      .secret_access_key("secret_access_key")
      // Enable SSE-S3
      .server_side_encryption_with_s3_key();

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

S3 with default ACL

use log::info;
use opendal_core::Operator;
use opendal_core::Result;
use opendal_service_s3::S3;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = S3::default()
      // Setup builders
      .root("/path/to/dir")
      .bucket("test")
      .region("us-east-1")
      .endpoint("https://s3.amazonaws.com")
      .access_key_id("access_key_id")
      .secret_access_key("secret_access_key")
      // Enable public-read ACL
      .default_acl("public-read");

    let op = Operator::new(builder)?.finish();
    info!("operator: {:?}", op);

    // New objects will be created with public-read ACL

    Ok(())
}