Skip to content

Commit b55c9c7

Browse files
authored
HADOOP-19349: S3A : Improve Client Side Encryption Documentation (#7191)
Followup to HADOOP-18708: S3A: Support S3 Client Side Encryption(CSE) (#6884) Contributed by Syed Shameerur Rahman
1 parent 464d7d9 commit b55c9c7

File tree

1 file changed

+21
-10
lines changed
  • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws

1 file changed

+21
-10
lines changed

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md

+21-10
Original file line numberDiff line numberDiff line change
@@ -680,7 +680,7 @@ client-side and then transmit it over to S3 storage. The same encrypted data
680680
is then transmitted over to client while reading and then
681681
decrypted on the client-side.
682682

683-
S3-CSE, uses `S3EncryptionClient.java` (V3) as the AmazonS3 client. The
683+
S3-CSE, uses `S3EncryptionClient.java`(V3) as the AmazonS3 client. The
684684
encryption and decryption is done by AWS SDK. Both CSE-KMS and CSE-CUSTOM
685685
methods are supported.
686686

@@ -712,14 +712,25 @@ like `AmazonS3EncryptionClient.java`(V1) and `AmazonS3EncryptionClientV2.java`(V
712712
- encryption information stored as headers in the uploaded object.
713713

714714
### Compatibility Issues
715-
- The V1 client support reading unencrypted S3 objects, whereas the V3 client does not.
716-
- Unlike the V2 and V3 clients, which always append 16 bytes to a file,
717-
the V1 client appends extra bytes to the next multiple of 16.
718-
For example, if the unencrypted object size is 28 bytes,
719-
the V1 client pads an extra 4 bytes to make it a multiple of 16.
720-
721-
Note: Inorder to workaround the above compatibility issues
722-
set `fs.s3a.encryption.cse.v1.compatibility.enabled=true`
715+
- The V1 client is capable of reading unencrypted S3 objects,
716+
a capability not supported by the V3 client.
717+
- Unlike V2 and V3 clients that consistently append 16 bytes to files,
718+
the V1 client implements a more dynamic padding strategy by appending
719+
extra bytes to reach the next multiple of 16. For example
720+
Consider an unencrypted object with 28 bytes: the V1 client strategically
721+
adds 4 additional bytes to ensure the total size becomes a precise multiple of 16.
722+
723+
This dynamic padding strategy in V1 complicates straightforward computation of unencrypted length
724+
, preventing the simple subtraction of 16 bytes used in V2 and V3 clients, hence requiring additional
725+
S3 GET call to identify the unencrypted length of objects encrypted by V1 client.
726+
727+
Mitigate V1 client encryption compatibility challenges by setting
728+
`fs.s3a.encryption.cse.v1.compatibility.enabled=true`.This configuration solution
729+
comes with a performance trade-off, necessitating additional S3 GET and HEAD calls.
730+
731+
Inorder to workaround the above compatibility issues set the configuration
732+
`fs.s3a.encryption.cse.v1.compatibility.enabled=true`. This will have some performance penalty
733+
in terms of a additional S3 GET and HEAD calls.
723734

724735
Note: The V1 client supports storing encryption metadata in a separate file with
725736
the suffix "fileName".instruction. However, these instruction files are not
@@ -786,7 +797,7 @@ S3-CSE to work.
786797
#### 2. CSE-CUSTOM
787798
- Set `fs.s3a.encryption.algorithm=CSE-CUSTOM`.
788799
- Set
789-
`fs.s3a.encryption.cse.custom.cryptographic.material.manager.class.name=<fully qualified class name>`.
800+
`fs.s3a.encryption.cse.custom.keyring.class.name=<fully qualified class name>`.
790801

791802
Example for custom keyring implementation
792803
```

0 commit comments

Comments
 (0)