feat: add external storage support to codec server (#4489)

* docs: add external storage support

* docs: remove redundant info from setup page

* address review comments

Author: lennessyy

docs/encyclopedia/data-conversion/codec-server.mdx

@@ -79,6 +79,61 @@ form.
 Your Codec Server should use the same Payload Codec implementation as your Workers to ensure consistent encoding and
 decoding.
 
+## Codec Server with External Storage {#external-storage}
+
+When your Workers and Clients use [External Storage](/external-storage), your storage drivers replace some payloads in
+the Event History with small references that point to data in an external store like Amazon S3. The Temporal Service and
+the Web UI only see these references, not the actual payload data. This is further complicated by setups where you run
+Codecs in proxy that encode payloads after the Data Converter has returned on the Worker. Your Codec Server must be able
+to handle downloading and decoding in the correct order for you to be able to view the Workflow data in the UI or CLI.
+
+To support External Storage, create a handler using `NewPayloadHTTPHandler` with `PayloadHTTPHandlerOptions`. The options
+accept your storage drivers, your pre-storage codecs (the Payload Codecs configured in your Worker's Data Converter),
+and any post-storage codecs (codecs applied by a proxy after external storage). The handler applies them in the correct
+order across all endpoints automatically. When you configure the handler with storage drivers, the existing endpoints
+become storage-aware and a new `/download` endpoint becomes available:
+
+:::caution
+
+`NewPayloadHTTPHandler` runs the full encode-store-encode and decode-retrieve-decode pipeline. Do not use it as a target
+for a remote Data Converter or remote codec on your Workers. For remote codecs, use `NewPayloadCodecHTTPHandler`
+separately. If you need both, set up `NewPayloadHTTPHandler` for the Web UI and CLI alongside
+`NewPayloadCodecHTTPHandler` for your Workers, and configure both with the same codecs.
+
+:::
+
+- **`/download`** retrieves the actual payload data from external storage and decodes it through the Payload Codec. This
+  endpoint is used internally by `/decode` when it encounters storage references, but you can also call it directly from
+  to retrieve the decoded payload. The Temporal Web UI uses this endpoint when you click to view the full payload for a
+  storage reference.
+- **`/decode`** still decodes encoded payloads, but also handles storage references. By default, `/decode` uses the
+  download logic internally to retrieve and decode any storage references in the request alongside regular payloads.
+  With the `?preserveStorageRefs=true` query parameter, `/decode` skips retrieval and returns storage references as-is.
+- **`/encode`** applies the Payload Codec, then uploads payloads that exceed the size threshold to external storage and
+  replaces them with reference tokens.
+
+<CaptionedImage
+  src="/diagrams/codec-server-with-external-storage.svg"
+  srcDark="/diagrams/codec-server-with-external-storage-dark.svg"
+  width="100%"
+  title="Codec Server with External Storage"
+/>
+
+The following example walks through how all three endpoints work together:
+
+1. A user starts a Workflow from the CLI with a plaintext input. The CLI sends the input to the Codec Server's `/encode`
+   endpoint.
+2. The Codec Server encodes the payload through the Payload Codec. The encoded payload exceeds the storage threshold,
+   so the Codec Server uploads it to external storage and returns a small reference token.
+3. The CLI sends the reference token to the Temporal Service, which stores it in the Event History.
+4. Later, a user views the Workflow in the Web UI. The Web UI retrieves the Event History from the Temporal Service and
+   sends the payloads to the Codec Server's `/decode` endpoint with the `?preserveStorageRefs=true` query parameter.
+5. The Codec Server decodes any non-reference payloads through the Payload Codec, but returns storage references as-is.
+   The Web UI displays the reference metadata, indicating the payload is stored externally.
+6. The user clicks to view the full payload. The Web UI sends the storage reference to the `/download` endpoint.
+7. The Codec Server retrieves the encoded payload from external storage, decodes it through the Payload Codec, and
+   returns the plaintext result to the Web UI.
+
 ## Codec Server vs. Payload Codec
 
 A Codec Server runs a [Payload Codec](/payload-codec) internally, so the two are directly connected. The difference is

docs/production-deployment/data-encryption.mdx

@@ -1,6 +1,6 @@
 ---
 id: data-encryption
-title: Codec Server - Temporal Platform feature guide
+title: Codecs and Encryption
 sidebar_label: Codecs and Encryption
 description: Encrypt data in Temporal Server to secure Workflow, Activity, and Worker information. Use custom Payload Codecs for encryption/decryption, set up Codec Servers for remote decoding, and ensure secure access.
 slug: /production-deployment/data-encryption
@@ -18,10 +18,11 @@ tags:
 
 import { CaptionedImage } from '@site/src/components';
 
-Temporal Server stores and persists the data handled in your Workflow Execution.
-Encrypting this data ensures that any sensitive application data is secure when handled by the Temporal Server.
+The Temporal Service persists data from your Workflow Executions, including inputs, outputs, and results. To protect
+sensitive data, use a [Payload Codec](/payload-codec) to encrypt payloads before they reach the Temporal Service. With
+encryption enabled, data exists unencrypted only on the Client and the Worker process, on hosts that you control.
 
-For example, if you have sensitive information passed in the following objects that are persisted in the Workflow Execution Event History, use encryption to secure it:
+The following data is persisted in the Event History and can be encrypted:
 
 - Inputs and outputs/results in your [Workflow](/workflow-execution), [Activity](/activity-execution), and [Child Workflow](/child-workflows)
 - [Signal](/sending-messages#sending-signals) inputs
@@ -30,51 +31,30 @@ For example, if you have sensitive information passed in the following objects t
 - [Query](/sending-messages#sending-queries) inputs and results
 - Results of [Local Activities](/local-activity) and [Side Effects](/workflow-execution/event#side-effect)
 - [Application errors and failures](/references/failures).
-  Failure messages and call stacks are not encoded as codec-capable Payloads by default; you must explicitly enable encoding these common attributes on failures.
-  For more details, see [Failure Converter](/failure-converter).
+  Failure messages and call stacks are not encoded as codec-capable Payloads by default; you must explicitly enable
+  encoding these common attributes on failures. For more details, see [Failure Converter](/failure-converter).
 
-Using encryption ensures that your sensitive data exists unencrypted only on the Client and the Worker Process that is executing the Workflows and Activities, on hosts that you control.
+To view encrypted data in the Web UI and CLI, set up a [Codec Server](/codec-server). The following sections cover how
+to set up a Codec Server and configure the Web UI and CLI to use it.
 
-By default, your data is serialized to a [Payload](/dataconversion#payload) by a [Data Converter](/dataconversion).
-To encrypt your Payload, configure your custom encryption logic with a [Payload Codec](/payload-codec) and set it with a [custom Data Converter](/default-custom-data-converters#custom-data-converter).
+For encryption implementation examples, see the following samples:
 
-A Payload Codec does byte-to-byte conversion to transform your Payload (for example, by implementing compression and/or encryption and decryption) and is an optional step that happens between the Client and the [Payload Converter](/payload-converter):
-
-<CaptionedImage
-    src="/diagrams/remote-data-encoding.svg"
-    title="Remote data encoding architecture" />
-
-You can run your Payload Codec with a [Codec Server](/codec-server) and use the Codec Server endpoints in the Web UI and CLI to decode your encrypted Payload locally.
-For details on how to set up a Codec Server, see [Codec Server setup](#codec-server-setup).
-
-However, if you plan to set up [remote data encoding](/remote-data-encoding) for your data, ensure that you consider all security implications of running encryption remotely before implementing it.
-
-When implementing a custom codec, it is recommended to perform your compression or encryption on the entire input Payload and store the result in the data field of a new Payload with a different encoding metadata field.
-This ensures that the input Payload's metadata is preserved.
-When the encoded Payload is sent to be decoded, you can verify the metadata field before applying the decryption.
-If your Payload is not encoded, it is recommended to pass the unencoded data to the decode function instead of failing the conversion.
-
-Examples for implementing encryption:
-
-- [Go sample](https://github.com/temporalio/samples-go/tree/main/encryption)
-- [Java sample](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads)
-- [Python sample](https://github.com/temporalio/samples-python/tree/main/encryption)
-- [TypeScript sample](https://github.com/temporalio/samples-typescript/tree/main/encryption)
-- [.NET sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption)
+- [Go](https://github.com/temporalio/samples-go/tree/main/encryption)
+- [Java](https://github.com/temporalio/samples-java/tree/main/core/src/main/java/io/temporal/samples/encryptedpayloads)
+- [Python](https://github.com/temporalio/samples-python/tree/main/encryption)
+- [TypeScript](https://github.com/temporalio/samples-typescript/tree/main/encryption)
+- [.NET](https://github.com/temporalio/samples-dotnet/tree/main/src/Encryption)
 
 ## Codec Server setup {#codec-server-setup}
 
 Use a Codec Server to programmatically decode your encoded [payloads](/dataconversion#payload).
 
 A Codec Server is an HTTP server that uses your custom Codec logic to decode your data remotely.
 The Codec Server is independent of the Temporal Service and decodes your encrypted payloads through predefined endpoints. You create, operate, and manage access to your Codec Server in your own environment.
-The Temporal CLI and the Web UI in turn provide built-in hooks to call the Codec Server to decode encrypted payloads on demand.
-
-The Codec Server is independent of the Temporal Server and decodes your encrypted payloads through endpoints.
-When you configure a Codec Server endpoint in the Temporal Web UI or CLI, the Web UI and CLI use the remote endpoint to receive decoded payloads from the Codec Server.
+When you configure a Codec Server endpoint in the Web UI or CLI, the Web UI and CLI use the remote endpoint to send and receive payloads from the Codec Server.
 See [API contract requirements](#api-contract-specifications).
 
-Decoded payloads can then be displayed in the Workflow Execution Event History on the Web UI. Note that when you use a Codec Server, the decoded payloads are decoded and returned on the client side only; payloads on the Temporal Server (whether on Temporal Cloud or a self-hosted Temporal Service) remain encrypted.
+Decoded payloads can then be displayed in the Workflow Execution Event History on the Web UI. When you use a Codec Server, the decoded payloads are decoded and returned on the client side only. Payloads on the Temporal Service (whether on Temporal Cloud or self-hosted) remain encrypted.
 
 Because you create, operate, and manage access to your Codec Server in your controlled environment, ensure that you consider the following:
 
@@ -91,7 +71,13 @@ When you create your Codec Server to handle requests from the Web UI, the follow
 
 #### Endpoints
 
-The Web UI and CLI send a POST to a `/decode` endpoint. In your Codec Server, create a `/decode` path and pass the incoming payload to the decode method in your Payload Codec.
+The Web UI and CLI send POST requests to the following endpoints on your Codec Server:
+
+- `/decode` passes incoming payloads to the decode method in your Payload Codec.
+- `/encode` passes incoming payloads to the encode method in your Payload Codec.
+- `/download` retrieves and decodes payloads from [External Storage](/external-storage). This endpoint is only needed if
+  your Workers use External Storage. See [Codec Server with External Storage](/codec-server#external-storage) for
+  details.
 
 For examples on how to create your Codec Server, see the following Codec Server implementation samples:
 
@@ -346,14 +332,12 @@ temporal workflow show \
    --codec-auth 'auth-header'
 ```
 
-### Working with Large Payloads
-
-Codec Servers can be used for more than encryption and decryption of sensitive data.
-Codec Server behavior is left up to implementers -- they can also call external services or perform other tasks, as long as they hook in at the encoding and decoding stages of a Workflow payload.
+### Working with large payloads
 
-By default, Temporal limits payload size to 4MB.
-If this limitation is problematic for your use case, you could implement a codec that persists your payloads to an object store outside of workflow histories.
-An example implementation is available from [DataDog](https://github.com/DataDog/temporal-large-payload-codec).
+If your payloads exceed the Temporal Service's size limits, use [External Storage](/external-storage) to offload large
+payloads to an external store like Amazon S3. When External Storage is configured, your Codec Server can also retrieve
+and decode these payloads for viewing in the Web UI and CLI. See
+[Codec Server with External Storage](/codec-server#external-storage) for details.
 
 ### Temporal Nexus