Skip to content

Commit

Permalink
feat: Break down s3-compatible configurations
Browse files Browse the repository at this point in the history
  • Loading branch information
tomasfarias committed Feb 12, 2025
1 parent 9ec2240 commit c188957
Showing 1 changed file with 24 additions and 2 deletions.
26 changes: 24 additions & 2 deletions contents/docs/cdp/batch-exports/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ With batch exports, data can be exported to an S3 bucket.
1. Subscribe to data pipelines add-on in [your billing settings](https://us.posthog.com/organization/billing) if you haven't already.
2. Click [Data pipelines](https://app.posthog.com/pipeline) in the navigation and go to the **Destinations** tab in your PostHog instance.
3. Search for **S3**.
4. Click the **+ Create** button.
4. Click the **+ Create** button.
5. Fill in the necessary [configuration details](#s3-configuration).
6. Finalize the creation by clicking on "Create".
7. Done! The batch export will schedule its first run on the start of the next period.
Expand Down Expand Up @@ -68,4 +68,26 @@ We intend to add support for other common formats, and format-specific configura

### S3-compatible blob storage

PostHog S3 batch exports may also export data to an S3-compatible blob storage like [MinIO](https://github.com/minio/minio). Simply set the *Endpoint URL* to your blob storage's host and port, for example: `https://my-minio-storage:9000`.
PostHog S3 batch exports may also export data to an S3-compatible blob storage like [MinIO](https://github.com/minio/minio), [Cloudflare R2](https://www.cloudflare.com/developer-platform/products/r2/), or [Google Cloud Storage (GCS)](https://cloud.google.com/storage). Here we describe configuration tweaks that are required for S3-compatible blob storage destinations that we have tested.

#### MinIO
* Set the *Endpoint URL* configuration to your MinIO instance's host and port, for example: `https://my-minio-storage:9000`.

#### Cloudflare R2
* Set the *Endpoint URL* configuration to the following after replacing your account id: `https://<ACCOUNT_ID>.r2.cloudflarestorage.com`.
* From the *Region* dropdown, select one of the Cloudflare R2 regions that correspond to your bucket, like "Automatic (AUTO)".

#### Google Cloud Storage (GCS)
Access to GCS for batch exports follows a similar process to accessing [BigQuery](/docs/cdp/batch-exports/bigquery)
as a Service Account is required:
1. Follow the steps in the [BigQuery documentation](/docs/cdp/batch-exports/bigquery) to create a Service Account.
2. Create a [HMAC key for your Service Account](https://cloud.google.com/storage/docs/authentication/managing-hmackeys#console).
3. Grant the Service Account the `Storage Object User` role or a custom role with at least the following permissions:
* `storage.multipartUploads.abort`
* `storage.multipartUploads.create`
* `storage.multipartUploads.list`
* `storage.multipartUploads.listParts`
* `storage.objects.create`
* `storage.objects.delete`
4. Use the HMAC key access key and secret key as *AWS Access Key ID* and *AWS Secret Access Key* respectively when configuring your batch export.
5. Finally, set the *Endpoint URL* configuration to: `https://storage.googleapis.com`.

0 comments on commit c188957

Please sign in to comment.