Skip to content
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
---
title: Typed Pipeline bindings, improved setup, and dropped event metrics
description: Generate schema-aware TypeScript types for Pipeline bindings, set up pipelines faster with Simple mode, and monitor dropped events in the dashboard and via GraphQL.
products:
- pipelines
- workers
date: 2026-02-19
---

## Typed pipeline bindings

Running `wrangler types` now generates schema-specific TypeScript types for pipeline bindings. When a stream has a defined schema, your pipeline binding uses a named record type instead of the generic `Pipeline<PipelineRecord>`, giving you full autocomplete and compile-time type checking.

```ts
declare namespace Cloudflare {
type EcommerceStreamRecord = {
user_id: string;
event_type: string;
product_id?: string;
amount?: number;
};
interface Env {
STREAM: import("cloudflare:pipelines").Pipeline<Cloudflare.EcommerceStreamRecord>;
}
}
```

TypeScript catches missing required fields and incorrect field types at compile time. You must be authenticated via `wrangler login` for schema fetching to work. If not authenticated or if the stream has no schema, `wrangler types` falls back to the generic type.

For more information, refer to [Typed pipeline bindings](/pipelines/streams/writing-to-streams/#typed-pipeline-bindings).

## Improved Pipeline setup

The `wrangler pipelines setup` command now offers a **Simple** setup mode that applies recommended defaults for format (Parquet), compression (zstd), rolling policy (100 MB), and credential generation. The setup command also automatically creates the [R2 bucket](/r2/buckets/) and enables [R2 Data Catalog](/r2/data-catalog/) if they do not already exist.

Validation errors during setup now prompt you to retry inline rather than restarting the entire process.

For a full walkthrough, refer to the [Getting started guide](/pipelines/getting-started/).

## Dropped event metrics

A new **Errors** tab in the Cloudflare dashboard shows dropped events with detailed error messages, grouped by error type.

![The Errors tab in the Cloudflare dashboard showing deserialization errors grouped by type with individual error details](~/assets/images/pipelines/pipelines-error-log-dash.png)

You can also query dropped events programmatically via the new `pipelinesUserErrorsAdaptiveGroups` GraphQL dataset. When structured stream events fail schema validation, you can query the specific error type (`missing_field`, `type_mismatch`, `parse_failure`, or `null_value`) and see how many events were affected.

```graphql
query GetPipelineUserErrors($accountTag: string!, $pipelineId: string!) {
viewer {
accounts(filter: { accountTag: $accountTag }) {
pipelinesUserErrorsAdaptiveGroups(
limit: 100
filter: {
pipelineId: $pipelineId
datetime_geq: "2026-02-12T00:00:00Z"
datetime_leq: "2026-02-19T23:59:59Z"
}
orderBy: [count_DESC]
) {
count
dimensions {
errorFamily
errorType
}
}
}
}
}
```

For the full list of dimensions, error types, and additional query examples, refer to [User error metrics](/pipelines/observability/metrics/#user-error-metrics).
193 changes: 80 additions & 113 deletions src/content/docs/pipelines/getting-started.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ import {
Tabs,
TabItem,
DashButton,
Details,
LinkCard,
} from "~/components";

This guide will instruct you through:

- Creating your first [R2 bucket](/r2/buckets/) and enabling its [data catalog](/r2/data-catalog/).
- Creating an [API token](/r2/api/tokens/) needed for pipelines to authenticate with your data catalog.
- Creating your first pipeline with a simple ecommerce schema that writes to an [Apache Iceberg](https://iceberg.apache.org/) table managed by R2 Data Catalog.
- Sending sample ecommerce data via HTTP endpoint.
Expand All @@ -28,74 +28,7 @@ This guide will instruct you through:

<Render file="prereqs" product="workers" />

## 1. Create an R2 bucket

<Tabs syncKey='CLIvDash'>
<TabItem label='Wrangler CLI'>

<Steps>
1. If not already logged in, run:

```
npx wrangler login
```

2. Create an R2 bucket:

```
npx wrangler r2 bucket create pipelines-tutorial
```

</Steps>

</TabItem>
<TabItem label='Dashboard'>

<Steps>
1. In the Cloudflare dashboard, go to the **R2 object storage** page.
<DashButton url="/?to=/:account/r2/overview" />

2. Select **Create bucket**.

3. Enter the bucket name: pipelines-tutorial

4. Select **Create bucket**.

</Steps>
</TabItem>
</Tabs>

## 2. Enable R2 Data Catalog

<Tabs syncKey='CLIvDash'>
<TabItem label='Wrangler CLI'>

Enable the catalog on your R2 bucket:

```
npx wrangler r2 bucket catalog enable pipelines-tutorial
```

When you run this command, take note of the "Warehouse" and "Catalog URI". You will need these later.

</TabItem>
<TabItem label='Dashboard'>

<Steps>
1. In the Cloudflare dashboard, go to the **R2 object storage** page.
<DashButton url="/?to=/:account/r2/overview" />

2. Select the bucket: pipelines-tutorial.

3. Switch to the **Settings** tab, scroll down to **R2 Data Catalog**, and select **Enable**.

4. Once enabled, note the **Catalog URI** and **Warehouse name**.

</Steps>
</TabItem>
</Tabs>

## 3. Create an API token
## 1. Create an API token

Pipelines must authenticate to R2 Data Catalog with an [R2 API token](/r2/api/tokens/) that has catalog and R2 permissions.

Expand All @@ -109,7 +42,7 @@ Pipelines must authenticate to R2 Data Catalog with an [R2 API token](/r2/api/to

4. Give your API token a name.

5. Under **Permissions**, choose the **Admin Read & Write** permission.
5. Under **Permissions**, select the **Admin Read & Write** permission.

6. Select **Create Account API Token**.

Expand All @@ -121,7 +54,7 @@ Pipelines must authenticate to R2 Data Catalog with an [R2 API token](/r2/api/to
This token also includes the R2 SQL Read permission, which allows you to query your data with R2 SQL.
:::

## 4. Create your first pipeline
## 2. Create your first pipeline

<Tabs syncKey='CLIvDash'>
<TabItem label='Wrangler CLI'>
Expand Down Expand Up @@ -163,6 +96,10 @@ Use the interactive setup to create a pipeline that writes to R2 Data Catalog:
npx wrangler pipelines setup
```

:::note
The setup command automatically creates the [R2 bucket](/r2/buckets/) and enables [R2 Data Catalog](/r2/data-catalog/) if they do not already exist, so you do not need to create them beforehand.
:::

Follow the prompts:

1. **Pipeline name**: Enter `ecommerce`
Expand All @@ -175,38 +112,68 @@ Follow the prompts:
- Schema file path: `schema.json` (or your file path)

3. **Sink configuration**:
- Destination type: `Data Catalog Table`
- R2 bucket name: `pipelines-tutorial`
- Namespace: `default`
- Destination type: `Data Catalog (Iceberg)`
- Setup mode: `Simple (recommended defaults)`
- R2 bucket name: `pipelines-tutorial` (created automatically if it does not exist)
- Table name: `ecommerce`
- Catalog API token: Enter your token from step 3
- Compression: `zstd`
- Roll file when size reaches (MB): `100`
- Roll file when time reaches (seconds): `10` (for faster data visibility in this tutorial)
- Catalog API token: Enter your token from step 1

4. **Review**: Confirm the summary and select `Create resources`

5. **SQL transformation**: Choose `Simple ingestion (SELECT * FROM stream)`

:::note
If you make a mistake during setup (such as an invalid name or incorrect credentials), you will be prompted to retry rather than needing to restart the entire setup process.
:::

<Details header="Advanced mode options">

4. **SQL transformation**: Choose `Use simple ingestion query` to use:
```sql
INSERT INTO ecommerce_sink SELECT * FROM ecommerce_stream
```
If you select **Advanced** instead of **Simple** during sink configuration, you can customize the following additional options:

After setup completes, note the HTTP endpoint URL displayed in the final output.
- **Format**: Output file format (for example, Parquet)
- **Compression**: Compression algorithm (for example, zstd)
- **Rolling policy**: File size threshold (minimum 5 MB) and time interval (minimum 10 seconds) for creating new files
- **Credentials**: Choose between automatic credential generation or manually entering R2 credentials
- **Namespace**: Data Catalog namespace (defaults to `default`)

</Details>

After setup completes, the command outputs a configuration snippet for your Wrangler file, a Worker binding example with sample data, and a curl command for the HTTP endpoint. Note the HTTP endpoint URL and the `pipelines` configuration for use in the following steps.

You can also pre-set the pipeline name using the `--name` flag:

```bash
npx wrangler pipelines setup --name ecommerce
```

</TabItem>
<TabItem label='Dashboard'>

<Steps>
1. In the Cloudflare dashboard, go to **Pipelines** > **Pipelines**.
1. In the Cloudflare dashboard, go to **R2 object storage**.
<DashButton url="/?to=/:account/r2/overview" />

2. Select **Create bucket** and enter the bucket name: `pipelines-tutorial`.

3. Select **Create bucket**.

4. Select the bucket, switch to the **Settings** tab, scroll down to **R2 Data Catalog**, and select **Enable**.

5. Once enabled, note the **Catalog URI** and **Warehouse name**.

6. Go to **Pipelines** > **Pipelines**.

<DashButton url="/?to=/:account/pipelines/overview" />

2. Select **Create Pipeline**.
7. Select **Create Pipeline**.

3. **Connect to a Stream**:
8. **Connect to a Stream**:
- Pipeline name: `ecommerce`
- Enable HTTP endpoint for sending data: Enabled
- HTTP authentication: Disabled (default)
- Select **Next**

4. **Define Input Schema**:
9. **Define Input Schema**:
- Select **JSON editor**
- Copy in the schema:
```json
Expand All @@ -229,42 +196,42 @@ After setup completes, note the HTTP endpoint URL displayed in the final output.
},
{
"name": "amount",
"type": "f64",
"type": "float64",
"required": false
}
]
}
```
- Select **Next**

5. **Define Sink**:
- Select your R2 bucket: `pipelines-tutorial`
- Storage type: **R2 Data Catalog**
- Namespace: `default`
- Table name: `ecommerce`
- **Advanced Settings**: Change **Maximum Time Interval** to `10 seconds`
- Select **Next**
10. **Define Sink**:
- Select your R2 bucket: `pipelines-tutorial`
- Storage type: **R2 Data Catalog**
- Namespace: `default`
- Table name: `ecommerce`
- **Advanced Settings**: Change **Maximum Time Interval** to `10 seconds`
- Select **Next**

6. **Credentials**:
- Disable **Automatically create an Account API token for your sink**
- Enter **Catalog Token** from step 3
- Select **Next**
11. **Credentials**:
- Disable **Automatically create an Account API token for your sink**
- Enter **Catalog Token** from step 1
- Select **Next**

7. **Pipeline Definition**:
- Leave the default SQL query:
```sql
INSERT INTO ecommerce_sink SELECT * FROM ecommerce_stream;
```
- Select **Create Pipeline**
12. **Pipeline Definition**:
- Leave the default SQL query:
```sql
INSERT INTO ecommerce_sink SELECT * FROM ecommerce_stream;
```
- Select **Create Pipeline**

8. After pipeline creation, note the **Stream ID** for the next step.
13. After pipeline creation, note the **Stream ID** for the next step.

</Steps>

</TabItem>
</Tabs>

## 5. Send sample data
## 3. Send sample data

Send ecommerce events to your pipeline's HTTP endpoint:

Expand Down Expand Up @@ -294,20 +261,20 @@ curl -X POST https://{stream-id}.ingest.cloudflare.com \

Replace `{stream-id}` with your actual stream endpoint from the pipeline setup.

## 6. Validate data in your bucket
## 4. Validate data in your bucket

<Steps>
1. In the Cloudflare dashboard, go to the **R2 object storage** page.

2. Select your bucket: `pipelines-tutorial`.

3. You should see Iceberg metadata files and data files created by your pipeline. Note: If you aren't seeing any files in your bucket, try waiting a couple of minutes and trying again.
3. You should see Iceberg metadata files and data files created by your pipeline. If you are not seeing any files in your bucket, wait a couple of minutes and try again.

4. The data is organized in the Apache Iceberg format with metadata tracking table versions.

</Steps>

## 7. Query your data using R2 SQL
## 5. Query your data using R2 SQL

Set up your environment to use R2 SQL:

Expand All @@ -317,11 +284,11 @@ export WRANGLER_R2_SQL_AUTH_TOKEN=YOUR_API_TOKEN

Or create a `.env` file with:

```
```txt
WRANGLER_R2_SQL_AUTH_TOKEN=YOUR_API_TOKEN
```

Where `YOUR_API_TOKEN` is the token you created in step 3. For more information on setting environment variables, refer to [Wrangler system environment variables](/workers/wrangler/system-environment-variables/).
Where `YOUR_API_TOKEN` is the token you created in step 1. For more information on setting environment variables, refer to [Wrangler system environment variables](/workers/wrangler/system-environment-variables/).

Query your data:

Expand All @@ -337,7 +304,7 @@ WHERE event_type = 'purchase'
LIMIT 10"
```

Replace `YOUR_WAREHOUSE_NAME` with the warehouse name from step 2.
Replace `YOUR_WAREHOUSE_NAME` with the warehouse name noted during pipeline setup. You can find it in the Cloudflare dashboard under **R2 object storage** > your bucket > **Settings** > **R2 Data Catalog**.

You can also query this table with any engine that supports Apache Iceberg. To learn more about connecting other engines to R2 Data Catalog, refer to [Connect to Iceberg engines](/r2/data-catalog/config-examples/).

Expand Down
Loading
Loading