Skip to content

Commit 262082c

Browse files
y-f-uewgeniusJeadie
authored
s3 parquet doc (#111)
* s3 parquet doc * Update index * Jack/24 03 15 (#116) * Update acknowledgments (#109) * Update acknowledgments * Remove spiceai packages * update S3 connector docs * Revert "Update acknowledgments (#109)" This reverts commit 8b2c3e1. --------- Co-authored-by: Evgenii Khramkov <evgenii@spice.ai> --------- Co-authored-by: Evgenii Khramkov <evgenii@spice.ai> Co-authored-by: Jack Eadie <jack@spice.ai>
1 parent 894ae5f commit 262082c

2 files changed

Lines changed: 51 additions & 0 deletions

File tree

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
type: docs
3+
title: "Data Connectors"
4+
linkTitle: "Data Connectors"
5+
description: ''
6+
weight: 70
7+
---
8+
9+
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
type: docs
3+
title: "S3 Data Connector"
4+
linkTitle: "S3 Data Connector"
5+
description: 'S3 Data Connector YAML reference'
6+
weight: 80
7+
---
8+
9+
S3 as a connector for federated SQL query across Parquet files stored in S3, or S3-compatible storage solutions (e.g. Minio, Cloudflare R2).
10+
11+
## `params`
12+
13+
- `endpoint`: The S3 endpoint, or equivalent (e.g. Minio endpoint), for the S3-compatible storage.
14+
- `region`: Region of the S3 bucket, if region specific.
15+
16+
## `auth`
17+
18+
Check [Secrets]({{<ref "reference/Secrets">}}).
19+
20+
Required attribbutes:
21+
- `key`: The access key authorised to access the S3 data (e.g. `AWS_ACCESS_KEY_ID` for AWS)
22+
- `secret`The secret key authorised to access the S3 data (e.g. `AWS_SECRET_ACCESS_KEY` for AWS)
23+
24+
25+
## Example
26+
### Minio
27+
```yaml
28+
- from: s3://s3-bucket-name/path/to/parquet/cool_dataset.parquet
29+
name: cool_dataset
30+
params:
31+
endpoint: https://my.minio.server
32+
region: "us-east-1" # Best practice for Minio
33+
```
34+
35+
#### S3
36+
```yaml
37+
- from: s3://my-startups-data/path/to/parquet/cool_dataset.parquet
38+
name: cool_dataset
39+
params:
40+
endpoint: http://my-startups-data.s3.amazonaws.com
41+
region: "ap-southeast-2"
42+
```

0 commit comments

Comments
 (0)