Skip to content

Commit c4ccc66

Browse files
committed
Update README formatting and project summary
Improved the project summary wording, added spacing for better readability, and updated the v1 vs v2.0 comparison table. Minor formatting adjustments were made to code blocks and section layouts.
1 parent dc7f368 commit c4ccc66

File tree

1 file changed

+7
-12
lines changed

1 file changed

+7
-12
lines changed

README.md

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33

44
## Project Summary
55

6-
Built a production-lite, serverless ELT framework on AWS (S3 bronze JSONL → Lambda ingest → SQS (+ DLQ) → Lambda transform → S3 silver Parquet) with optional orchestration, catalog/query, quality gates, and observability.
6+
Built a end to end production-lite, serverless ELT framework on AWS (S3 bronze JSONL → Lambda ingest → SQS (+ DLQ) → Lambda transform → S3 silver Parquet) with optional orchestration, catalog/query, quality gates, and observability.
77

88
**Highlights**
99

@@ -37,6 +37,7 @@ S3 Silver (parquet)
3737

3838

3939
```text
40+
4041
S3 (bronze/*.jsonl)
4142
└─ ObjectCreated
4243
└─ Lambda ingest (Powertools logs/metrics/idempotency)
@@ -45,6 +46,7 @@ S3 (bronze/*.jsonl)
4546
└─ S3 (silver/…/*.parquet)
4647
└─ (optional) Glue Catalog/Crawler → Athena
4748
└─ (optional) Step Functions → Glue Job (+ Great Expectations gate)
49+
4850
```
4951

5052
No VPC/EC2 is required for the minimal path.
@@ -67,17 +69,6 @@ This repo is designed so you can keep a minimal, low-cost baseline (the core S3
6769

6870
## v1 vs v2.0
6971

70-
| Aspect | v1 (Minimal) | v2.0 (Enterprise track) |
71-
|---|---|---|
72-
| Pipeline | S3 → Lambda → SQS → Lambda → S3 | Same + optional workflows |
73-
| Idempotency | DynamoDB object-level | Powertools Idempotency (DDB TTL) |
74-
| Recovery | Basic | Replay + DLQ redrive helpers |
75-
| Storage | JSONL → Parquet | Same (+ optional compaction job) |
76-
| Queryability | S3 only | Optional Glue Catalog/Crawler + Athena |
77-
| Data quality || Optional Glue Job + GE gate |
78-
| Observability | Logs only | Optional CloudWatch dashboards + alarms |
79-
| CI/CD | Local apply | CI + manual Terraform workflow |
80-
8172
| Aspect | v1 (Minimal) | v2.0 (Enterprise track) |
8273
|---|---|----|
8374
| Core pipeline | S3 → Lambda → SQS → Lambda → S3 (Parquet) | Same + production options |
@@ -93,6 +84,7 @@ This repo is designed so you can keep a minimal, low-cost baseline (the core S3
9384
## Quickstart
9485

9586
```bash
87+
9688
git clone https://github.com/wyang10/AWS-Serverless-ELT-Pipeline-Enterprise.git
9789
cd AWS-Serverless-ELT-Pipeline-Enterprise
9890

@@ -107,6 +99,7 @@ aws sts get-caller-identity
10799
make build
108100
make tf-init
109101
TF_AUTO_APPROVE=1 make tf-apply
102+
110103
```
111104

112105
Run end-to-end verification (screenshot-able checks):
@@ -144,6 +137,7 @@ Recommendation: keep `ge_emit_events_from_transform=false` and `ge_eventbridge_e
144137
## Repo layout
145138

146139
```text
140+
147141
.
148142
├─ .github/workflows/ # CI + manual terraform workflow
149143
├─ infra/terraform/
@@ -163,6 +157,7 @@ Recommendation: keep `ge_emit_events_from_transform=false` and `ge_eventbridge_e
163157
├─ demo/ # screenshots
164158
├─ Instructions.md
165159
└─ LICENSE
160+
166161
```
167162

168163
## Screenshots

0 commit comments

Comments
 (0)