Skip to content

Commit b2cd4a5

Browse files
authored
chore: updated dataset ingestion diagram (#485)
* chore: updated dataset ingestion diagram
1 parent 359fe13 commit b2cd4a5

File tree

2 files changed

+32
-1
lines changed

2 files changed

+32
-1
lines changed

docs/CryoET_Backend_Workflow.png

-29.8 KB
Binary file not shown.

docs/data_ingestion.md

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,38 @@ The data ingestion process involves transforming input data from various sources
1515
8. Update the database in the production environment.
1616

1717

18-
<img width="1500" alt="CryoET Architecture Diagram V1 0" src="./CryoET_Backend_Workflow.png">
18+
```mermaid
19+
---
20+
title: Cryoet Data Ingestion Pipeline
21+
---
22+
23+
flowchart LR
24+
subgraph env:local
25+
direction TB
26+
B1[Create ingestion config files] --> B2[Validate config files]
27+
28+
end
29+
style env:local fill:#ffffcc
30+
31+
subgraph env:staging
32+
direction TB
33+
C1[Validate source data] --> C2[Ingest data to S3 bucket] --> C3[Validate standardized ingested data] --> C4[Ingest data to the staging database]
34+
35+
end
36+
style env:staging fill:#ccccff
37+
38+
subgraph env:prod
39+
direction TB
40+
D1[Promote data to prod bucket] --> D2[Ingest to prod database]
41+
end
42+
style env:prod fill:#ccffcc
43+
44+
A[Fetch data from data generator] --> env:local
45+
env:local --> env:staging
46+
env:staging --> env:prod
47+
48+
```
49+
1950

2051

2152
## Fetching Source Data

0 commit comments

Comments
 (0)