File tree Expand file tree Collapse file tree 2 files changed +32
-1
lines changed
Expand file tree Collapse file tree 2 files changed +32
-1
lines changed Original file line number Diff line number Diff line change @@ -15,7 +15,38 @@ The data ingestion process involves transforming input data from various sources
15158 . Update the database in the production environment.
1616
1717
18- <img width =" 1500 " alt =" CryoET Architecture Diagram V1 0 " src =" ./CryoET_Backend_Workflow.png " >
18+ ``` mermaid
19+ ---
20+ title: Cryoet Data Ingestion Pipeline
21+ ---
22+
23+ flowchart LR
24+ subgraph env:local
25+ direction TB
26+ B1[Create ingestion config files] --> B2[Validate config files]
27+
28+ end
29+ style env:local fill:#ffffcc
30+
31+ subgraph env:staging
32+ direction TB
33+ C1[Validate source data] --> C2[Ingest data to S3 bucket] --> C3[Validate standardized ingested data] --> C4[Ingest data to the staging database]
34+
35+ end
36+ style env:staging fill:#ccccff
37+
38+ subgraph env:prod
39+ direction TB
40+ D1[Promote data to prod bucket] --> D2[Ingest to prod database]
41+ end
42+ style env:prod fill:#ccffcc
43+
44+ A[Fetch data from data generator] --> env:local
45+ env:local --> env:staging
46+ env:staging --> env:prod
47+
48+ ```
49+
1950
2051
2152## Fetching Source Data
You can’t perform that action at this time.
0 commit comments