Skip to content

Commit 6026555

Browse files
sfc-gh-ridasafdarsfc-gh-ridasafdar
authored
Merge branch 'master' into coco-control-hub
2 parents ffc7622 + 4f635f8 commit 6026555

19 files changed

Lines changed: 2060 additions & 252 deletions

‎journeys/well-architected-framework/ai-data-governance-lens/well-architected-framework-ai-data-governance-lens.md‎

Lines changed: 81 additions & 80 deletions
Large diffs are not rendered by default.

‎journeys/well-architected-framework/cost-optimization/well-architected-framework-cost-optimization-and-finops.md‎

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -542,13 +542,35 @@ contains two key schemas for this purpose:
542542
(for a consolidated view across all accounts).
543543

544544

545-
| Metric Category | Description | Key Metrics | Primary Data Sources |
546-
|---|---|---|---|
547-
| Compute & query metrics | Understand the cost of query execution, warehouse consumption, and overall compute health. These are often the most dynamic and largest portion of your spend. | - Credits used: total credits by warehouse<br>- Query performance: execution time, bytes scanned, compilation time, parameterized query hash<br>- Warehouse health: % idle time, queueing, spilling, concurrency | - `ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY` (hourly warehouse credit usage)<br>- `ACCOUNT_USAGE.QUERY_HISTORY` (detailed query metrics and associated warehouses) |
548-
| Storage metrics | Costs for compressed data, including active data, Time Travel, and Fail‑safe. | - Storage volume (avg monthly compressed GB/TB)<br>- Inactive storage (Time Travel, Fail‑safe)<br>- Storage growth rates<br>- Table access (stale/unused) | - `ACCOUNT_USAGE.TABLE_STORAGE_METRICS`<br>- `ACCOUNT_USAGE.DATABASE_STORAGE_USAGE_HISTORY`<br>- `ACCOUNT_USAGE.ACCESS_HISTORY` |
549-
| Serverless & AI metrics | Track credit consumption by Snowflake‑managed services and AI features. | - Credits used by service<br>- Cost per credit‑consuming events | - `ACCOUNT_USAGE.<Serverless Feature>_HISTORY`<br>- `ORGANIZATION_USAGE.METERING_DAILY_HISTORY`<br>- AI views such as `CORTEX_FUNCTIONS_USAGE_HISTORY`, `CORTEX_ANALYST_USAGE_HISTORY`, `DOCUMENT_AI_USAGE_HISTORY` |
550-
| Data transfer | Cost of moving data into (ingress) and out of (egress) Snowflake, especially cross‑region/cloud. | - Bytes transferred<br>- Transfer cost by destination<br>- Replication vs. egress | - `ACCOUNT_USAGE.DATA_TRANSFER_HISTORY`<br>- `ORGANIZATION_USAGE.DATA_TRANSFER_DAILY_HISTORY` |
551-
| Financial metrics | Translate credits to currency and provide org‑wide spend view. | - Overall dollar spend (daily)<br>- Spend by service type | - `ORGANIZATION_USAGE.USAGE_IN_CURRENCY_DAILY`<br>- `ORGANIZATION_USAGE.RATE_SHEET_DAILY` |
545+
**Compute & query metrics**
546+
547+
| Description | Key Metrics | Primary Data Sources |
548+
|---|---|---|
549+
| Understand the cost of query execution, warehouse consumption, and overall compute health. These are often the most dynamic and largest portion of your spend. | - Credits used: total credits by warehouse<br>- Query performance: execution time, bytes scanned, compilation time, parameterized query hash<br>- Warehouse health: % idle time, queueing, spilling, concurrency | - `ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY` (hourly warehouse credit usage)<br>- `ACCOUNT_USAGE.QUERY_HISTORY` (detailed query metrics and associated warehouses) |
550+
551+
**Storage metrics**
552+
553+
| Description | Key Metrics | Primary Data Sources |
554+
|---|---|---|
555+
| Costs for compressed data, including active data, Time Travel, and Fail‑safe. | - Storage volume (avg monthly compressed GB/TB)<br>- Inactive storage (Time Travel, Fail‑safe)<br>- Storage growth rates<br>- Table access (stale/unused) | - `ACCOUNT_USAGE.TABLE_STORAGE_METRICS`<br>- `ACCOUNT_USAGE.DATABASE_STORAGE_USAGE_HISTORY`<br>- `ACCOUNT_USAGE.ACCESS_HISTORY` |
556+
557+
**Serverless & AI metrics**
558+
559+
| Description | Key Metrics | Primary Data Sources |
560+
|---|---|---|
561+
| Track credit consumption by Snowflake‑managed services and AI features. | - Credits used by service<br>- Cost per credit‑consuming events | - `ACCOUNT_USAGE.<Serverless Feature>_HISTORY`<br>- `ORGANIZATION_USAGE.METERING_DAILY_HISTORY`<br>- AI views such as `CORTEX_FUNCTIONS_USAGE_HISTORY`, `CORTEX_ANALYST_USAGE_HISTORY`, `DOCUMENT_AI_USAGE_HISTORY` |
562+
563+
**Data transfer**
564+
565+
| Description | Key Metrics | Primary Data Sources |
566+
|---|---|---|
567+
| Cost of moving data into (ingress) and out of (egress) Snowflake, especially cross‑region/cloud. | - Bytes transferred<br>- Transfer cost by destination<br>- Replication vs. egress | - `ACCOUNT_USAGE.DATA_TRANSFER_HISTORY`<br>- `ORGANIZATION_USAGE.DATA_TRANSFER_DAILY_HISTORY` |
568+
569+
**Financial metrics**
570+
571+
| Description | Key Metrics | Primary Data Sources |
572+
|---|---|---|
573+
| Translate credits to currency and provide org‑wide spend view. | - Overall dollar spend (daily)<br>- Spend by service type | - `ORGANIZATION_USAGE.USAGE_IN_CURRENCY_DAILY`<br>- `ORGANIZATION_USAGE.RATE_SHEET_DAILY` |
552574

553575

554576

‎journeys/well-architected-framework/operational-excellence/well-architected-framework-operational-excellence.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -307,9 +307,7 @@ users report them.
307307
statement. For instance, a task can run every five minutes to look
308308
for failed tasks.
309309

310-
4. **Trigger alerts:** From the task, use an **External Function** to
311-
send a notification directly to your team's incident management
312-
tool, like PagerDuty or Slack.
310+
4. **Trigger alerts:** From the task, use a **Notification Integration** (webhook-based, supporting Slack, PagerDuty, email, and more) to send a notification directly to your team's incident management tool. This is the recommended approach — it requires no additional infrastructure, unlike External Functions.
313311

314312
Build your automated alert system by following the guide [Introduction to
315313
Tasks](https://docs.snowflake.com/en/user-guide/tasks-intro).
@@ -582,7 +580,7 @@ preparation to start collecting and visualizing observability data.
582580
|---|---|
583581
| Cost & performance intelligence | - **Configure resource monitors:** set up warehouse-level monitors to prevent budget overruns by suspending warehouses or sending notifications at defined credit thresholds.<br>- **Build foundational dashboards:** create Snowsight dashboards to visualize credit usage by warehouse/tag, identify long-running queries (`QUERY_HISTORY`), and monitor warehouse queuing. |
584582
| Workload health & reliability | - **Implement error notifications:** configure notifications for failed tasks (`SYSTEM$SEND_EMAIL`) or Snowpipe copy errors to immediately alert the responsible teams.<br>- **Monitor data ingestion:** use the `COPY_HISTORY` and `PIPE_USAGE_HISTORY` views to track the latency and health of data loading processes. |
585-
| Security & access analytics | - **Enable access monitoring:** build dashboards on top of the `ACCESS_HISTORY` and `LOGIN_HISTORY` views to visualize user login patterns, query activity on sensitive tables, and privilege grants.<br>- **Set up security alerts:**** implement Snowflake alerts to trigger notifications for defined security events, such as a user being granted the `ACCOUNTADMIN` role. |
583+
| Security & access analytics | - **Enable access monitoring:** build dashboards on top of the `ACCESS_HISTORY` and `LOGIN_HISTORY` views to visualize user login patterns, query activity on sensitive tables, and privilege grants.<br>- **Set up security alerts:** implement Snowflake alerts to trigger notifications for defined security events, such as a user being granted the `ACCOUNTADMIN` role. |
586584
| Data integrity & lineage | - **Deploy data quality tests:** implement data quality checks as part of your data transformation pipeline (e.g., using dbt tests) that run on a schedule.<br>- **Utilize object tagging for lineage:** apply tags to tables and columns to create a basic, searchable framework for tracking data lineage. |
587585

588586

@@ -989,6 +987,8 @@ shorten the time from detection to diagnosis.
989987

990988
Centralize incident detection with ACCOUNT_USAGE views
991989

990+
> **Note:** ACCOUNT_USAGE views have up to 45 minutes of latency. They are best suited for trend analysis, dashboards, and scheduled alerting — not real-time incident detection. For zero-latency detection of active events, use **INFORMATION_SCHEMA** table functions (e.g., `INFORMATION_SCHEMA.QUERY_HISTORY()`, `INFORMATION_SCHEMA.LOGIN_HISTORY()`) which reflect data immediately.
991+
992992
- **Action**: Your on-call SRE and Engineering teams should treat the
993993
SNOWFLAKE.ACCOUNT_USAGE schema as the definitive event log. Build
994994
pre-configured Snowsight dashboards and alerts that immediately

‎journeys/well-architected-framework/well-architected-framework.json‎

Lines changed: 45 additions & 161 deletions
Original file line numberDiff line numberDiff line change
@@ -3,209 +3,93 @@
33
{
44
"id": "well-architected-framework",
55
"label": "Well Architected Framework",
6-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/",
6+
"url": "https://www.snowflake.com/en/developers/guides/",
77
"subcategories": [
88
{
99
"id": "cost-optimization",
1010
"label": "Cost Optimization",
11-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/cost-optimization/",
11+
"url": "https://www.snowflake.com/en/developers/guides/cost-optimization/",
1212
"topics": [
1313
{
14-
"id": "business-impact",
15-
"label": "Business Impact",
16-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/cost-optimization/business-impact/"
17-
},
18-
{
19-
"id": "visibility",
20-
"label": "Visibility",
21-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/cost-optimization/visibility/"
22-
},
23-
{
24-
"id": "control",
25-
"label": "Control",
26-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/cost-optimization/control/"
27-
},
28-
{
29-
"id": "optimize",
30-
"label": "Optimize",
31-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/cost-optimization/optimize/"
14+
"id": "overview",
15+
"label": "Overview",
16+
"url": "https://www.snowflake.com/en/developers/guides/cost-optimization/#overview"
3217
}
3318
]
3419
},
3520
{
36-
"id": "security-governance",
21+
"id": "security-and-governance",
3722
"label": "Security & Governance",
38-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/security-governance/",
23+
"url": "https://www.snowflake.com/en/developers/guides/security-and-governance/",
3924
"topics": [
4025
{
41-
"id": "secure-the-perimeter",
42-
"label": "Secure the Perimeter",
43-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/security-governance/secure-the-perimeter/"
44-
},
45-
{
46-
"id": "data-governance",
47-
"label": "Data Governance",
48-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/security-governance/data-governance/"
49-
},
50-
{
51-
"id": "responsible-ai",
52-
"label": "Responsible AI",
53-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/security-governance/responsible-ai/"
54-
},
55-
{
56-
"id": "assume-breach",
57-
"label": "Assume Breach",
58-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/security-governance/assume-breach/"
26+
"id": "overview",
27+
"label": "Overview",
28+
"url": "https://www.snowflake.com/en/developers/guides/security-and-governance/#overview"
5929
}
6030
]
6131
},
6232
{
6333
"id": "reliability",
6434
"label": "Reliability",
65-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/reliability/",
35+
"url": "https://www.snowflake.com/en/developers/guides/reliability/",
6636
"topics": [
6737
{
68-
"id": "strategy-governance",
69-
"label": "Strategy & Governance",
70-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/reliability/strategy-governance/"
71-
},
72-
{
73-
"id": "high-availability",
74-
"label": "High Availability",
75-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/reliability/high-availability/"
76-
},
77-
{
78-
"id": "data-resiliency",
79-
"label": "Data Resiliency",
80-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/reliability/data-resiliency/"
81-
},
82-
{
83-
"id": "resilient-pipelines",
84-
"label": "Resilient Pipelines",
85-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/reliability/resilient-pipelines/"
86-
},
87-
{
88-
"id": "monitoring-operations",
89-
"label": "Monitoring & Operations",
90-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/reliability/monitoring-operations/"
91-
},
92-
{
93-
"id": "disaster-recovery",
94-
"label": "Disaster Recovery",
95-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/reliability/disaster-recovery/"
38+
"id": "overview",
39+
"label": "Overview",
40+
"url": "https://www.snowflake.com/en/developers/guides/reliability/#overview"
9641
}
9742
]
9843
},
9944
{
10045
"id": "performance",
10146
"label": "Performance",
102-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/",
47+
"url": "https://www.snowflake.com/en/developers/guides/performance/",
10348
"topics": [
10449
{
105-
"id": "performance-objectives",
106-
"label": "Performance Objectives",
107-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/performance-objectives/"
108-
},
109-
{
110-
"id": "foundational-first-steps",
111-
"label": "Foundational First Steps",
112-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/foundational-first-steps/"
113-
},
114-
{
115-
"id": "elasticity",
116-
"label": "Leveraging Elasticity",
117-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/elasticity/"
118-
},
119-
{
120-
"id": "test-first-design",
121-
"label": "Test-First Design",
122-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/test-first-design/"
123-
},
124-
{
125-
"id": "data-architecture",
126-
"label": "Data Architecture",
127-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/data-architecture/"
128-
},
129-
{
130-
"id": "data-access",
131-
"label": "Data Access",
132-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/data-access/"
133-
},
134-
{
135-
"id": "scalability",
136-
"label": "Scalability",
137-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/scalability/"
138-
},
139-
{
140-
"id": "warehouses",
141-
"label": "Warehouse Optimization",
142-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/warehouses/"
143-
},
144-
{
145-
"id": "concurrency",
146-
"label": "Concurrency",
147-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/concurrency/"
148-
},
149-
{
150-
"id": "serverless",
151-
"label": "Serverless Features",
152-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/serverless/"
153-
},
154-
{
155-
"id": "dynamic-tables",
156-
"label": "Dynamic Tables",
157-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/dynamic-tables/"
158-
},
159-
{
160-
"id": "monitoring",
161-
"label": "Performance Monitoring",
162-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/performance/monitoring/"
50+
"id": "overview",
51+
"label": "Overview",
52+
"url": "https://www.snowflake.com/en/developers/guides/performance/#overview"
16353
}
16454
]
16555
},
16656
{
16757
"id": "operational-excellence",
16858
"label": "Operational Excellence",
169-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/",
59+
"url": "https://www.snowflake.com/en/developers/guides/operational-excellence/",
17060
"topics": [
17161
{
172-
"id": "operational-readiness",
173-
"label": "Operational Readiness",
174-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/operational-readiness/"
175-
},
176-
{
177-
"id": "automation",
178-
"label": "Automation",
179-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/automation/"
180-
},
181-
{
182-
"id": "observability",
183-
"label": "Observability",
184-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/observability/"
185-
},
186-
{
187-
"id": "incident-management",
188-
"label": "Incident Management",
189-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/incident-management/"
190-
},
191-
{
192-
"id": "collaboration",
193-
"label": "Collaboration & Sharing",
194-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/collaboration/"
195-
},
62+
"id": "overview",
63+
"label": "Overview",
64+
"url": "https://www.snowflake.com/en/developers/guides/operational-excellence/#overview"
65+
}
66+
]
67+
},
68+
{
69+
"id": "ai-data-governance-lens",
70+
"label": "AI & Data Governance Lens",
71+
"url": "https://www.snowflake.com/en/developers/guides/ai-data-governance-lens/",
72+
"topics": [
19673
{
197-
"id": "sdlc",
198-
"label": "SDLC Management",
199-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/sdlc/"
200-
},
74+
"id": "overview",
75+
"label": "Overview",
76+
"url": "https://www.snowflake.com/en/developers/guides/ai-data-governance-lens/#overview"
77+
}
78+
]
79+
},
80+
{
81+
"id": "interoperable",
82+
"label": "Open and Interoperable",
83+
"url": "https://www.snowflake.com/en/developers/guides/interoperable/",
84+
"topics": [
20185
{
202-
"id": "continuous-improvement",
203-
"label": "Continuous Improvement",
204-
"url": "https://www.snowflake.com/en/developers/guides/well-architected-framework/operational-excellence/continuous-improvement/"
86+
"id": "overview",
87+
"label": "Overview",
88+
"url": "https://www.snowflake.com/en/developers/guides/interoperable/#overview"
20589
}
20690
]
20791
}
20892
]
20993
}
210-
]
94+
]
21195
}

‎site/sfguides/src/build-autonomous-sql-pipelines-for-ai-agents/assets/lab_architecture.png‎

342 KB
Loading

‎site/sfguides/src/build-autonomous-sql-pipelines-for-ai-agents/assets/openflow_destination_params.png‎

90.3 KB
Loading

‎site/sfguides/src/build-autonomous-sql-pipelines-for-ai-agents/assets/openflow_enable_controller_services.png‎

125 KB
Loading

‎site/sfguides/src/build-autonomous-sql-pipelines-for-ai-agents/assets/openflow_import_flow.png‎

110 KB
Loading

‎site/sfguides/src/build-autonomous-sql-pipelines-for-ai-agents/assets/openflow_import_from_registry.png‎

130 KB
Loading

‎site/sfguides/src/build-autonomous-sql-pipelines-for-ai-agents/assets/openflow_ingestion_params.png‎

191 KB
Loading

0 commit comments

Comments
 (0)