Skip to content

Commit 7df0e2e

Browse files
author
Zach Olinske
committed
Update README and changelog with targeted deletion + ADX limitation
- Fix quick-start examples: ADX requires -Force, storage-only example added - Update Remove-FinOpsTestData section with targeted folder deletion - Add ADX limitation warning to README - Update changelog entry to describe targeted deletion accurately
1 parent c8057ba commit 7df0e2e

File tree

2 files changed

+13
-9
lines changed

2 files changed

+13
-9
lines changed

docs-mslearn/toolkit/changelog.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ The following section lists features and enhancements that are currently in deve
4444

4545
- **Added**
4646
- Added [New-FinOpsTestData](powershell/data/New-FinOpsTestData.md) to generate synthetic, multi-cloud FOCUS-compliant cost data for Azure, AWS, GCP, and on-premises providers. Supports FOCUS versions 1.0-1.3 with version-specific column sets, commitment discounts, Azure Hybrid Benefit, tag variation, and inline budget scaling. Includes `-Seed` for reproducibility and upload to Azure storage. Generated manifests include a `_ftkTestData` watermark for safe cleanup identification.
47-
- Added [Remove-FinOpsTestData](powershell/data/Remove-FinOpsTestData.md) to purge test data from Azure Data Explorer tables, storage containers, and local test-data folders for a clean reset. Includes multi-layer safety features: local-only cleanup by default, storage marker verification (checks for `_ftkTestData` in manifests before deleting), and ADX cleanup requires `-Force`. Supports optional ADF trigger management and ADX update policy verification.
47+
- Added [Remove-FinOpsTestData](powershell/data/Remove-FinOpsTestData.md) to purge test data from Azure Data Explorer tables, storage containers, and local test-data folders for a clean reset. Includes multi-layer safety features: local-only cleanup by default, targeted storage deletion (scans manifests for `_ftkTestData` marker and removes only test-data folders — production data is preserved), and ADX cleanup requires `-Force` because `.clear table` removes all rows. Supports optional ADF trigger management and ADX update policy verification.
4848

4949
<br>
5050

src/templates/finops-hub/test/README.md

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,11 @@ New-FinOpsTestData -FocusVersion 1.0 -ServiceProvider Azure -RowCount 50000
3030
# Generate and upload to Azure Storage with ADF trigger management
3131
New-FinOpsTestData -StorageAccountName "stfinopshub" -ResourceGroupName "rg-finopshub" -AdfName "adf-finopshub" -StartTriggers
3232
33-
# Remove all test data (ADX tables, storage blobs, local files) for a clean reset
34-
Remove-FinOpsTestData -AdxClusterUri "https://mycluster.region.kusto.windows.net" -StorageAccountName "stfinopshub"
33+
# Remove test-data folders from storage (production data is preserved)
34+
Remove-FinOpsTestData -StorageAccountName "stfinopshub"
3535
36-
# Full cleanup with ADF trigger management
37-
Remove-FinOpsTestData -AdxClusterUri "https://mycluster.region.kusto.windows.net" -StorageAccountName "stfinopshub" -AdfName "adf-finopshub" -ResourceGroupName "rg-finopshub" -StopTriggers
36+
# Full cleanup including ADX (requires -Force because .clear table removes ALL rows)
37+
Remove-FinOpsTestData -AdxClusterUri "https://mycluster.region.kusto.windows.net" -StorageAccountName "stfinopshub" -AdfName "adf-finopshub" -ResourceGroupName "rg-finopshub" -StopTriggers -Force
3838
```
3939

4040
### Features
@@ -80,14 +80,18 @@ Plus FinOps Hub-specific `x_` prefixed extension columns for dashboard compatibi
8080
8181
### Remove-FinOpsTestData (Clean Reset)
8282

83-
The `Remove-FinOpsTestData` command performs a full cleanup for re-testing:
83+
The `Remove-FinOpsTestData` command performs a targeted cleanup for re-testing:
8484

8585
1. **Stops ADF triggers** (optional, with `-StopTriggers -AdfName -ResourceGroupName`) to prevent re-ingestion during cleanup
86-
2. **Purges all ADX tables** in both Hub and Ingestion databases via REST API
86+
2. **Purges all ADX tables** in both Hub and Ingestion databases via REST API (requires `-Force`)
8787
3. **Verifies ADX update policies** are intact after clearing tables
88-
4. **Deletes all blobs** from `msexports` and `ingestion` storage containers
88+
4. **Deletes test-data folders from storage** — scans manifests for `_ftkTestData` marker and removes only marked folders (production data is preserved)
8989
5. **Removes local test-data folder**
9090

91-
Requires `-AdxClusterUri` and `-StorageAccountName`. Uses `ConfirmImpact = 'High'` so PowerShell will prompt for confirmation unless `-Confirm:$false` is passed.
91+
All cloud parameters are optional. By default only local files are deleted. Pass `-StorageAccountName` for storage cleanup and `-AdxClusterUri -Force` for ADX cleanup.
92+
93+
> **Important:** Do not ingest test data into an ADX cluster that also contains production data. Storage cleanup is targeted (only test-data folders are removed), but ADX `.clear table` removes **all** rows. If you accidentally mix test and production data in ADX, clear the tables and re-ingest production data from storage.
94+
95+
Uses `ConfirmImpact = 'High'` so PowerShell will prompt for confirmation unless `-Confirm:$false` is passed.
9296

9397
Run `Get-Help Remove-FinOpsTestData -Detailed` for the full parameter reference.

0 commit comments

Comments
 (0)