Skip to content

Commit cb567d7

Browse files
akshay-vizCopilot
andcommitted
chore(model-apps): remove real environment identifiers from a public repo
The genpage eval fixtures are captured agent transcripts, so they recorded whatever live environment each eval ran against — including `pac auth list` output carrying the operator's UPN, tenant, and environment URL. That had accumulated to 98 occurrences of three environments, two tenants and two user accounts across 28 files, plus provenance comments in four scripts and the roadmap. Fixture replacements are equal-length. The fixtures embed `pac auth list` output as a fixed-width table, so a shorter placeholder would misalign every following column and make a captured artifact look hand-edited. Verified: the diff is exactly 65 insertions / 65 deletions with no line-ending drift (the fixtures are LF, the scripts CRLF). Script provenance comments are generalised rather than substituted. "Verified live on <envname>" carries a real claim; swapping in a fake environment name would keep the sentence readable but make it false, so the claim is kept and the environment dropped ("verified live on a Dataverse test environment"). Adds `scripts/validate-no-real-environments.js` so the scrub sticks. Pasting a fresh live transcript is the realistic regression path and it looks correct to a reviewer. The guard matches on shape, not just on the strings removed here: `org<8 hex>` is what Dataverse auto-generates, so it is rejected even though it starts with the otherwise-allowed word `org`. Scoped to model-apps, because other plugins still carry pre-existing references of this class and widening the scan today would fail unrelated PRs; the limitation is documented in the script header and in AGENTS.md rather than left implicit. No behaviour change: 1474 plugin tests, 159 eval tests, and 12/12 layer-1 fixtures still pass. The layer-1 assertions match generic URL shapes (`/https?:\/\/[a-z0-9-]+\.(crm|dynamics)/i`), not any specific environment, so the rewritten fixtures still exercise them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 42626da2-b66f-4162-acaa-b1127ef23d89
1 parent d2ec894 commit cb567d7

34 files changed

Lines changed: 394 additions & 65 deletions

File tree

.github/workflows/validate-repository-metadata.yml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ jobs:
1010
runs-on: ubuntu-latest
1111
env:
1212
POWER_PLATFORM_SKILLS_TELEMETRY_POWER_PAGES_OPTOUT: "1"
13+
POWER_PLATFORM_SKILLS_TELEMETRY_MODEL_APPS_OPTOUT: "1"
1314
steps:
1415
- name: checkout
1516
uses: actions/checkout@v4
@@ -36,3 +37,13 @@ jobs:
3637

3738
- name: validate-secure-process-execution
3839
run: node scripts/validate-secure-process-execution.js
40+
41+
# This repository is public, and the genpage eval fixtures are captured
42+
# agent transcripts that will faithfully record whatever live environment
43+
# an eval was run against. Guarding this in CI is the only reliable
44+
# control: a pasted transcript looks correct to a reviewer.
45+
- name: test-no-real-environments-validator
46+
run: node --test scripts/tests/validate-no-real-environments.test.js
47+
48+
- name: validate-no-real-environments
49+
run: node scripts/validate-no-real-environments.js

AGENTS.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,21 @@ rebuild a vendored bundle. Both carry real "why" context and disclose no content
3737
a committed file. When editing an existing doc, keep this rule in mind for the lines around your
3838
change, not just the ones you add.
3939

40+
**CI enforcement (partial).** `node scripts/validate-no-real-environments.js` (wired into the
41+
`validate-repository-metadata` workflow) fails the build when a real Dataverse host, tenant, or
42+
previously-removed identifier appears under `plugins/model-apps/**` or `evals/model-apps/**`. It
43+
matches on *shape*`org<8 hex>` is what Dataverse auto-generates, so it is rejected even though it
44+
starts with the otherwise-allowed word `org` — rather than only re-catching known strings. Run it
45+
locally after touching eval fixtures or any file that quotes an environment URL.
46+
47+
The scan is **scoped to model-apps only**, and this is a real gap rather than an oversight: other
48+
plugins still carry pre-existing references of this class (for example real `org<8 hex>` orgs cited
49+
in power-pages provenance comments), so widening the scan today would fail unrelated PRs. Scrub a
50+
plugin first, then add it to `SCAN_PATHS`. The guard also cannot see the *local part* of a UPN, so
51+
`firstname.lastname@contoso.onmicrosoft.com` passes — use a role word (`maker@`, `tester@`).
52+
Captured `pac auth list` transcripts are the most common source of all three; when scrubbing one,
53+
prefer an **equal-length** placeholder so the fixed-width table stays aligned.
54+
4055
## Repository Structure
4156

4257
```

evals/model-apps/genpage/fixtures/1-account-card-gallery/genpage-plan.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ D:\Projects\power-platform-skills\plugins\model-apps
1616

1717
## Environment
1818

19-
- Active Profile: aurora365-user1@auroratstgeo.onmicrosoft.com
20-
- URL: https://aurorabapenv4ab3f.crm10.dynamics.com/
19+
- Active Profile: contoso-user001@contosotest1.onmicrosoft.com
20+
- URL: https://contosobapenv0002.crm10.dynamics.com/
2121
- App: Sales Hub (12345678-1234-1234-1234-123456789abc)
2222
- Languages: English (1033) only
2323
- Solution: Default

evals/model-apps/genpage/fixtures/1-account-card-gallery/workflow-log.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212
- (Run separately, not chained with &&)
1313

1414
### Auth check
15-
- `pac auth list` → active profile aurora365-user1@auroratstgeo.onmicrosoft.com
16-
- Active environment: https://aurorabapenv4ab3f.crm10.dynamics.com/ (reported to user)
15+
- `pac auth list` → active profile contoso-user001@contosotest1.onmicrosoft.com
16+
- Active environment: https://contosobapenv0002.crm10.dynamics.com/ (reported to user)
1717

1818
### Entity discovery
1919
- `pac model list-tables --search 'account'` — Account entity detected as existing (exact logical-name match: `account`)

evals/model-apps/genpage/fixtures/11-recruitment-multi-page/genpage-plan.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ D:\Projects\power-platform-skills\plugins\model-apps
1616

1717
## Environment
1818

19-
- Active Profile: aurora365-user1@auroratstgeo.onmicrosoft.com
20-
- URL: https://aurorabapenv4ab3f.crm10.dynamics.com/
19+
- Active Profile: contoso-user001@contosotest1.onmicrosoft.com
20+
- URL: https://contosobapenv0002.crm10.dynamics.com/
2121
- App: Recruitment Hub (44444444-3333-4444-5555-666666666666)
2222
- Languages: English (1033) only
2323
- Solution: Default

evals/model-apps/genpage/fixtures/11-recruitment-multi-page/workflow-log.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,8 @@
1111
- `pac help` → PAC CLI Version 2.11.0 (> 2.10.0 verified)
1212

1313
### Auth check
14-
- `pac auth list` → active profile aurora365-user1@auroratstgeo.onmicrosoft.com
15-
- Active environment: https://aurorabapenv4ab3f.crm10.dynamics.com/ (reported to user)
14+
- `pac auth list` → active profile contoso-user001@contosotest1.onmicrosoft.com
15+
- Active environment: https://contosobapenv0002.crm10.dynamics.com/ (reported to user)
1616

1717
### Entity discovery
1818
- `pac model list-tables --search 'contact,appointment'` — both entities detected as existing

evals/model-apps/genpage/fixtures/11-recruitment-pages-real/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ First real capture taken after the v2.2 planner-spec tightening landed.
66
## Source
77

88
Captured from a real `/genpage` session against
9-
`https://aurorabapenv610b3.crmtest.dynamics.com`. Driven by
9+
`https://contosobapenv0001.crmtest.dynamics.com`. Driven by
1010
`claude-sonnet-4-6` under the v2.1 plugin + v2.2 Phase 0.5 manifest generator
1111
+ v2.2 spec-tightening. Working dir was `D:/temp/recruitment-app/`.
1212

evals/model-apps/genpage/fixtures/11-recruitment-pages-real/genpage-plan.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ D:/temp/recruitment-app
1010
D:/Projects/power-platform-skills/plugins/model-apps
1111

1212
## Environment
13-
- URL: https://aurorabapenv610b3.crmtest.dynamics.com
13+
- URL: https://contosobapenv0001.crmtest.dynamics.com
1414
- App: create new: Recruitment App
1515
- Languages: English (1033) only
1616
- Solution: Crdec34

evals/model-apps/genpage/fixtures/11-recruitment-pages-real/workflow-log.md

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,13 @@ PAC CLI Version 2.11.0+g06bb2eb (.NET 10.0.8)
1313
`pac auth list`
1414
Index Active Kind Name User Cloud Type Environment Environment Url
1515
[1] UNIVERSAL akmaloo@microsoft.com Public OperatingSystem
16-
[2] UNIVERSAL aurorauser14@capintegration01.onmicrosoft.com Test User TMSBAPEnv5ee52 https://tmsbapenv5ee52.crmtest.dynamics.com/
17-
[3] * UNIVERSAL Aurora365-User1@auroratstgeo.onmicrosoft.com Test User AuroraBAPEnv610b3 https://aurorabapenv610b3.crmtest.dynamics.com/
16+
[2] UNIVERSAL fabrikamusr1@fabrikamtenant01.onmicrosoft.com Test User FabrikamEnv001 https://fabrikamenv001.crmtest.dynamics.com/
17+
[3] * UNIVERSAL Contoso-User001@contosotest1.onmicrosoft.com Test User ContosoBAPEnv0001 https://contosobapenv0001.crmtest.dynamics.com/
1818

19-
AskUserQuestion: Which environment should be used for the recruitment app? → [3] Aurora365-User1@auroratstgeo.onmicrosoft.comAuroraBAPEnv610b3 (currently active)
19+
AskUserQuestion: Which environment should be used for the recruitment app? → [3] Contoso-User001@contosotest1.onmicrosoft.comContosoBAPEnv0001 (currently active)
2020

2121
`pac auth select --index 3`
22-
New default profile: * UNIVERSAL : Aurora365-User1@auroratstgeo.onmicrosoft.com Test https://aurorabapenv610b3.crmtest.dynamics.com/
22+
New default profile: * UNIVERSAL : Contoso-User001@contosotest1.onmicrosoft.com Test https://contosobapenv0001.crmtest.dynamics.com/
2323

2424
`pac model list-languages`
2525
Found 1 enabled language(s):
@@ -75,7 +75,7 @@ User response: approved
7575
- Files written: package.json, genpage.d.ts (features: charts)
7676

7777
### Decisions
78-
- Environment: AuroraBAPEnv610b3 (https://aurorabapenv610b3.crmtest.dynamics.com/)
78+
- Environment: ContosoBAPEnv0001 (https://contosobapenv0001.crmtest.dynamics.com/)
7979
- PAC CLI: 2.11.0 (> 2.10.0 requirement met)
8080
- Node: v20.18.2
8181
- Languages: English only — no localization code needed

evals/model-apps/genpage/fixtures/13-contact-localization/genpage-plan.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,8 +14,8 @@ D:\Projects\power-platform-skills\plugins\model-apps
1414

1515
## Environment
1616

17-
- Active Profile: aurora365-user1@auroratstgeo.onmicrosoft.com
18-
- URL: https://aurorabapenv4ab3f.crm10.dynamics.com/
17+
- Active Profile: contoso-user001@contosotest1.onmicrosoft.com
18+
- URL: https://contosobapenv0002.crm10.dynamics.com/
1919
- App: Sales Hub (55555555-4444-5555-6666-777777777777)
2020
- Languages: English (1033) only
2121
- Solution: Default

0 commit comments

Comments
 (0)