Skip to content

Commit 5dfbbd4

Browse files
tyaginidhiclaudeCopilot
authored
[Pages][ALM] Site-referenced table discovery + dependency-aware solution splitting (#193)
* EDM (enhanced/standard data-model) Power Pages site support in ALM discovery Enhanced data-model sites are downloaded with `pac pages download` (not download-code-site): they have NO `powerpages.config.json` and no SPA build output — just `.powerpages-site/` with a config tree + `website.yml`. The shared discovery helpers hard-required `powerpages.config.json`, so every ALM skill broke on EDM sites. - `findProjectRoot` (validation-helpers.js): treats a `.powerpages-site/` directory as a project-root marker, not just `powerpages.config.json`. - `detect-project-context.js`: falls back to `.powerpages-site/website.yml` (`id`→websiteRecordId, `name`→siteName) when no `powerpages.config.json`; returns a new `siteType` ("code" | "data-model"); exits 1 only when neither marker exists. - `check-activation-status.js`: same fallback (verified live against an EDM site → resolves identity + activation status instead of erroring). Backward-compatible: `powerpages.config.json` stays the primary signal for code sites. PR 2 of 4 (stacked, on plan-alm-plan-only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address #192 review: drop "standard" data-model wording + nits + test coverage - PR comment (do not refer "standard data model"): the skill supports SPA and EDM (enhanced data model) only. Removed every "standard/enhanced" / "standard and enhanced" reference in detect-project-context.js, validation-helpers.js, and AGENTS.md — now "enhanced data-model (EDM)". - Finding 1 (test coverage): check-activation-status.js had no test (CLAUDE.md requires tests for modified scripts). Extracted resolveSiteIdentity() (deps injectable) + guarded the CLI flow behind `require.main === module`, then added check-activation-status.test.js (7 cases): EDM website.yml resolution + SKIPS pac pages list, code-site-with-GUID skips it, code-site-without-GUID consults it, neither-marker error, malformed-config error, pac-failure non-fatal. - Finding 2 (style): braced the bare `if (!websiteRecordId) try {…}`. - Finding 3 (robustness): readWebsiteYml now strips inline YAML comments on unquoted values (`id: abc # note` -> `abc`); a `#` inside quotes is left intact. CLI behavior unchanged (require.main guard verified). 1202 tests pass, alm-lint 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review round 2: terminology (B) + detection signal (3) + AGENTS (4) Per discussion (verified against MS Learn — "enhanced data model" is a Dataverse STORAGE axis, orthogonal to the code-vs-declarative BUILD axis): - (B) Terminology: stop labeling the .powerpages-site/ path "EDM" specifically — a declarative site can be on the standard OR enhanced data model, and code sites can also be EDM. siteType "data-model" now reads as the "declarative (design-studio)" bucket (kept for compat with plan-alm); every "EDM"-exclusive comment now says "declarative … (standard or enhanced data model)". A full rename to "declarative" is tracked as a separate follow-up (Option A). - (3) Detection: added `.powerpages-site/.portalconfig/` as the AUTHORITATIVE positive declarative marker (only declarative sites have it; BOTH site types carry website.yml, so website.yml alone isn't proof of "declarative" — this matches plan-alm's #191 logic and guards against misclassifying a config-less checkout). website.yml stays the identity source. New isDirectory() helper + a .portalconfig-without-website.yml test. - (4) AGENTS.md: refreshed the stale check-activation-status.js entry (now describes the config → website.yml → pac pages list order + the GUID-known skip + resolveSiteIdentity export), and the detect-project-context.js entry (build-vs-data-model axis note + the .portalconfig signal). 1203 tests pass, alm-lint 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Site-referenced table discovery + dependency-aware solution splitting Replaces publisher-prefix table discovery (which over-counts catastrophically with a shared/default publisher — a 6-table site matched 22 unrelated tables — and misses real tables from a different prefix) with site-referenced scoping: the custom tables the site's `.powerpages-site/table-permissions/` (+ datamodel manifest) actually reference, intersected with the env's custom-unmanaged tables. SME-confirmed: table permissions are the complete signal. Replaces the old one-solution-per-table-name-stem split heuristic (which produced ~one solution per table — e.g. a 21-solution split) with a dependency-aware, capacity-bounded packer: union-find connected-component clusters over table relationships (lookups + N:N), then first-fit-decreasing bin-packing of whole clusters into the fewest solutions under maxTableCount/maxSchemaAttrs, capped at maxSchemaSplitSolutions (8). The split trigger + thresholds are unchanged — only the packing. New shared libs: - resolve-site-tables.js — site-referenced table scoping (single source of truth) - query-metadata.js — consolidated custom-unmanaged-table query - query-table-relationships.js — relationship edges (lib; audit-permissions CLI is now a thin wrapper) - validation-helpers.js — odataGet/odataGetAll shared paginator estimate-solution-size.js now emits tableCountScope + tableRelationships[]; compute-split-plan.js consumes the edges. setup-solution Phase 5.2.D uses the shared discovery helper. 1209 tests pass, alm-lint 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix FFD bin-packing under-allocation that overflowed the schema-attr cap deriveDomainsByCapacity seeded the packer's bin count from a LOWER bound (max(ceil(tables/maxTableCount), ceil(attrs/maxSchemaAttrs))), so when independent (no-edge) clusters fragment, FFD ran out of bins and dropped the non-fitting cluster into the least-loaded bucket — overflowing it past maxSchemaAttrs with no warning (the oversized-cluster guard only checks per- cluster table COUNT, not attrs). Verified repro: 4 independent 8000-attr tables, maxSchemaAttrs 15000 -> seed n=3 -> one bucket holds 16000 attrs. Seed the packer with the maximum permitted bins instead (one per cluster, capped at maxSchemaSplitSolutions). FFD still consolidates — clusters that fit together share a bin and empty bins are dropped, so the solution count stays minimal — but a cluster that fits nowhere opens a NEW bin rather than overflowing. The existing 16000-attr/2-solution test is unchanged (FFD still consolidates); added a regression test for the 4-independent-table overflow case. 1211 tests pass, alm-lint 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Wire --projectRoot into the remaining discover-site-components consumers The site-referenced table discovery (this PR) made discover-site-components return customTables (and thus missing.customTables) ONLY when --projectRoot (or --datamodelManifest) is passed — without a local signal it returns [] rather than the old publisher-prefix dump. setup-solution Step D was updated to pass --projectRoot, but three other consumers that read missing.customTables were missed, so their "custom tables missing from the solution" completeness check silently reported 0 for every site: - export-solution Phase 2.5 (pre-export completeness) - plan-alm Phase 1 (pre-plan completeness) - deploy-pipeline Phase 3.5 (pre-sync completeness) Added --projectRoot "." to all three so the check is restored with the correct site-scoped count. (setup-solution Phase 5.4b/5.4c calls consume missing.envVars / missing.powerpagecomponents / siteLanguages, not customTables, so they're correct as-is with --publisherPrefix/--solutionId.) 1211 tests pass, alm-lint 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1 parent f001179 commit 5dfbbd4

24 files changed

Lines changed: 1198 additions & 136 deletions

marketplace.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313
"source": "./plugins/power-pages",
1414
"description": "Power Pages development and management plugin for Claude Code and GitHub Copilot",
1515
"category": "development",
16-
"version": "2.2.0",
16+
"version": "2.4.0",
1717
"license": "MIT",
1818
"tags": [
1919
"power platform",

plugins/power-pages/.plugin/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "power-pages",
3-
"version": "2.2.0",
3+
"version": "2.4.0",
44
"description": "Create and deploy Power Pages sites using modern development approaches. Supports code sites (SPAs) with React, Angular, Vue, or Astro. Includes ALM orchestration (plan-alm) with a solution-splitting decision tree, per-solution pipelines, Azure Blob asset advisory, manifest schema v2 for multi-solution deployments, and force-link remediation for cross-host pipeline migrations.",
55
"author": {
66
"name": "Microsoft",

plugins/power-pages/AGENTS.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -204,8 +204,12 @@ Shared lib modules live at `scripts/lib/` and are imported by other scripts via
204204
#### Solution Splitting Decision Tree (v1.3.0+)
205205

206206
- `scripts/lib/alm-thresholds.js`: Central default threshold constants for the split decision tree. Loads optional `.alm-config.json` from project root and merges over defaults. Exports `DEFAULTS`, `DEFAULT_CONFIG`, `loadConfig(projectRoot)`, `classifyTier(value, greenUpperExclusive, yellowUpperExclusive)`, `deepMerge(target, source)`. Used by `estimate-solution-size.js` and `compute-split-plan.js`.
207-
- `scripts/lib/estimate-solution-size.js`: Estimates solution size + component counts by querying Dataverse. Args: `--envUrl`, `--websiteRecordId`, `--token` (opt), `--publisherPrefix` (opt), `--siteName` (opt), `--solutionId` (opt — scopes env var count to the target solution; without it falls back to a publisher-prefix tenant-wide query that overcounts when prefix is shared), `--datamodelManifest` (opt), `--projectRoot` (opt — enables disk cross-check: walks the local build-output directory (`dist/`, `public-output/`, `build/`, `.output/`) and surfaces the byte total). Output: `{ totalSizeMB, componentCountSiteTotal, componentCountSiteActionable, componentCountInSolution, orphansOnSite, tableCount, schemaAttrCount, webFilesAggregateMB, webFilesIndividual[], webFileCount, webFileSampleSize, webFilesDiskMeasuredMB, webFilesDiskMeasuredPath, webFilesDiskFileCount, cloudFlowCount, botCount, envVarCount, envVarCountScope, envVarCountTenantWide, mediaRatio, siteType, tables[], breakdown, estimationMethod, estimationAccuracyPct, truncationSuspected, truncationWarnings, ppcGroundTruthCount }`. Metadata-based estimation with ±15% caveat. Web-file size is measured via stratified sample (first 50 + middle 50 + last 50, cap 150) scaled to full count. Disk fields are null unless `--projectRoot` was passed AND a build-output directory was found. Truncation canaries fire when Dataverse pagination disagrees with `@odata.count`, when ppcs land on a page-size boundary, when sampled average bytes/file < 1 KB at scale, or when the disk total exceeds the Dataverse total by >2× — any signal flips `truncationSuspected: true` with a per-cause `truncationWarnings[]` entry. Used by `plan-alm` Phase 1 Step 10.
208-
- `scripts/lib/compute-split-plan.js`: Runs the split decision tree against a size-estimate blob. Args: `--estimate <path>`, `--projectRoot` (opt — for `.alm-config.json` overrides), `--siteName` (opt), `--publisherPrefix` (opt). Output: `{ sizeAnalysis, assetAdvisory, splitStrategy, appliedStrategies, compositeSubPartitioned, proposedSolutions[], recommendations[], truncationSuspected, truncationWarnings }`. Evaluates strategies in priority order: Strategy 3 (Schema Segmentation) → Strategy 1 (Layer Split) → Strategy 2 (Change-Frequency) → Strategy 4 (Config Isolation). Strategy 4 stacks additively. Strategy 1 also runs a composite sub-partition pass: when Core still exceeds the size OR component-count cap after Web Assets are peeled off, Core is replaced with change-frequency-shaped sub-children (`_Foundation`/`_Config`/`_Content`, plus `_Integration` whenever the parent had any flows or bots — coverage takes priority over the `changeFreqMinFlows` heuristic, which governs only the TOP-LEVEL strategy choice). When additive Strategy 4 is firing concurrently (a top-level `_EnvVars` solution), `_Config` drops `Environment Variable` from its componentTypes to avoid double-claim; when it isn't, `_Config` absorbs env vars so they have an owner. Sub-partitioning sets `compositeSubPartitioned: true` and appends `composite-sub-partition` to `appliedStrategies`. `validateSplits` checks BOTH the size AND component-count cap per split (skipping `isFutureBuffer` solutions). Supports `.alm-config.json` overrides including `strategyOverride` to bypass the tree. See `solution-splitting-logic.md` spec in design docs for full logic.
207+
- `scripts/lib/estimate-solution-size.js`: Estimates solution size + component counts by querying Dataverse. Args: `--envUrl`, `--websiteRecordId`, `--token` (opt), `--publisherPrefix` (opt), `--siteName` (opt), `--solutionId` (opt — scopes env var count to the target solution; without it falls back to a publisher-prefix tenant-wide query that overcounts when prefix is shared), `--datamodelManifest` (opt), `--projectRoot` (opt — enables disk cross-check: walks the local build-output directory (`dist/`, `public-output/`, `build/`, `.output/`) and surfaces the byte total). Output: `{ totalSizeMB, componentCountSiteTotal, componentCountSiteActionable, componentCountInSolution, orphansOnSite, tableCount, tableCountScope, schemaAttrCount, webFilesAggregateMB, webFilesIndividual[], webFileCount, webFileSampleSize, webFilesDiskMeasuredMB, webFilesDiskMeasuredPath, webFilesDiskFileCount, cloudFlowCount, botCount, envVarCount, envVarCountScope, envVarCountTenantWide, mediaRatio, siteType, tables[], tableRelationships[], breakdown, estimationMethod, estimationAccuracyPct, truncationSuspected, truncationWarnings, ppcGroundTruthCount }`. **Table discovery is site-referenced, NOT publisher-prefix:** `tableCount`/`tables[]` are scoped to the custom tables the site actually references — its `.powerpages-site/table-permissions/` (+ datamodel manifest) intersected with the env's custom-unmanaged tables (via `resolve-site-tables.js` + `query-metadata.js`). `tableCountScope` ∈ `"site-referenced" | "manifest-only" | "unavailable"` (the last → 0 tables, never an env-wide prefix dump). `--publisherPrefix` now scopes ONLY the env var count, not tables. `tableRelationships[]` are `[a,b]` dependency edges (lookups + N:N, via `query-table-relationships.js`) among the scoped tables, consumed by `compute-split-plan.js` to cluster related tables into the same solution. Metadata-based estimation with ±15% caveat. Web-file size is measured via stratified sample (first 50 + middle 50 + last 50, cap 150) scaled to full count. Disk fields are null unless `--projectRoot` was passed AND a build-output directory was found. Truncation canaries fire when Dataverse pagination disagrees with `@odata.count`, when ppcs land on a page-size boundary, when sampled average bytes/file < 1 KB at scale, or when the disk total exceeds the Dataverse total by >2× — any signal flips `truncationSuspected: true` with a per-cause `truncationWarnings[]` entry. Used by `plan-alm` Phase 1 Step 10.
208+
- `scripts/lib/compute-split-plan.js`: Runs the split decision tree against a size-estimate blob. Args: `--estimate <path>`, `--projectRoot` (opt — for `.alm-config.json` overrides), `--siteName` (opt), `--publisherPrefix` (opt). Output: `{ sizeAnalysis, assetAdvisory, splitStrategy, appliedStrategies, compositeSubPartitioned, proposedSolutions[], recommendations[], truncationSuspected, truncationWarnings }`. Evaluates strategies in priority order: Strategy 3 (Schema Segmentation) → Strategy 1 (Layer Split) → Strategy 2 (Change-Frequency) → Strategy 4 (Config Isolation). **Schema Segmentation is dependency-aware + capacity-bounded:** it builds connected-component clusters from `estimate.tableRelationships` (union-find), then bin-packs whole clusters (never splitting a relationship) into the fewest solutions that keep each under `maxTableCount`/`maxSchemaAttrs` **where possible** — capped at `maxSchemaSplitSolutions` (default 8). This replaced the old one-solution-per-table-name-stem heuristic that produced ~one solution per table. Two cases CAN exceed a per-solution cap, and BOTH raise an `recommendations[]` warning rather than failing silently: (a) an indivisible dependency cluster larger than `maxTableCount` stays whole (oversized-cluster table-count warning); (b) when MORE than `maxSchemaSplitSolutions` independent attr-heavy clusters must share the capped solution count, the FFD least-loaded fallback co-locates clusters and a solution's summed columns exceed `maxSchemaAttrs` (oversized-schema attr-cap warning). The split trigger + thresholds are unchanged — only the packing. Strategy 4 stacks additively. Strategy 1 also runs a composite sub-partition pass: when Core still exceeds the size OR component-count cap after Web Assets are peeled off, Core is replaced with change-frequency-shaped sub-children (`_Foundation`/`_Config`/`_Content`, plus `_Integration` whenever the parent had any flows or bots — coverage takes priority over the `changeFreqMinFlows` heuristic, which governs only the TOP-LEVEL strategy choice). When additive Strategy 4 is firing concurrently (a top-level `_EnvVars` solution), `_Config` drops `Environment Variable` from its componentTypes to avoid double-claim; when it isn't, `_Config` absorbs env vars so they have an owner. Sub-partitioning sets `compositeSubPartitioned: true` and appends `composite-sub-partition` to `appliedStrategies`. `validateSplits` checks BOTH the size AND component-count cap per split (skipping `isFutureBuffer` solutions). Supports `.alm-config.json` overrides including `strategyOverride` to bypass the tree. See `solution-splitting-logic.md` spec in design docs for full logic.
209+
- `scripts/lib/resolve-site-tables.js`: Single source of truth for "which custom tables does this site actually use." `collectReferencedEntityNames({ projectRoot, datamodelManifestPath })` reads `.powerpages-site/table-permissions/*.tablepermission.yml` (`entitylogicalname`, via `powerpages-config.js → loadTablePermissions`) + the datamodel manifest → `{ names:Set, available, sources }`. `scopeCustomTables(referencedNames, customUnmanagedTables)` intersects that set with the env's custom-unmanaged tables. SME-confirmed: table permissions are the complete signal ("if a table is used in the site there will be permissions for it"), so forms/lists are NOT scanned. Used by `estimate-solution-size.js` and `discover-site-components.js` to replace the publisher-prefix table dump.
210+
- `scripts/lib/query-metadata.js`: `queryCustomUnmanagedTables(envUrl, token, makeRequest?)``[{ logicalName, metadataId, schemaName, displayName }]` (the single `EntityDefinitions?$filter=IsCustomEntity` query, `IsManaged===false` filtered). Consolidates the formerly-triplicated custom-table query (estimator, discover-site-components, setup-solution). Reuses `odataGetAll` from `validation-helpers.js`.
211+
- `scripts/lib/query-table-relationships.js`: `fetchTableRelationships(envUrl, table, token, makeRequest?)``{ oneToMany[], manyToMany[] }`. Extracted from `skills/audit-permissions/scripts/query-table-relationships.js` (now a thin CLI wrapper over this lib) and extended with ManyToMany. OneToMany errors propagate; ManyToMany is best-effort. Used by the estimator to build `tableRelationships[]` and by audit-permissions for relationship-scope validation.
212+
- `scripts/lib/validation-helpers.js` also exports `odataGet(url, token, makeRequest?)` + `odataGetAll(url, token, makeRequest?, maxPages?)` — the shared, injectable OData GET + `@odata.nextLink` pagination used by the new metadata/relationship helpers (avoids each lib rolling its own paginator).
209213

210214
#### Solution Management
211215

plugins/power-pages/scripts/lib/alm-thresholds.js

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,12 @@ const DEFAULTS = Object.freeze({
2121
hardFlagComponentCount: 10000,
2222
maxSchemaAttrs: 15000,
2323
maxTableCount: 20,
24+
// Safety ceiling on the number of auto-derived schema-split solutions. The
25+
// schema-segmentation packing keeps each solution under maxTableCount /
26+
// maxSchemaAttrs, but caps the COUNT here so a pathological schema can't
27+
// explode into dozens of solutions — beyond this, the hardFlagComponentCount
28+
// recommendation tells the user to archive/consolidate instead.
29+
maxSchemaSplitSolutions: 8,
2430
maxAggregateWebFilesMB: 40,
2531
maxSingleFileMB: 2,
2632
maxEnvVarCount: 500,

0 commit comments

Comments
 (0)