Site-referenced table discovery + dependency-aware solution splitting

Replaces publisher-prefix table discovery (which over-counts catastrophically
with a shared/default publisher — a 6-table site matched 22 unrelated tables —
and misses real tables from a different prefix) with site-referenced scoping:
the custom tables the site's `.powerpages-site/table-permissions/` (+ datamodel
manifest) actually reference, intersected with the env's custom-unmanaged
tables. SME-confirmed: table permissions are the complete signal.

Replaces the old one-solution-per-table-name-stem split heuristic (which
produced ~one solution per table — e.g. a 21-solution split) with a
dependency-aware, capacity-bounded packer: union-find connected-component
clusters over table relationships (lookups + N:N), then first-fit-decreasing
bin-packing of whole clusters into the fewest solutions under
maxTableCount/maxSchemaAttrs, capped at maxSchemaSplitSolutions (8). The split
trigger + thresholds are unchanged — only the packing.

New shared libs:
- resolve-site-tables.js — site-referenced table scoping (single source of truth)
- query-metadata.js — consolidated custom-unmanaged-table query
- query-table-relationships.js — relationship edges (lib; audit-permissions CLI
  is now a thin wrapper)
- validation-helpers.js — odataGet/odataGetAll shared paginator

estimate-solution-size.js now emits tableCountScope + tableRelationships[];
compute-split-plan.js consumes the edges. setup-solution Phase 5.2.D uses the
shared discovery helper. 1209 tests pass, alm-lint 0 findings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: tyaginidhi

plugins/power-pages/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "power-pages",
-  "version": "2.3.0",
+  "version": "2.4.0",
   "description": "Create and deploy Power Pages sites using modern development approaches. Supports code sites (SPAs) with React, Angular, Vue, or Astro. Includes ALM orchestration (plan-alm) with a solution-splitting decision tree, per-solution pipelines, Azure Blob asset advisory, manifest schema v2 for multi-solution deployments, and force-link remediation for cross-host pipeline migrations.",
   "author": {
     "name": "Microsoft",

plugins/power-pages/AGENTS.md

@@ -204,8 +204,12 @@ Shared lib modules live at `scripts/lib/` and are imported by other scripts via
 #### Solution Splitting Decision Tree (v1.3.0+)
 
 - `scripts/lib/alm-thresholds.js`: Central default threshold constants for the split decision tree. Loads optional `.alm-config.json` from project root and merges over defaults. Exports `DEFAULTS`, `DEFAULT_CONFIG`, `loadConfig(projectRoot)`, `classifyTier(value, greenUpperExclusive, yellowUpperExclusive)`, `deepMerge(target, source)`. Used by `estimate-solution-size.js` and `compute-split-plan.js`.
-- `scripts/lib/estimate-solution-size.js`: Estimates solution size + component counts by querying Dataverse. Args: `--envUrl`, `--websiteRecordId`, `--token` (opt), `--publisherPrefix` (opt), `--siteName` (opt), `--solutionId` (opt — scopes env var count to the target solution; without it falls back to a publisher-prefix tenant-wide query that overcounts when prefix is shared), `--datamodelManifest` (opt), `--projectRoot` (opt — enables disk cross-check: walks the local build-output directory (`dist/`, `public-output/`, `build/`, `.output/`) and surfaces the byte total). Output: `{ totalSizeMB, componentCountSiteTotal, componentCountSiteActionable, componentCountInSolution, orphansOnSite, tableCount, schemaAttrCount, webFilesAggregateMB, webFilesIndividual[], webFileCount, webFileSampleSize, webFilesDiskMeasuredMB, webFilesDiskMeasuredPath, webFilesDiskFileCount, cloudFlowCount, botCount, envVarCount, envVarCountScope, envVarCountTenantWide, mediaRatio, siteType, tables[], breakdown, estimationMethod, estimationAccuracyPct, truncationSuspected, truncationWarnings, ppcGroundTruthCount }`. Metadata-based estimation with ±15% caveat. Web-file size is measured via stratified sample (first 50 + middle 50 + last 50, cap 150) scaled to full count. Disk fields are null unless `--projectRoot` was passed AND a build-output directory was found. Truncation canaries fire when Dataverse pagination disagrees with `@odata.count`, when ppcs land on a page-size boundary, when sampled average bytes/file < 1 KB at scale, or when the disk total exceeds the Dataverse total by >2× — any signal flips `truncationSuspected: true` with a per-cause `truncationWarnings[]` entry. Used by `plan-alm` Phase 1 Step 10.
-- `scripts/lib/compute-split-plan.js`: Runs the split decision tree against a size-estimate blob. Args: `--estimate <path>`, `--projectRoot` (opt — for `.alm-config.json` overrides), `--siteName` (opt), `--publisherPrefix` (opt). Output: `{ sizeAnalysis, assetAdvisory, splitStrategy, appliedStrategies, compositeSubPartitioned, proposedSolutions[], recommendations[], truncationSuspected, truncationWarnings }`. Evaluates strategies in priority order: Strategy 3 (Schema Segmentation) → Strategy 1 (Layer Split) → Strategy 2 (Change-Frequency) → Strategy 4 (Config Isolation). Strategy 4 stacks additively. Strategy 1 also runs a composite sub-partition pass: when Core still exceeds the size OR component-count cap after Web Assets are peeled off, Core is replaced with change-frequency-shaped sub-children (`_Foundation`/`_Config`/`_Content`, plus `_Integration` whenever the parent had any flows or bots — coverage takes priority over the `changeFreqMinFlows` heuristic, which governs only the TOP-LEVEL strategy choice). When additive Strategy 4 is firing concurrently (a top-level `_EnvVars` solution), `_Config` drops `Environment Variable` from its componentTypes to avoid double-claim; when it isn't, `_Config` absorbs env vars so they have an owner. Sub-partitioning sets `compositeSubPartitioned: true` and appends `composite-sub-partition` to `appliedStrategies`. `validateSplits` checks BOTH the size AND component-count cap per split (skipping `isFutureBuffer` solutions). Supports `.alm-config.json` overrides including `strategyOverride` to bypass the tree. See `solution-splitting-logic.md` spec in design docs for full logic.
+- `scripts/lib/estimate-solution-size.js`: Estimates solution size + component counts by querying Dataverse. Args: `--envUrl`, `--websiteRecordId`, `--token` (opt), `--publisherPrefix` (opt), `--siteName` (opt), `--solutionId` (opt — scopes env var count to the target solution; without it falls back to a publisher-prefix tenant-wide query that overcounts when prefix is shared), `--datamodelManifest` (opt), `--projectRoot` (opt — enables disk cross-check: walks the local build-output directory (`dist/`, `public-output/`, `build/`, `.output/`) and surfaces the byte total). Output: `{ totalSizeMB, componentCountSiteTotal, componentCountSiteActionable, componentCountInSolution, orphansOnSite, tableCount, tableCountScope, schemaAttrCount, webFilesAggregateMB, webFilesIndividual[], webFileCount, webFileSampleSize, webFilesDiskMeasuredMB, webFilesDiskMeasuredPath, webFilesDiskFileCount, cloudFlowCount, botCount, envVarCount, envVarCountScope, envVarCountTenantWide, mediaRatio, siteType, tables[], tableRelationships[], breakdown, estimationMethod, estimationAccuracyPct, truncationSuspected, truncationWarnings, ppcGroundTruthCount }`. **Table discovery is site-referenced, NOT publisher-prefix:** `tableCount`/`tables[]` are scoped to the custom tables the site actually references — its `.powerpages-site/table-permissions/` (+ datamodel manifest) intersected with the env's custom-unmanaged tables (via `resolve-site-tables.js` + `query-metadata.js`). `tableCountScope` ∈ `"site-referenced" | "manifest-only" | "unavailable"` (the last → 0 tables, never an env-wide prefix dump). `--publisherPrefix` now scopes ONLY the env var count, not tables. `tableRelationships[]` are `[a,b]` dependency edges (lookups + N:N, via `query-table-relationships.js`) among the scoped tables, consumed by `compute-split-plan.js` to cluster related tables into the same solution. Metadata-based estimation with ±15% caveat. Web-file size is measured via stratified sample (first 50 + middle 50 + last 50, cap 150) scaled to full count. Disk fields are null unless `--projectRoot` was passed AND a build-output directory was found. Truncation canaries fire when Dataverse pagination disagrees with `@odata.count`, when ppcs land on a page-size boundary, when sampled average bytes/file < 1 KB at scale, or when the disk total exceeds the Dataverse total by >2× — any signal flips `truncationSuspected: true` with a per-cause `truncationWarnings[]` entry. Used by `plan-alm` Phase 1 Step 10.
+- `scripts/lib/compute-split-plan.js`: Runs the split decision tree against a size-estimate blob. Args: `--estimate <path>`, `--projectRoot` (opt — for `.alm-config.json` overrides), `--siteName` (opt), `--publisherPrefix` (opt). Output: `{ sizeAnalysis, assetAdvisory, splitStrategy, appliedStrategies, compositeSubPartitioned, proposedSolutions[], recommendations[], truncationSuspected, truncationWarnings }`. Evaluates strategies in priority order: Strategy 3 (Schema Segmentation) → Strategy 1 (Layer Split) → Strategy 2 (Change-Frequency) → Strategy 4 (Config Isolation). **Schema Segmentation is dependency-aware + capacity-bounded:** it builds connected-component clusters from `estimate.tableRelationships` (union-find), then bin-packs whole clusters (never splitting a relationship) into the fewest solutions that keep each under `maxTableCount`/`maxSchemaAttrs`, capped at `maxSchemaSplitSolutions` (default 8). This replaced the old one-solution-per-table-name-stem heuristic that produced ~one solution per table. An indivisible cluster over the cap stays whole and raises an oversized-cluster `recommendations[]` warning. The split trigger + thresholds are unchanged — only the packing. Strategy 4 stacks additively. Strategy 1 also runs a composite sub-partition pass: when Core still exceeds the size OR component-count cap after Web Assets are peeled off, Core is replaced with change-frequency-shaped sub-children (`_Foundation`/`_Config`/`_Content`, plus `_Integration` whenever the parent had any flows or bots — coverage takes priority over the `changeFreqMinFlows` heuristic, which governs only the TOP-LEVEL strategy choice). When additive Strategy 4 is firing concurrently (a top-level `_EnvVars` solution), `_Config` drops `Environment Variable` from its componentTypes to avoid double-claim; when it isn't, `_Config` absorbs env vars so they have an owner. Sub-partitioning sets `compositeSubPartitioned: true` and appends `composite-sub-partition` to `appliedStrategies`. `validateSplits` checks BOTH the size AND component-count cap per split (skipping `isFutureBuffer` solutions). Supports `.alm-config.json` overrides including `strategyOverride` to bypass the tree. See `solution-splitting-logic.md` spec in design docs for full logic.
+- `scripts/lib/resolve-site-tables.js`: Single source of truth for "which custom tables does this site actually use." `collectReferencedEntityNames({ projectRoot, datamodelManifestPath })` reads `.powerpages-site/table-permissions/*.tablepermission.yml` (`entitylogicalname`, via `powerpages-config.js → loadTablePermissions`) + the datamodel manifest → `{ names:Set, available, sources }`. `scopeCustomTables(referencedNames, customUnmanagedTables)` intersects that set with the env's custom-unmanaged tables. SME-confirmed: table permissions are the complete signal ("if a table is used in the site there will be permissions for it"), so forms/lists are NOT scanned. Used by `estimate-solution-size.js` and `discover-site-components.js` to replace the publisher-prefix table dump.
+- `scripts/lib/query-metadata.js`: `queryCustomUnmanagedTables(envUrl, token, makeRequest?)` → `[{ logicalName, metadataId, schemaName, displayName }]` (the single `EntityDefinitions?$filter=IsCustomEntity` query, `IsManaged===false` filtered). Consolidates the formerly-triplicated custom-table query (estimator, discover-site-components, setup-solution). Reuses `odataGetAll` from `validation-helpers.js`.
+- `scripts/lib/query-table-relationships.js`: `fetchTableRelationships(envUrl, table, token, makeRequest?)` → `{ oneToMany[], manyToMany[] }`. Extracted from `skills/audit-permissions/scripts/query-table-relationships.js` (now a thin CLI wrapper over this lib) and extended with ManyToMany. OneToMany errors propagate; ManyToMany is best-effort. Used by the estimator to build `tableRelationships[]` and by audit-permissions for relationship-scope validation.
+- `scripts/lib/validation-helpers.js` also exports `odataGet(url, token, makeRequest?)` + `odataGetAll(url, token, makeRequest?, maxPages?)` — the shared, injectable OData GET + `@odata.nextLink` pagination used by the new metadata/relationship helpers (avoids each lib rolling its own paginator).
 
 #### Solution Management
 

plugins/power-pages/scripts/lib/alm-thresholds.js

@@ -21,6 +21,12 @@ const DEFAULTS = Object.freeze({
   hardFlagComponentCount: 10000,
   maxSchemaAttrs: 15000,
   maxTableCount: 20,
+  // Safety ceiling on the number of auto-derived schema-split solutions. The
+  // schema-segmentation packing keeps each solution under maxTableCount /
+  // maxSchemaAttrs, but caps the COUNT here so a pathological schema can't
+  // explode into dozens of solutions — beyond this, the hardFlagComponentCount
+  // recommendation tells the user to archive/consolidate instead.
+  maxSchemaSplitSolutions: 8,
   maxAggregateWebFilesMB: 40,
   maxSingleFileMB: 2,
   maxEnvVarCount: 500,

plugins/power-pages/scripts/lib/compute-split-plan.js

@@ -299,7 +299,7 @@ function partitionByChangeFrequency(estimate, meta) {
 function partitionBySchema(estimate, meta, config) {
   const explicitDomains = Array.isArray(config.domains) && config.domains.length > 0
     ? config.domains
-    : deriveDomainsFromPrefix(estimate);
+    : deriveDomainsByCapacity(estimate, config.thresholds);
 
   // Derive domain vs site size shares from the estimator's breakdown when available,
   // falling back to a 50/50 heuristic only if breakdown is absent.
@@ -319,9 +319,12 @@ function partitionBySchema(estimate, meta, config) {
     componentTypes: ['Table'],
     description: `Schema domain: ${dom.name}. Tables: ${(dom.tableLogicalNames || []).join(', ') || '(derived)'}${domainDescSuffix}`,
     sizeMB: round(sizePerDomain),
-    componentCount: Math.ceil(
-      (estimate.schemaAttrCount || 0) / domainCount,
-    ),
+    // A Table domain's component count IS its table count when known (each table
+    // is one Entity solution component). Falls back to an even attr-share split
+    // only for explicit domains that didn't list their tables.
+    componentCount: (dom.tableLogicalNames && dom.tableLogicalNames.length > 0)
+      ? dom.tableLogicalNames.length
+      : Math.ceil((estimate.schemaAttrCount || 0) / domainCount),
     components: [],
     tableLogicalNames: dom.tableLogicalNames || [],
   }));
@@ -345,23 +348,99 @@ function partitionBySchema(estimate, meta, config) {
   return [...domainSolutions, siteSolution];
 }
 
-function deriveDomainsFromPrefix(estimate) {
-  const tables = estimate.tables || [];
-  if (tables.length === 0) return [{ name: 'All', tableLogicalNames: [] }];
+// --- Dependency-aware schema packing ---------------------------------------
+//
+// Replaces the old "one solution per table-name stem" heuristic (which produced
+// ~one solution per table for any distinctly-named schema). Tables connected by
+// a relationship MUST ship together, so we:
+//   1. Group tables into connected components (union-find over the estimator's
+//      `tableRelationships` edges). Because components have no edges between
+//      them, packing whole components into separate solutions never cuts a
+//      relationship — so there are no cross-/circular-solution table deps and
+//      import order among the table solutions is irrelevant.
+//   2. Bin-pack the components into the FEWEST solutions that keep each under the
+//      per-solution caps (maxTableCount tables AND maxSchemaAttrs columns),
+//      capped at maxSchemaSplitSolutions.
+
+function normalizeTables(estimate) {
+  return (estimate.tables || [])
+    .map((t) => ({
+      logicalName: (t && (t.logicalName || t)).toString(),
+      attributeCount: (t && t.attributeCount) || 0,
+    }))
+    .filter((t) => t.logicalName);
+}
 
+// Union-find over tables + relationship edges -> array of clusters (each a list
+// of table objects). A table with no edges is its own singleton cluster.
+function buildTableClusters(tables, edges) {
+  const idx = new Map();
+  tables.forEach((t, i) => idx.set(t.logicalName.toLowerCase(), i));
+  const parent = tables.map((_, i) => i);
+  const find = (x) => { while (parent[x] !== x) { parent[x] = parent[parent[x]]; x = parent[x]; } return x; };
+  const union = (a, b) => { const ra = find(a), rb = find(b); if (ra !== rb) parent[ra] = rb; };
+  for (const e of edges || []) {
+    if (!Array.isArray(e) || e.length < 2) continue;
+    const ia = idx.get(String(e[0]).toLowerCase());
+    const ib = idx.get(String(e[1]).toLowerCase());
+    if (ia != null && ib != null) union(ia, ib);
+  }
   const groups = new Map();
-  for (const t of tables) {
-    const name = (t.logicalName || t).toString();
-    const afterPrefix = name.includes('_') ? name.split('_').slice(1).join('_') : name;
-    const stem = afterPrefix.split(/[_]/)[0] || 'misc';
-    const key = stem.charAt(0).toUpperCase() + stem.slice(1);
-    if (!groups.has(key)) groups.set(key, []);
-    groups.get(key).push(name);
+  tables.forEach((t, i) => {
+    const r = find(i);
+    if (!groups.has(r)) groups.set(r, []);
+    groups.get(r).push(t);
+  });
+  return [...groups.values()];
+}
+
+function clusterAttrs(cluster) {
+  return cluster.reduce((s, t) => s + (t.attributeCount || 0), 0);
+}
+
+// First-fit-decreasing pack of whole clusters into `n` buckets, respecting the
+// per-solution table + attribute caps. A cluster that fits nowhere under the
+// caps (oversized, or n too small) goes to the least-loaded bucket — that bucket
+// then exceeds a cap and is surfaced by the oversized-cluster recommendation.
+function packClusters(clusters, n, thresholds) {
+  const sorted = [...clusters].sort((a, b) => (clusterAttrs(b) - clusterAttrs(a)) || (b.length - a.length));
+  const buckets = Array.from({ length: Math.max(n, 1) }, () => ({ tables: [], attrs: 0 }));
+  for (const cluster of sorted) {
+    const cAttrs = clusterAttrs(cluster);
+    let target = buckets.findIndex(
+      (b) => b.tables.length + cluster.length <= thresholds.maxTableCount &&
+             b.attrs + cAttrs <= thresholds.maxSchemaAttrs,
+    );
+    if (target === -1) {
+      target = buckets.reduce((best, b, i) => (b.attrs < buckets[best].attrs ? i : best), 0);
+    }
+    buckets[target].tables.push(...cluster);
+    buckets[target].attrs += cAttrs;
   }
+  return buckets.filter((b) => b.tables.length > 0);
+}
 
-  return Array.from(groups.entries()).map(([name, tableLogicalNames]) => ({
-    name,
-    tableLogicalNames,
+// Returns capacity-bounded "domains" (one per packed bucket) in the same shape
+// the schema partitioner consumes: { name, tableLogicalNames }.
+function deriveDomainsByCapacity(estimate, thresholds) {
+  const tables = normalizeTables(estimate);
+  if (tables.length === 0) return [{ name: 'Tables', tableLogicalNames: [] }];
+
+  const clusters = buildTableClusters(tables, estimate.tableRelationships || []);
+  const totalAttrs = tables.reduce((s, t) => s + t.attributeCount, 0);
+  const ceiling = (thresholds && thresholds.maxSchemaSplitSolutions) || 8;
+  let n = Math.max(
+    1,
+    Math.ceil(tables.length / thresholds.maxTableCount),
+    Math.ceil(totalAttrs / Math.max(thresholds.maxSchemaAttrs, 1)),
+  );
+  n = Math.min(n, ceiling, clusters.length);
+
+  const buckets = packClusters(clusters, n, thresholds);
+  const multi = buckets.length > 1;
+  return buckets.map((b, i) => ({
+    name: multi ? `Tables ${i + 1}` : 'Tables',
+    tableLogicalNames: b.tables.map((t) => t.logicalName),
   }));
 }
 
@@ -704,6 +783,17 @@ function computeSplitPlan({ estimate, config, meta }) {
   proposedSolutions = appendFutureBuffer(proposedSolutions, meta);
 
   const splitWarnings = validateSplits(proposedSolutions, config.thresholds);
+  // Oversized-cluster guard: a Table solution holding more tables than the
+  // per-solution cap means a single connected dependency cluster couldn't be
+  // split without cutting a relationship. Name it so the user can decide whether
+  // to denormalize the schema or raise the cap — we never silently split a cluster.
+  const oversizedClusterWarnings = proposedSolutions
+    .filter((s) => Array.isArray(s.tableLogicalNames) &&
+      s.tableLogicalNames.length > config.thresholds.maxTableCount)
+    .map((s) => ({
+      type: 'warning',
+      message: `Solution ${s.uniqueName} holds ${s.tableLogicalNames.length} related tables — above the ${config.thresholds.maxTableCount}-per-solution cap — because they form one dependency cluster that cannot be split without breaking a relationship. Consider denormalizing the schema or raising maxTableCount in .alm-config.json.`,
+    }));
   // Surface estimator-side truncation warnings as `recommendations[]` entries
   // so the rendered plan shows them inline. These get the `error` type because
   // a truncated input is more dangerous than a normal split-decision warning
@@ -715,7 +805,8 @@ function computeSplitPlan({ estimate, config, meta }) {
     }));
   const recommendations = truncationRecs
     .concat(buildRecommendations(estimate, strategy, config))
-    .concat(splitWarnings);
+    .concat(splitWarnings)
+    .concat(oversizedClusterWarnings);
 
   const appliedStrategies = [strategy.primary];
   if (strategy.additive) appliedStrategies.push('strategy-4-config-isolation');