Feat/oleksii bielov snow feat

.cortex/skills/backload-matching/SKILL.md

@@ -56,15 +56,13 @@ See `references/use-case-narrative.md` for the full story. Summary anchored in t
 | SCHEMA | `BACKLOAD_MATCHING` | Schema for backload tables and views |
 | WAREHOUSE | `ROUTING_ANALYTICS` | Warehouse for queries |
 | REGION | (active preset) | Auto-derived from `BACKLOAD_MATCHING.CONFIG`, which mirrors the active Control App region/vehicle. No hardcoded city. |
-| HOME_REGION | `Nordics` | Country group counted as "back-to-home" |
-| HOME_LAT / HOME_LON | `55.6759 / 12.5655` | Anchor (Copenhagen) used as vehicle `end` |
-| TRAILER_COUNT | `80` | Idle-bound trailers seeded |
-| INTERNAL_VOLUMES_COUNT | `120` | Internal waiting loads seeded |
-| EXTERNAL_OFFERS_COUNT | `300` | Synthetic external offers seeded |
+| VEHICLE_CLASS_PROFILE | `OPENROUTESERVICE_APP.CORE.VEHICLE_CLASS_PROFILE` | Single source of truth for per-vehicle-class capacity (`PAYLOAD_KG_TYP`), shipment-weight band, ORS profile, costs (€/km, €/hr), and UI label. The skill is transport-type agnostic — no HGV-specific constants. Seeded with 8 classes (`bicycle`, `ebike`, `foot`, `motorcycle`, `car`, `van`, `hgv`, `truck`). Unknown `vehicle_type` → bootstrap and React both fail loudly so a custom preset never silently runs with wrong-class defaults. |
+| TRAILER_COUNT | up to ~80 (driven by Data Studio dataset) | Idle-bound trailers for the active preset |
+| INTERNAL_VOLUMES_COUNT | 120 | Internal waiting loads (most-recent FACT_TRIPS) |
+| EXTERNAL_OFFERS_COUNT | 300 | Synthetic external offers per region |
 | INTERNAL_PRIORITY | `100` | VROOM `priority` on internal jobs |
 | EXTERNAL_PRIORITY | `10` | VROOM `priority` on external offers |
 | TIME_WINDOW_TOLERANCE_HRS | `4` | Pickup-window slack added to jobs |
-| MAX_EMPTY_KM_PER_LEG | `200` | Hard skip in candidate pre-filter |
 | MAX_VEHICLES_PER_SOLVE | `30` | Solver caps vehicles per call to keep ORS responsive |
 | EUR_PER_EMPTY_KM | `1.20` | Used for KPI ("EUR/day reclaimed") |
 | IDLE_COST_EUR_PER_DAY | `650` | Used for KPI ("EUR/day reclaimed") |

.cortex/skills/backload-matching/references/optimization-vrp-mapping.md

@@ -1,6 +1,6 @@
 # Backload Matching Engine — OPTIMIZATION VRP mapping
 
-This document shows the exact JSON the page sends to `OPENROUTESERVICE_APP.CORE.OPTIMIZATION(...)` and how each field maps to the DHL Freight backload story.
+This document shows the exact JSON the page sends to `OPENROUTESERVICE_APP.CORE.OPTIMIZATION(...)` and how each visible UI lever maps 1:1 to a real VROOM or ORS field. The new design has **no JS pre-filters or post-filters** — every knob you turn lands inside the solver.
 
 ## Vehicles (one per idle-bound trailer)
 
@@ -9,82 +9,124 @@ This document shows the exact JSON the page sends to `OPENROUTESERVICE_APP.CORE.
   "id": 1,
   "profile": "driving-hgv",
   "start": [6.9603, 50.9375],
-  "end":   [12.5655, 55.6759],
-  "capacity": [24000],
+  "end":   [12.5655, 55.6759],            // omit when "Open-ended"
+  "capacity": [24000, 33, 90],            // [kg, pallets, m³] when multi-dim
   "skills": [1, 2],
-  "time_window": [1715874000, 1715917200]
+  "max_tasks": 2,                         // Max stops per trailer
+  "max_travel_time": 28800,               // ideal_empty_hrs + slack
+  "max_distance": 600000,                 // ideal_empty_km × (1 + dev%)
+  "costs": { "fixed": 12000, "per_km": 160 },
+  "time_window": [1715874000, 1715906400],         // optional shift
+  "breaks": [{ "id": 1, "service": 2700,
+               "time_windows": [[1715890200, 1715895600]] }],
+  "profile_options": { "avoid_polygons": { "type": "MultiPolygon", ... } }
 }
 ```
 
-| Field | Source | Why |
+| Field | UI lever | Why |
 |---|---|---|
 | `start` | `VW_TRAILERS.DROPOFF_LON / DROPOFF_LAT` | Where the trailer becomes idle |
-| `end` | `VW_TRAILERS.HOME_LON / HOME_LAT` | Forces the solver to bias jobs that point home |
-| `profile` | `'driving-hgv'` | HGV graph — respects truck restrictions |
-| `capacity` | `[VW_TRAILERS.MAX_PAYLOAD_KG]` | Single dimension (mass); add volume / pallets as needed |
-| `skills` | `[1, 2]` if non-ADR; `[1, 2, 3]` if `HAZMAT_CERT = TRUE` | Skill `1` = internal, `2` = external, `3` = ADR-only |
-| `time_window` | `[ETA_TS_unix, ETA_TS + 12h_unix]` | Trailer becomes available at ETA, expires after 12h shift |
-
-## Jobs (internal volumes + external offers, unioned)
+| `end` | radio: Home / Shared / Open | When `Open`, omit the field — solver lets the trailer finish anywhere |
+| `profile` | derived from region | ORS routing graph |
+| `capacity` | toggle: multi-dim capacity | `[kg]` or `[kg, pallets, m³]` |
+| `skills` | HAZMAT cert flag | `[1,2,3]` if certified, `[1,2]` otherwise |
+| `max_tasks` | **Max stops per trailer** | 1 = single backload, 2-6 = consolidation |
+| `max_travel_time` | **Detour budget** (+h on top of ideal empty trip) | Hard time cap on the whole tour |
+| `max_distance` | **Allowed deviation** (+% from ideal empty trip) | Hard distance cap on the whole tour |
+| `costs.fixed` | **Fixed dispatch cost** (€) × 100 | Pay-to-add-vehicle penalty |
+| `costs.per_km` | (**€/km** + **€/h** ÷ 60 km/h) × 100 | Time + distance weighted into VROOM's per-km cost. Note: the deployed regional VROOM image is `vroom-docker:v1.0.4`, which does NOT yet support `costs.per_hour` (added upstream in VROOM v1.13). When the gateway upgrades to v1.13+, switch back to `per_hour` for cleaner semantics. |
+| `time_window` | toggle: shift / hours-of-service | `[shiftStart, shiftEnd]` |
+| `breaks` | toggle: enforce driver break | EU 45-min rest after 4.5 h |
+| `profile_options.avoid_polygons` | multi-select: avoid zones | Forwarded to ORS routing for LEZ / construction / hazard avoidance |
+
+## Shipments (internal volumes + external offers)
 
 ```json
 {
-  "id": 41,
-  "location": [6.9512, 50.9301],
-  "service": 1800,
-  "amount": [12500],
-  "skills": [1],
-  "priority": 100,
-  "time_windows": [[1715874000, 1715901000]]
+  "pickup":   { "id": 41, "location": [...], "service": 1800,
+                "time_windows": [[t1_from, t1_to], [t2_from, t2_to]] },
+  "delivery": { "id": 41, "location": [...], "service": 600 },
+  "amount":   [12500, 17, 50],
+  "skills":   [1],
+  "priority": 90
 }
 ```
 
-| Field | Source | Why |
+| Field | UI lever | Why |
 |---|---|---|
-| `location` | `INTERNAL_VOLUMES.PICKUP_GEOM` or `EXTERNAL_OFFERS.PICKUP_GEOM` | Pickup point |
-| `amount` | `WEIGHT_KG` | Must fit in vehicle capacity |
-| `skills` | `[1]` for INTERNAL, `[2]` for EXTERNAL, plus `3` if ADR | Gates assignability |
-| `priority` | `INTERNAL_PRIORITY=100` for INTERNAL, `EXTERNAL_PRIORITY=10` for EXTERNAL | VROOM unloads jobs in priority order under contention |
-| `service` | 1800 sec (30 min) | Loading time at pickup |
-| `time_windows` | `[PICKUP_FROM_TS, PICKUP_TO_TS]` (unix) | Hard window |
-
-## The full `OPTIMIZATION(...)` call
-
-The page builds:
-
-```javascript
-const challenge = {
-  jobs:     [...internalJobs, ...externalJobs],
-  vehicles: trailers.map(toVehicle),
-  options:  { g: true } // return geometry for each route
-};
-
-const sql = `SELECT * FROM TABLE(OPENROUTESERVICE_APP.CORE.OPTIMIZATION(
-  PARSE_JSON('${JSON.stringify(challenge)}'), '${regionName}'))`;
-```
+| `pickup.location` / `delivery.location` | source data | Pickup / dropoff |
+| `service` | hard-coded (1800 / 600 sec) | Loading / unloading time |
+| `amount` | toggle: multi-dim | `[kg]` or `[kg, pallets, m³]` |
+| `skills` | HAZMAT flag on shipment | `[1]` internal, `[2]` external, `+3` if ADR |
+| `priority` | **Internal-first weight** (single slider) | Internal = `W`, External = `100 - W` |
+| `time_windows` | **Window slack** (±h) and toggle: multi-window | Widened ±slack; second window synthesised at +8 h when toggled |
 
-VROOM returns one row per vehicle with:
+## Top-level options
+- `options.g = true` — return GeoJSON geometry per route. Always on.
 
-- `VEHICLE` — the vehicle id we sent.
-- `STEPS` — ordered jobs assigned (start, job_1, job_2, ..., end).
-- `COST` — total travel time in seconds for that vehicle.
-- `GEOJSON` — the LineString for the whole route (when `options.g = true`).
-- `UNASSIGNED` (top-level row) — jobs the solver could not place (capacity, time-window, skill, or `max_empty_km` exceeded).
+## Post-solve economics (computed in the React layer, not VROOM)
+
+| Display column | Formula |
+|---|---|
+| Tour km | `r.DISTANCE / 1000` (or fallback from duration × speed) |
+| Tour hrs | `r.DURATION / 3600` |
+| Cost (€) | `dispatch + tourHrs × €/h + tourKm × €/km + nDeliveries × €/delivery` |
+| Revenue (€) | external: `PRICE_EUR`. Internal: `loaded_km × €/loaded-km` |
+| Net benefit (€) | `Revenue − Cost` (red badge when negative) |
+| Wait time | per-step `waiting_time` from VROOM (chip when > 30 min) |
 
-## How DHL's "structural process" maps to VROOM parameters
+## What got removed from the old UI
 
-| Customer parameter | VROOM lever |
+| Old control | Why it's gone |
 |---|---|
-| Internal-first | Job `priority`: 100 vs 10 |
-| Direction-to-home | Vehicle `end` location set to home depot |
-| ADR equipment gating | Skill id `3` on both ADR jobs and ADR-certified vehicles |
-| Time-window tolerance | Widen `time_windows` on jobs by `tolerance_hrs` |
-| Max empty km per leg | Pre-filter the `jobs[]` array in SQL: `WHERE empty_km <= MAX_EMPTY_KM` |
-| Trailer capacity | Vehicle `capacity[0]` |
-| Cargo weight | Job `amount[0]` |
+| `Max empty km/leg` | Was a JS post-filter that silently dropped solver decisions. Empty km is now minimised directly by `costs.per_hour` + `max_travel_time`. |
+| `Max detour km` | Was a JS pre-filter that hid candidates from VROOM. Redundant with `max_distance` (deviation %). |
+| Two priority sliders (internal + external) | Collapsed into one **Internal-first weight** slider. |
+| Match mode radio (`single` / `consolidate`) | Replaced by **Max stops** number (1 = single, ≥2 = consolidation). |
+| `Force shared destination` checkbox | Replaced by the 3-way **Trailer end** radio (Home / Shared / Open). |
 
 ## Productisation notes (out of scope for the demo)
 
 - **Real-time refresh**: replace the polled `VW_TRAILERS` with Snowpipe Streaming on a `TELEMETRY_RAW` topic; rebuild the view as a Dynamic Table with a 5-minute target lag.
-- **Live freight-exchange feeds**: the four portals (Timocom, WTransnet, Teleroute, B2P) all publish offers via either REST APIs (Timocom Smart Logistics System REST), webhook subscriptions, or paid CSV pulls. A small Snowpark Container Services worker can normalize them into `EXTERNAL_OFFERS` with the same schema.
-- **Solver scale**: VROOM solves 200+ vehicles x 1000+ jobs in a few seconds. Beyond that, partition by region (NRW, Bavaria, Île-de-France) and solve in parallel — the customer's existing dispatcher mental model already partitions this way.
+- **Live freight-exchange feeds**: the four EU portals (Timocom, WTransnet, Teleroute, B2P) all publish offers via REST APIs or webhook subscriptions. A small Snowpark Container Services worker can normalize them into `EXTERNAL_OFFERS` with the same schema.
+- **Solver scale**: VROOM solves 200+ vehicles × 1000+ jobs in a few seconds. Beyond that, partition by region (NRW, Bavaria, Île-de-France) and solve in parallel.
+- **Pre-computed matrices**: for repeated what-if solves on the same point set, call `OPENROUTESERVICE_APP.CORE._OPTIMIZATION_TABULAR_RAW(jobs, vehicles, matrices, region)` with a cached matrix. (Cache table not seeded in v1.1.35; follow-up work.)
+- **Pinned stops** (`vehicle.steps[]`): dispatcher overrides where a specific stop is locked to a specific trailer. (UI hook not shipped in v1.1.35; follow-up work.)
+- **Mixed jobs + shipments**: VROOM accepts both `jobs[]` (single-location) and `shipments[]` (paired). Useful for return / empty-container relocation tasks. (Follow-up.)
+
+## Sanity probe: "OPTIMIZATION returned 0 rows"
+
+If the UI surfaces "OPTIMIZATION returned 0 rows", first verify VROOM itself is
+healthy for the region with this minimal probe (replace `$REGION` and the four
+coordinates with values that lie inside the region's boundary). One returned
+row means VROOM works — the issue is then in the payload your code is
+generating (almost always `max_travel_time` or `max_distance` too tight, or an
+unknown VROOM v1.0.4 field; see compatibility table above).
+
+```sql
+SET REGION = 'Germany';
+SELECT VEHICLE FROM TABLE(OPENROUTESERVICE_APP.CORE.OPTIMIZATION(
+  PARSE_JSON('{
+    "vehicles":[{"id":1,"profile":"driving-hgv",
+      "start":[13.4050,52.5200],"end":[11.5820,48.1351],
+      "capacity":[24000],"max_tasks":3}],
+    "shipments":[{"pickup":{"id":1001,"location":[8.6821,50.1109],"service":1800},
+      "delivery":{"id":1001,"location":[9.9937,53.5511],"service":600},
+      "amount":[10000]}]
+  }'), $REGION));
+```
+
+### Common silent-rejection causes (VROOM v1.0.4)
+
+VROOM v1.0.4 silently drops the entire response (0 rows, no error) when:
+
+- `max_travel_time` is smaller than the shortest feasible tour. The React UI
+  computes this from the haversine envelope of the candidate pool times
+  `(2 * maxStops + 1)` legs plus the user's `detourSlackHrs` slider — a fixed
+  4 h baseline like the legacy formula collapses on continental graphs.
+- `max_distance` is smaller than the shortest feasible tour distance.
+- `costs.per_hour` is set (added in upstream VROOM v1.13). Fold into
+  `costs.per_km` via assumed speed (HGV ≈ 60 km/h).
+- Multi-dim `capacity` does not match `amount` length on a shipment.
+- `vehicle.profile_options.avoid_polygons` is set but the routing gateway
+  does not yet forward it to the ORS matrix call.

.cortex/skills/build-routing-solution/SKILL.md

@@ -55,6 +55,33 @@ Deploys the OpenRouteService route optimization application using Snowpark Conta
 | IMAGE_REPO | `ORS_REPOSITORY` | Image repository for SPCS containers |
 | COMPUTE_POOL | `ORS_COMPUTE_POOL` | Compute pool for ORS services |
 
+## Sizing
+
+Pre-check before provisioning a new region. The preflight procedure `OPENROUTESERVICE_APP.CORE.VALIDATE_REGION_PREFLIGHT(...)` (added in #52) returns a JSON verdict including the recommended compute size and instance family for any (bounding-box, profiles, compute_size) combination. Call it from the control-app's Region Builder before kicking off `PROVISION_REGION_WRAPPER`, or directly from SQL:
+
+```sql
+CALL OPENROUTESERVICE_APP.CORE.VALIDATE_REGION_PREFLIGHT(
+    36.0, 42.0, -125.0, -114.0,    -- California-sized bbox
+    'driving-car,driving-hgv',
+    'L'
+);
+-- Returns {"ok":true, "estimated_pbf_gib":2.83, "estimated_graph_gib":8.49,
+--          "recommended_compute_size":"XXL", "recommended_instance_family":"HIGHMEM_X64_L",
+--          "warnings":["Estimated graph size 8.49 GiB exceeds the headroom of compute size L. ..."]}
+```
+
+Reference table (typical Geofabrik extracts; one driving profile):
+
+| Region scale | bbox area (km²) | PBF (GiB) | Graph (GiB) | Compute size | Instance family | JVM `-Xmx` | `graphs_data_access` |
+|---|---|---|---|---|---|---|---|
+| Single city (Berlin, SF, Munich) | < 2 500 | < 0.1 | < 0.5 | `S` | HIGHMEM_X64_S | 20 GiB | `RAM_STORE` |
+| State / small country (California, Bavaria) | 2 500 – 50 000 | 0.5 – 3 | 1 – 6 | `L` | HIGHMEM_X64_M | 700 GiB | `RAM_STORE` |
+| Country / region (USA, EU) | > 50 000 | 3 – 15 | 6 – 20+ | `XXL` | HIGHMEM_X64_L | 1100 GiB | `MMAP` |
+
+Each additional enabled profile multiplies the graph size by ~1.5×. Disable unused profiles in the active `ors-config.yml` (or pick a narrower preset from `.cortex/skills/routing-customization/references/ors-config-presets/`) to keep build time bounded.
+
+**MMAP guidance for continental extracts.** Set `ors.engine.graphs_data_access: MMAP` in `ors-config.yml` whenever the estimated graph exceeds ~5 GiB. This trades a small per-query latency increase for the ability to load graphs that do not fit in RAM, and is mandatory for any extract that hits the 3-day silent hang documented at <https://ask.openrouteservice.org/t/ors-stuck-when-creating-graph-with-european-borders-enabled/>. The shipped [`continental.yml`](../routing-customization/references/ors-config-presets/continental.yml) preset enables this automatically.
+
 ## Workflow
 
 > **Fresh install assumed.** This workflow targets a clean Snowflake account with no pre-existing ORS objects. All DDL uses `CREATE ... IF NOT EXISTS` or `CREATE OR REPLACE` with complete schemas from the start. All columns (JOB_ID, GEOGRAPHY, etc.) are defined in the initial CREATE TABLE statements -- no ALTER TABLE migration steps are needed.
@@ -252,9 +279,19 @@ Follow the full build instructions in `references/build-images.md`. Summary:
    MODULES_DIR=".cortex/skills/build-routing-solution/openrouteservice_app/app/modules"
 
    for m in 01_core_infra.sql 02_routing_functions.sql 03_region_management.sql \
-            04_service_lifecycle.sql 05_matrix_pipeline.sql 06_matrix_ops.sql; do
+            04_service_lifecycle.sql 05_matrix_pipeline.sql 06_matrix_ops.sql \
+            07_studio_jobs.sql 08_observability.sql 15_route_optimization_seed.sql; do
      bash "$RUN_SQL" <connection> "$MODULES_DIR/$m" || exit 1
    done
+
+   # Resume the observability ingest + retention tasks created (suspended) by
+   # module 08. Without this the Observability page in the control app stays
+   # empty even though the schema exists. Safe to run repeatedly — ALTER TASK
+   # RESUME is idempotent.
+   snow sql -c <connection> -q "
+     ALTER TASK OPENROUTESERVICE_APP.OBSERVABILITY.ORS_METRICS_INGEST_TASK RESUME;
+     ALTER TASK OPENROUTESERVICE_APP.OBSERVABILITY.ORS_REQUEST_LOG_PURGE_TASK RESUME;
+   "
    ```
 
    > **Recovery if 01_core_infra.sql fails partway:** Fix the underlying issue (e.g., grant missing privileges), then re-run the full file. All DDL uses `IF NOT EXISTS` or `CREATE OR REPLACE`, making re-runs safe and idempotent. Alternatively, create only the missing service(s) individually using the corresponding `CREATE SERVICE` statement from the SQL file.