-
Notifications
You must be signed in to change notification settings - Fork 159
feat: model size status page #8604
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
The OLAPListTables API returns paginated results with a nextPageToken field. Previously only the first page was fetched, causing materialized models and tables on subsequent pages to show as having no size data. This change replaces the derived store pattern with a readable store that: - Fetches all pages sequentially for each connector - Detects nextPageToken in responses and creates follow-up queries - Accumulates all tables before building the final size map - Handles loading/error states correctly This fixes the issue where 5 models were detected but only 2 tables (from the first page) were returned, causing models like "auction_data_model" to show "-" instead of their actual size. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <[email protected]>
The issue was that useModelTableSizes was creating a new store on every render when resources changed. If the resources loaded before the queries completed, the component would be unmounted/remounted, cancelling the queries. This change: 1. Caches stores by instanceId:connectorArray to prevent recreation 2. Eagerly subscribes to each store to keep queries alive 3. Uses WeakMap to prevent memory leaks from old subscriptions 4. Ensures queries complete even if component re-renders quickly Now when you refresh the status page, queries stay alive in the background and complete asynchronously, updating the table when ready. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <[email protected]>
|
a first pass at getting something in while project status 2.0 gets started as well as https://linear.app/rilldata/issue/PLAT-376/instrumentation-for-physical-table-size-of-models |
The store cache was keeping stale subscriptions alive across page refreshes. When you refreshed the page, the old cached store would be returned even if the resources had changed or needed fresh data. This change simplifies the approach: - Create fresh stores on each useModelTableSizes call - Let TanStack Query handle result caching (HTTP level) - Ensure queries always run with latest connector data - No issues with premature garbage collection Now on page refresh, sizes will load correctly immediately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <[email protected]>
The issue was that columns were defined as a static const, so the accessor function captured the initial tableSizes reference and never updated when tableSizes changed. Changes: 1. Made columns reactive with $: so they update when tableSizes changes 2. Added console logging to ProjectResources and ProjectResourcesTable to track when stores and columns are updated 3. Added logging to selectors to see preload and store update timing This should fix the issue where the table shows "-" on initial load but correctly shows sizes after navigating away and back. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <[email protected]>
The core issue: columns were defined as a static const, capturing the initial tableSizes reference. When the store updated with new data, the column accessor functions still referenced the old empty Map. Solution: Make columns reactive with $: so they recreate whenever tableSizes changes, ensuring accessor functions get the current data. This allows the UI to update asynchronously as size data arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <[email protected]>
|
fixed bug, where values not shown unless action done on cloud, now it updates automatically: https://www.loom.com/share/825b70c51d234177a8949d7c920e4894 |
ericpgreen2
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for taking this on! I've been thinking about the design for this feature and want to propose a different direction before we proceed.
The current approach adds a "Size" column to the existing Resources table. While this works, I think it mixes two conceptually different concerns:
- Resources table = "Is my deployment healthy?" (reconciliation status, refresh times, errors)
- Table metrics = "How much data do I have?" (size, row count, column count)
This is why most rows show "-" for size — only Models map to database tables. Explores, Metrics Views, Canvas, etc. don't have a "size" concept.
Proposed Alternative: Separate "Tables" Section
I'd recommend mirroring the CLI's approach, where rill project status and rill project tables are distinct commands:
Resources
┌──────────────┬─────────────────────┬────────┬───────────────┬──────────────┐
│ Type │ Name │ Status │ Last refresh │ Next refresh │
├──────────────┼─────────────────────┼────────┼───────────────┼──────────────┤
│ Model │ bids_data_model │ ✓ │ Jan 12, 9:05 │ - │
│ Metrics View │ bids_metrics │ ✓ │ Jan 12, 9:05 │ - │
│ Explore │ bids_explore │ ✓ │ Jan 12, 9:05 │ - │
└──────────────┴─────────────────────┴────────┴───────────────┴──────────────┘
Tables
┌─────────────────────┬───────────┬─────────────┬───────────┐
│ Name │ Row Count │ Column Count│ Size │
├─────────────────────┼───────────┼─────────────┼───────────┤
│ bids_data_model │ 1,854,664 │ 31 │ 46.7 MiB │
│ auction_data_model │ 82,995 │ 9 │ 952.4 MiB │
└─────────────────────┴───────────┴─────────────┴───────────┘
Benefits:
- Cleaner separation of concerns (deployment health vs. data volume)
- No empty columns — every row in the Tables section has meaningful data
- Mirrors CLI experience (
rill project tablesoutput) - Room to add row count and column count (like CLI) without cluttering the Resources table
- Simpler data model — no need to join two APIs with matching logic
Trade-offs:
- More UI surface area (two tables instead of one)
- Users need to look in two places
What do you think? Happy to discuss if you see it differently or have concerns with this approach.
Specific code comments (if we proceed with current approach):
-
selectors.ts— The caching and preloading logic is quite complex (~200 lines). If we go with a separate Tables section, this could be simplified since we'd just callOLAPListTablesdirectly without needing to join with resources. -
ProjectResourcesTable.svelte— The{#key tableSizes}block will cause the entire table to re-render when sizes load. This might cause a jarring UX with the table flickering.
Developed in collaboration with Claude Code

as discussed: https://rilldata.slack.com/archives/CTCJ58H3M/p1767734884155469
Checklist: