Skip to content

Conversation

@royendo
Copy link
Contributor

@royendo royendo commented Jan 7, 2026

as discussed: https://rilldata.slack.com/archives/CTCJ58H3M/p1767734884155469

Screenshot 2026-01-08 at 10 26 48

Checklist:

  • Covered by tests
  • Ran it and it works as intended
  • Reviewed the diff before requesting a review
  • Checked for unhandled edge cases
  • Linked the issues it closes
  • Checked if the docs need to be updated. If so, create a separate Linear DOCS issue
  • Intend to cherry-pick into the release branch
  • I'm proud of this work!

royendo and others added 3 commits January 7, 2026 18:27
The OLAPListTables API returns paginated results with a nextPageToken field.
Previously only the first page was fetched, causing materialized models and
tables on subsequent pages to show as having no size data.

This change replaces the derived store pattern with a readable store that:
- Fetches all pages sequentially for each connector
- Detects nextPageToken in responses and creates follow-up queries
- Accumulates all tables before building the final size map
- Handles loading/error states correctly

This fixes the issue where 5 models were detected but only 2 tables (from the
first page) were returned, causing models like "auction_data_model" to show
"-" instead of their actual size.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
The issue was that useModelTableSizes was creating a new store on every
render when resources changed. If the resources loaded before the queries
completed, the component would be unmounted/remounted, cancelling the queries.

This change:
1. Caches stores by instanceId:connectorArray to prevent recreation
2. Eagerly subscribes to each store to keep queries alive
3. Uses WeakMap to prevent memory leaks from old subscriptions
4. Ensures queries complete even if component re-renders quickly

Now when you refresh the status page, queries stay alive in the background
and complete asynchronously, updating the table when ready.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
@royendo royendo requested a review from ericpgreen2 January 7, 2026 23:34
@royendo
Copy link
Contributor Author

royendo commented Jan 7, 2026

a first pass at getting something in while project status 2.0 gets started as well as https://linear.app/rilldata/issue/PLAT-376/instrumentation-for-physical-table-size-of-models

royendo and others added 6 commits January 8, 2026 09:45
The store cache was keeping stale subscriptions alive across page refreshes.
When you refreshed the page, the old cached store would be returned even if
the resources had changed or needed fresh data.

This change simplifies the approach:
- Create fresh stores on each useModelTableSizes call
- Let TanStack Query handle result caching (HTTP level)
- Ensure queries always run with latest connector data
- No issues with premature garbage collection

Now on page refresh, sizes will load correctly immediately.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
The issue was that columns were defined as a static const, so the accessor
function captured the initial tableSizes reference and never updated when
tableSizes changed.

Changes:
1. Made columns reactive with $: so they update when tableSizes changes
2. Added console logging to ProjectResources and ProjectResourcesTable
  to track when stores and columns are updated
3. Added logging to selectors to see preload and store update timing

This should fix the issue where the table shows "-" on initial load but
correctly shows sizes after navigating away and back.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
The core issue: columns were defined as a static const, capturing
the initial tableSizes reference. When the store updated with new data,
the column accessor functions still referenced the old empty Map.

Solution: Make columns reactive with $: so they recreate whenever
tableSizes changes, ensuring accessor functions get the current data.

This allows the UI to update asynchronously as size data arrives.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Haiku 4.5 <[email protected]>
@royendo
Copy link
Contributor Author

royendo commented Jan 8, 2026

fixed bug, where values not shown unless action done on cloud, now it updates automatically: https://www.loom.com/share/825b70c51d234177a8949d7c920e4894

@royendo
Copy link
Contributor Author

royendo commented Jan 8, 2026

when not materialized:
Screenshot 2026-01-08 at 10 30 17

Copy link
Contributor

@ericpgreen2 ericpgreen2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on! I've been thinking about the design for this feature and want to propose a different direction before we proceed.

The current approach adds a "Size" column to the existing Resources table. While this works, I think it mixes two conceptually different concerns:

  1. Resources table = "Is my deployment healthy?" (reconciliation status, refresh times, errors)
  2. Table metrics = "How much data do I have?" (size, row count, column count)

This is why most rows show "-" for size — only Models map to database tables. Explores, Metrics Views, Canvas, etc. don't have a "size" concept.

Proposed Alternative: Separate "Tables" Section

I'd recommend mirroring the CLI's approach, where rill project status and rill project tables are distinct commands:

Resources
┌──────────────┬─────────────────────┬────────┬───────────────┬──────────────┐
│ Type         │ Name                │ Status │ Last refresh  │ Next refresh │
├──────────────┼─────────────────────┼────────┼───────────────┼──────────────┤
│ Model        │ bids_data_model     │ ✓      │ Jan 12, 9:05  │ -            │
│ Metrics View │ bids_metrics        │ ✓      │ Jan 12, 9:05  │ -            │
│ Explore      │ bids_explore        │ ✓      │ Jan 12, 9:05  │ -            │
└──────────────┴─────────────────────┴────────┴───────────────┴──────────────┘

Tables
┌─────────────────────┬───────────┬─────────────┬───────────┐
│ Name                │ Row Count │ Column Count│ Size      │
├─────────────────────┼───────────┼─────────────┼───────────┤
│ bids_data_model     │ 1,854,664 │ 31          │ 46.7 MiB  │
│ auction_data_model  │ 82,995    │ 9           │ 952.4 MiB │
└─────────────────────┴───────────┴─────────────┴───────────┘

Benefits:

  • Cleaner separation of concerns (deployment health vs. data volume)
  • No empty columns — every row in the Tables section has meaningful data
  • Mirrors CLI experience (rill project tables output)
  • Room to add row count and column count (like CLI) without cluttering the Resources table
  • Simpler data model — no need to join two APIs with matching logic

Trade-offs:

  • More UI surface area (two tables instead of one)
  • Users need to look in two places

What do you think? Happy to discuss if you see it differently or have concerns with this approach.


Specific code comments (if we proceed with current approach):

  1. selectors.ts — The caching and preloading logic is quite complex (~200 lines). If we go with a separate Tables section, this could be simplified since we'd just call OLAPListTables directly without needing to join with resources.

  2. ProjectResourcesTable.svelte — The {#key tableSizes} block will cause the entire table to re-render when sizes load. This might cause a jarring UX with the table flickering.


Developed in collaboration with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants