plugins/sql: Add refresh time tracking, time_msec support, and extend pagination to all indexed tables

[Andezion]

Problem

The SQL plugin had several TODO items:

• No visibility into data freshness - Users couldn't see when tables were last refreshed
• Missing time_msec field support - Timestamp fields from JSON schemas weren't properly handled
• Limited pagination - Only chainmoves and channelmoves used pagination, causing performance issues on large datasets

Solution

1. TODO 2: Refresh time tracking in API
Added tracking of table refresh completion times:

• New fields in table_desc: last_refresh_time and has_been_refreshed
• Updated listsqlschemas to expose last_refresh_seconds_ago
• New sqlstatus command to monitor refresh status with detailed info:
  • has_been_refreshed, needs_refresh, refreshing flags
  • last_refresh_seconds_ago - time since last update
  • refresh_duration_seconds - current refresh duration
  • last_created_index - for pagination tracking

2. TODO 8: time_msec field type support
Added FIELD_TIME_MSEC enum type for millisecond timestamps:

• Maps to INTEGER SQL type for efficient storage
• Automatically handles JSON time_msec fields
• Enables SQL datetime functions on timestamp fields

3. TODO 10: General pagination API
Extended pagination from 2 tables to 6 tables using refresh_by_created_index:

• listhtlcs - now uses pagination with 'htlcs' wait
• listforwards - now uses pagination with 'forwards' wait
• listinvoices - now uses pagination with 'invoices' wait
• listsendpays - now uses pagination with 'sendpays' wait

About TODO 11: Normalize account_id fields.
The problem is - account_id in chainmoves and channelmoves tables is TEXT and highly duplicated (same values like "wallet", "external" repeated thousands of times).

I propose such solution:

• Create accounts table with unique account_id values
• Change chainmoves/channelmoves to reference accounts table via INTEGER foreign key
• Create VIEWs for backward compatibility with current API

That will give us:

• Reduced storage (account_id stored once, not N times)
• Faster queries (INTEGER joins vs TEXT comparison)
• Referential integrity via foreign keys

Implementation may be like this:

-- New table
CREATE TABLE accounts (
    id INTEGER PRIMARY KEY,
    account_id TEXT UNIQUE
);

-- Modify existing tables
ALTER TABLE chainmoves ADD COLUMN account_id_ref INTEGER REFERENCES accounts(id);

-- Backward compatible view
CREATE VIEW chainmoves_compat AS 
SELECT c.*, a.account_id 
FROM chainmoves c 
JOIN accounts a ON c.account_id_ref = a.id;

Should this be automatic migration or optional flag?
Keep old column as deprecated or remove completely?

Changelog

• Changelog-Added: Added sqlstatus command to monitor table refresh status
• Changelog-Added: Added time_msec field type support for timestamp fields
• Changelog-Changed: listsqlschemas now includes last_refresh_seconds_ago to show data freshness
• Changelog-Changed: listhtlcs, listforwards, listinvoices, and listsendpays now use pagination for better performance

Important

26.04 FREEZE March 11th: Non-bugfix PRs not ready by this date will wait for 26.06.

RC1 is scheduled on March 23rd

The final release is scheduled for April 15th.

Checklist

Before submitting the PR, ensure the following tasks are completed. If an item is not applicable to your PR, please mark it as checked:

• The changelog has been updated in the relevant commit(s) according to the guidelines.
• Tests have been added or modified to reflect the changes.
• Documentation has been reviewed and updated as needed.
• Related issues have been listed and linked, including any that this PR closes.
• Important All PRs must consider how to reverse any persistent changes for tools/lightning-downgrade