Add compatibility API for OSM

[svenklemm]

Add some helper function to decouple OSM from direct catalog access:

Disable-check: force-changelog-file

[github-actions]

@dbeck, @akuzm: please review this pull request.

Powered by pull-review

[akuzm]

Let's add more incorrect values to the tests.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[zilder]

I analyzed the access patterns to timescaledb catalog with claude and they seem to fall into one of the five categories (this is a rough draft, may need more thorough review):

1. Resolve a relation to a hypertable

  -- handles both real hypertables and continuous aggregates
  -- (returns the materialization hypertable for a cagg)
  get_hypertable_info(relation REGCLASS)
  RETURNS TABLE (
      hypertable_id    INTEGER,
      schema_name      NAME,
      table_name       NAME,
      is_cagg          BOOLEAN
  )

If relation is a cagg, lookup its materialization hypertable. Or potentially we can use timescaledb_information.continuous_aggregates to find materialization hypertable and make get_hypertable_info simpler.

2. Primary dimension of a hypertable or a Cagg:

get_primary_dimension(hypertable_id INTEGER)
  RETURNS TABLE (
      column_name         NAME,
      column_type         REGTYPE,
      integer_now_func    REGPROC
  )

Note that for a cagg/materialization hypertable it may be necessary to check integer_now_func of its raw hypertable.

3. Chunk info, by oid and by id

  get_chunk_info(chunk REGCLASS)        -- and an _by_id(chunk_id INTEGER) variant
  RETURNS TABLE (
      chunk_id        INTEGER,
      hypertable_id   INTEGER,
      ht_schema_name  NAME,
      ht_table_name   NAME,
      chunk_schema    NAME,
      chunk_name      NAME,
      is_osm_chunk    BOOLEAN,
      is_frozen       BOOLEAN   -- or the `chunk.status` column
  )

4. Chunk's primary-dimension range

  get_chunk_primary_range(chunk_id INTEGER)   -- or chunk REGCLASS
  RETURNS TABLE (
      range_start   BIGINT,   -- internal units
      range_end     BIGINT
  )

5. Lock osm chunk's dimension slice

lock_osm_dimension_slice(hypertable REGCLASS) RETURNS VOID -- or hypertable_id

Runs SELECT * FROM _timescaledb_catalog.dimension_slice ... FOR UPDATE for the OSM chunk.

[svenklemm]

You dont really want to return table, right? You want named record

[zilder]

I'm still considering another approach with views. We'll still need a couple of functions (the dimension slice lock and the recursive integer_now_func lookup function). The views could look something like this:

CREATE OR REPLACE VIEW ts_hypertable AS
  WITH
  primary_dimension AS (
    SELECT DISTINCT ON (d.hypertable_id)
        d.hypertable_id,
        d.id                        AS dimension_id,
        d.column_name,
        d.column_type
    FROM _timescaledb_catalog.dimension d
    ORDER BY d.hypertable_id, d.id
  )
  SELECT
    h.id                            AS hypertable_id,
    h.schema_name,
    h.table_name,
    class.oid                       AS hypertable_relid,
    pd.column_name                  AS dim_column_name,
    pd.column_type                  AS dim_column_type
  FROM _timescaledb_catalog.hypertable h
  JOIN primary_dimension pd  ON pd.hypertable_id = h.id
  JOIN pg_class class ON class.relname = h.table_name
  JOIN pg_namespace nsp ON nsp.nspname = h.schema_name
      AND nsp.oid = class.relnamespace;

CREATE OR REPLACE VIEW ts_chunk AS
  SELECT DISTINCT ON (c.id)
      c.id                            AS chunk_id,
      c.hypertable_id,
      c.schema_name                   AS chunk_schema,
      c.table_name                    AS chunk_name,
      class.oid                       AS chunk_relid,
      c.osm_chunk                     AS is_osm_chunk,
      (c.status & 4) <> 0             AS is_frozen, -- bit 4 == frozen
      ds.range_start,
      ds.range_end
  FROM _timescaledb_catalog.chunk c
  JOIN _timescaledb_catalog.dimension_slice ds
         ON ds.chunk_id = c.id
  JOIN pg_class class ON class.relname = c.table_name
  JOIN pg_namespace nsp ON nsp.nspname = c.schema_name
    AND nsp.oid = class.relnamespace;

I want to try to rewrite OSM to use these.

[zilder]

I'm working on a PoC build of OSM that uses this API. There are a couple of issues I found so far. Also one more version of get_hypertable_info is needed with hypertable id as an argument:

CREATE OR REPLACE FUNCTION _timescaledb_functions.get_hypertable_info(IN hypertable_id INTEGER, OUT hypertable_id INTEGER, OUT schema_name name, OUT table_name name) AS $$
  SELECT ht.id, ht.schema_name, ht.table_name
  FROM _timescaledb_catalog.hypertable ht
  WHERE id = hypertable_id;
$$ LANGUAGE SQL STABLE SET search_path = pg_catalog, pg_temp;

[timescale-automation]

Automated backport to 2.28.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.28.x
You are currently cherry-picking commit e1db40e38.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   sql/osm_api.sql
	modified:   tsl/test/shared/expected/extension.out
	new file:   tsl/test/shared/expected/osm_api.out
	modified:   tsl/test/shared/sql/CMakeLists.txt
	new file:   tsl/test/shared/sql/osm_api.sql

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   sql/updates/reverse-dev.sql

Job log