feat(meta): add materialized view metadata support

[TCeason]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR adds the meta-service foundation for materialized views. Query-layer integration, INSERT maintenance, REFRESH execution, and source-table DDL invalidation will be implemented in follow-up PRs.

A materialized view reuses the existing table storage and lifecycle, and its table ID is also its MV ID.

• TableMeta stores the physical schema and storage metadata. The source table ID is stored in the materialized_view_source_table_id option.
• MVDefinition is stored separately under the same table ID. It contains the original SQL, fully qualified SQL, logical schema, and maintenance mode.

Three MV-specific key families are added:

  __fd_materialized_view_definition/<tenant>/<mv_table_id>
      -> MVDefinition

  __fd_materialized_view_by_source/<tenant>/<source_table_id>/<mv_table_id>
      -> EmptyProto

  __fd_materialized_view_source_binding_version/<tenant>/<source_table_id>
      -> EmptyProto

Each source-to-MV relationship has its own key.

The source binding-version key is independent of MV membership. Future source-table RENAME and schema-changing DDL will atomically:

  1. Delete all SourceTableMVIdent(source_table_id, *) relationship keys.
  2. Rewrite MVSourceBindingVersionIdent(source_table_id) with the same EmptyProto value, advancing its KV sequence.
  3. Apply the source-table metadata or name-mapping change.

If CREATE MV commits first, the source-table DDL deletes its relationship.
If the source-table DDL commits first, the binding-version condition rejects the stale CREATE MV.
A CREATE MV committed between the DDL relationship list and transaction commit causes the DDL transaction to retry.

CREATE MV

TableApi::create_table() creates a visible, empty MV and atomically writes:

• TableMeta
• MVDefinition
• Table-name mappings
• SourceTableMVIdent(source_table_id, mv_table_id)

CREATE performs no initial refresh and records no source snapshot.

CREATE validates the source binding-version sequence captured while binding the MV definition.
It adds that sequence as a transaction condition before publishing the source relationship.

CREATE OR REPLACE MV

CREATE OR REPLACE atomically:

1. Marks the old table as dropped.
2. Removes the old MVDefinition and exact source relationship when the old table is an MV.
3. Creates the new MV definition and source relationship.

The replacement may reference the same or a different source table.

When the source table is unchanged, the transaction still replaces:

SourceTableMVIdent(source_table_id, old_mv_id)

with:

SourceTableMVIdent(source_table_id, new_mv_id)

because the replacement MV has a new table ID.

CREATE OR REPLACE does not advance the source binding version because it does not change the source-table definition.

DROP MV

DROP MV atomically marks the MV as dropped and removes its MVDefinition and exact source relationship.

Materialized views do not support UNDROP.

DROP and UNDROP Source Table

DROP source table retains its relationship keys and binding-version key unchanged, so UNDROP restores the same MV associations.

Garbage Collection

• MV table GC uses the ordinary table GC path and idempotently removes any remaining MVDefinition and exact source relationship.
• Source-table GC removes all relationship keys under the source-table prefix and deletes its binding-version key.

MaterializedViewApi

MaterializedViewApi::list_mvs_by_source_table_id() lists relationship keys by source-table prefix and returns Vec.

Each MVInfo contains:

• MV table ID
• MVDefinition
• MV TableMeta

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

---

This change is 

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

[drmingdrmer]

do not quite understand the relationship between a table and a materialized view. add some doc comment in the source code, not only in external feishu doc, to explain the design.

@drmingdrmer partially reviewed 15 files and made 8 comments.
Reviewable status: 14 of 35 files reviewed, 7 unresolved discussions (waiting on TCeason).

---

src/meta/api/src/api_impl/materialized_view_api.rs line 60 at r1 (raw file):

        let ident = SourceTableMVIdsIdent::new(tenant, source_table_id);
        self.get_pb(&ident).await
    }

store mvids in multiple keys: table_1/mv_1, table_1/mv_2... so that this method is implemented with a list operation. or defend your design.

Code quote:

    async fn get_mv_ids_by_source_table_id(
        &self,
        tenant: &Tenant,
        source_table_id: u64,
    ) -> Result<Option<SeqV<SourceTableMVIds>>, MetaError> {
        let ident = SourceTableMVIdsIdent::new(tenant, source_table_id);
        self.get_pb(&ident).await
    }

---

src/meta/api/src/api_impl/materialized_view_api.rs line 105 at r1 (raw file):

                deserialize_struct_get_response::<TableId>(table_meta_response)?;
            assert_eq!(definition_ident, MVDefinitionIdent::new(tenant, *mv_id));
            assert_eq!(table_ident, TableId::new(*mv_id));

a data error should not panic databend-query.

Code quote:

            assert_eq!(definition_ident, MVDefinitionIdent::new(tenant, *mv_id));
            assert_eq!(table_ident, TableId::new(*mv_id));

---

src/meta/app/src/schema/materialized_view/source_table_mv_ids_ident.rs line 20 at r1 (raw file):

/// `__fd_materialized_view_by_source/<tenant>/<source_table_id>` -> [`SourceTableMVIds`]
pub type SourceTableMVIdsIdent = TIdent<SourceTableMVIdsResource, u64>;

why not using multiple key/value to represent the relationship? but using a list embedded in a value.

Code quote:

/// `__fd_materialized_view_by_source/<tenant>/<source_table_id>` -> [`SourceTableMVIds`]
pub type SourceTableMVIdsIdent = TIdent<SourceTableMVIdsResource, u64>;

---

src/meta/app/src/schema/table/mod.rs line 532 at r1 (raw file):

    /// Definition stored with a materialized view created as a hidden table.
    pub mv_definition: Option<MVDefinition>,

the doc comment can be improved to express the usage and role of this field.

Code quote:

    /// Definition stored with a materialized view created as a hidden table.
    pub mv_definition: Option<MVDefinition>,

---

src/meta/protos/proto/materialized_view.proto line 30 at r1 (raw file):

  string original_query = 1;
  // Fully qualified SQL used by Databend.
  string query = 2;

give example to show the differences between them.

and the query is text based, what if some tables are renamed? how is the consistency maintained?

Code quote:

  // SQL text submitted by the user.
  string original_query = 1;
  // Fully qualified SQL used by Databend.
  string query = 2;

---

src/meta/protos/proto/materialized_view.proto line 32 at r1 (raw file):

  string query = 2;
  // Logical schema of the user's defining query, used for the externally
  // visible column types. TableMeta.schema stores the rewritten physical

what TableMeta? there is no table mentioned here.

---

src/meta/protos/proto/materialized_view.proto line 42 at r1 (raw file):

message SourceTableMVIds {
  uint64 ver = 100;
  uint64 min_reader_ver = 101;

does it need to be serialized/deserialized alone? if not, it is only contained in other message and does not need to check compatibility and does not need ver and min_reader_ver

Code quote:

  uint64 ver = 100;
  uint64 min_reader_ver = 101;

[TCeason]

@TCeason made 7 comments.
Reviewable status: 12 of 35 files reviewed, 7 unresolved discussions (waiting on drmingdrmer).

---

src/meta/api/src/api_impl/materialized_view_api.rs line 60 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

store mvids in multiple keys: table_1/mv_1, table_1/mv_2... so that this method is implemented with a list operation. or defend your design.

With one KV per relationship, the sequence number of each individual KV only versions that relationship; it does not version the relationship set as a whole. We would therefore need an additional per-source version key, for example:

source_mv/<source_id>/<mv_id> -> ()
source_mv_version/<source_id> -> ()

Every CREATE, DROP, or CREATE OR REPLACE operation would have to update both the relationship key and the version key atomically. Readers would use the version key’s sequence as a transaction condition to detect relationships added or removed during the scan.

With the current representation:

source_mv/<source_id> -> [mv_id_1, mv_id_2, ...]

the KV record’s own sequence number already acts as the version of the complete relationship set, so no separate version field or version key is required.

---

src/meta/api/src/api_impl/materialized_view_api.rs line 105 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

a data error should not panic databend-query.

They are redundant internal invariant checks. Already deleted.

---

src/meta/app/src/schema/materialized_view/source_table_mv_ids_ident.rs line 20 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

why not using multiple key/value to represent the relationship? but using a list embedded in a value.

With one KV per relationship, the sequence number of each individual KV only versions that relationship; it does not version the relationship set as a whole. We would therefore need an additional per-source version key, for example:

source_mv/<source_id>/<mv_id> -> ()
source_mv_version/<source_id> -> ()

Every CREATE, DROP, or CREATE OR REPLACE operation would have to update both the relationship key and the version key atomically. Readers would use the version key’s sequence as a transaction condition to detect relationships added or removed during the scan.

With the current representation:

source_mv/<source_id> -> [mv_id_1, mv_id_2, ...]

the KV record’s own sequence number already acts as the version of the complete relationship set, so no separate version field or version key is required.

---

src/meta/app/src/schema/table/mod.rs line 532 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

the doc comment can be improved to express the usage and role of this field.

Done

---

src/meta/protos/proto/materialized_view.proto line 30 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

give example to show the differences between them.

and the query is text based, what if some tables are renamed? how is the consistency maintained?

If a referenced table is renamed, this text is not updated. Queries against the materialized view then fail, and the materialized view must be recreated.

---

src/meta/protos/proto/materialized_view.proto line 32 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

what TableMeta? there is no table mentioned here.

I modify the comment.

// Logical schema of the user's defining query, used for the externally
// visible column types. The corresponding materialized-view table's
// TableMeta.schema stores the rewritten physical schema that Fuse uses
// to read and write the materialized data.

---

src/meta/protos/proto/materialized_view.proto line 42 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

does it need to be serialized/deserialized alone? if not, it is only contained in other message and does not need to check compatibility and does not need ver and min_reader_ver

Yes. SourceTableMVIds is serialized and deserialized independently as the value of __fd_materialized_view_by_source//<source_table_id>, so it needs its own ver and min_reader_ver fields

[TCeason]

@TCeason made 1 comment.
Reviewable status: 12 of 35 files reviewed, 7 unresolved discussions (waiting on drmingdrmer).

---

src/meta/app/src/schema/materialized_view/source_table_mv_ids_ident.rs line 20 at r1 (raw file):

Previously, TCeason wrote…

With one KV per relationship, the sequence number of each individual KV only versions that relationship; it does not version the relationship set as a whole. We would therefore need an additional per-source version key, for example:

source_mv/<source_id>/<mv_id> -> ()
source_mv_version/<source_id> -> ()

Every CREATE, DROP, or CREATE OR REPLACE operation would have to update both the relationship key and the version key atomically. Readers would use the version key’s sequence as a transaction condition to detect relationships added or removed during the scan.

With the current representation:

source_mv/<source_id> -> [mv_id_1, mv_id_2, ...]

the KV record’s own sequence number already acts as the version of the complete relationship set, so no separate version field or version key is required.

The reason for tracking changes to the complete MV list is source-table INSERT consistency, not only the implementation of CREATE or DROP.

If a source table has materialized views, an INSERT into that source is effectively a multi-table insert. The commit includes the source table and all materialized views that depend on it.

For example, suppose an INSERT reads:

  seq = 10
  source_mv/source_1 -> [mv_1]

The INSERT then prepares writes for source_1 and mv_1.

If mv_2 is created concurrently, the relationship becomes:

  seq = 11
  source_mv/source_1 -> [mv_1, mv_2]

Before committing, the INSERT checks that the sequence is still 10. Since it has changed to 11, the commit fails and the INSERT retries with the new MV list.

Without this check, the INSERT could commit data to source_1 and mv_1 without updating mv_2. Concurrent DROP or CREATE OR REPLACE has a similar problem: the INSERT could write to a dropped or superseded MV instead of the current MV.

This check gives INSERT and MV DDL a consistent order. If the DDL commits first, the INSERT detects the changed sequence and retries with the new MV list. If the INSERT commits first, the DDL proceeds after that INSERT.

With one KV per relationship:

  source_mv/<source_id>/<mv_id> -> ()

Each KV sequence number only tracks one relationship. After listing the relationship keys, an INSERT can check the sequence numbers of the relationships it found, but it cannot detect a new relationship inserted after the listing. The new relationship is represented by a new key that was not part of the original result.

To detect that case, the multiple-key representation would need another per-source key:

  source_mv/<source_id>/<mv_id> -> ()
  source_mv_version/<source_id> -> ()

Every CREATE, DROP, and CREATE OR REPLACE operation would have to update both the relationship key and source_mv_version atomically. The INSERT would then check the sequence number of source_mv_version before committing.

With the current representation:

  source_mv/<source_id> -> [mv_id_1, mv_id_2, ...]

adding, removing, or replacing any MV updates the same KV. Therefore, that KV’s sequence number tells the INSERT whether the complete MV list changed after it was read. No additional per-source version key is required.

The INSERT and REFRESH paths that use this sequence as a commit condition will be implemented in the follow-up PR.

[drmingdrmer]

@drmingdrmer made 1 comment.
Reviewable status: 11 of 35 files reviewed, 7 unresolved discussions (waiting on TCeason).

---

src/meta/protos/proto/materialized_view.proto line 32 at r1 (raw file):

Previously, TCeason wrote…

I modify the comment.

// Logical schema of the user's defining query, used for the externally
// visible column types. The corresponding materialized-view table's
// TableMeta.schema stores the rewritten physical schema that Fuse uses
// to read and write the materialized data.

what's the definition of logical schema and physical schema? undefined concepts should not be used.

[TCeason]

@TCeason made 1 comment.
Reviewable status: 11 of 35 files reviewed, 7 unresolved discussions (waiting on drmingdrmer).

---

src/meta/protos/proto/materialized_view.proto line 32 at r1 (raw file):

Previously, drmingdrmer (张炎泼) wrote…

what's the definition of logical schema and physical schema? undefined concepts should not be used.

Added more specific comment

  // Columns and types returned when querying the materialized view. For example,
  // if orders.amount is a non-null Float64:
  // SELECT AVG(amount) AS avg_amount FROM orders
  // this schema is [avg_amount: Float64]. The materialized-view TableMeta.schema
  // is [avg_amount: Tuple(Float64, UInt64)]. The Fuse table stores this mergeable
  // AVG state (sum and count), which is finalized as avg_amount when queried.
  DataSchema schema = 3;