INSERT INTO fails on tables with sparse partition-tree primary key encoding

[discord9]

What type of bug is this?

Unexpected error

What subsystems are affected?

Storage Engine, Table Engine, Datanode, Frontend, Other

Minimal reproduce step

Create a Mito table with generic string primary-key columns and sparse partition-tree primary key encoding enabled:

CREATE TABLE sparse_pk_write_repro (
    tag_a STRING NULL,
    tag_b STRING NULL,
    tag_c STRING NULL,
    ts TIMESTAMP(3) NOT NULL,
    val DOUBLE NULL,
    TIME INDEX (ts),
    PRIMARY KEY (tag_a, tag_b, tag_c)
)
ENGINE=mito
WITH(
    'memtable.type' = 'partition_tree',
    'memtable.partition_tree.primary_key_encoding' = 'sparse',
    physical_metric_table = 'true'
);

Then write a row whose primary-key columns are normal string values:

INSERT INTO sparse_pk_write_repro VALUES
    ('tag-a', 'tag-b', 'tag-c', '2024-01-01 00:00:00', 10.0);

A similar failure can also be reached through a flow sink write path: the flow can be created successfully, but flushing/writing into a sink table with sparse partition-tree primary key encoding fails.

Reproduction matrix from an anonymized flow-sink test:

• No special sink options: succeeds, rows are written.
• Same table/options with sparse partition-tree primary key encoding removed: succeeds.
• Same table/options with physical_metric_table = 'true' removed but sparse partition-tree primary key encoding kept: still fails.
• Sparse partition-tree primary key encoding enabled: fails.

What did you expect to see?

INSERT INTO should either:

1. support tables configured with sparse partition-tree primary key encoding and accept normal string primary-key values, or
2. reject such incompatible table options earlier with a clear error message instead of failing later during writes.

For flow sinks, CREATE FLOW should not succeed if the sink table options are incompatible with the flow write path.

What did you see instead?

The write fails with a String-to-Binary cast error. In the flow-sink reproduction, CREATE FLOW succeeds, but ADMIN FLUSH_FLOW fails when it tries to write into the sink table.

What operating system did you use?

Linux x86_64

What version of GreptimeDB did you use?

Current main / development build.

Relevant log output and stack trace

Failed to execute admin function flush_flow: Execution error: Failed to cast value ref String("<redacted>") to Binary

Additional note: the failure appears tied to sparse partition-tree primary key encoding. physical_metric_table = 'true' is not required for the observed String-to-Binary failure.

[agrawalx]

I would like to take this one.
Proposed fix - reject early with a clean error message. I think supporting sparse encoding here adds little value and will involve creating a similar RowModifier like Metric engine and having something similar tsid which is a much bigger change.

Happy to open a PR if this approach sounds right

[evenyag]

I would like to take this one. Proposed fix - reject early with a clean error message. I think supporting sparse encoding here adds little value and will involve creating a similar RowModifier like Metric engine and having something similar tsid which is a much bigger change.

Happy to open a PR if this approach sounds right

Sounds good. This case isn't a common use case. Feel free to have a try.