Support COMPACTED format for log tables #1605

polyzos · 2025-08-28T09:48:21Z

Introduce Compacted Log Format, for scenarios that the KV store doesn't require projection pushdown, like tables with aggregates

Copilot

Pull Request Overview

Introduces Compacted Log Format for scenarios where the KV store doesn't require projection pushdown, particularly for primary key tables with aggregates.

Adds support for COMPACTED log format as a third option alongside ARROW and INDEXED
Implements CompactedRow-based record encoding/decoding with space-optimized binary format
Updates validation logic to allow both ARROW and COMPACTED formats when KV format is COMPACTED

Reviewed Changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
TableDescriptorValidation.java	Updates validation to allow COMPACTED log format for primary key tables
CompactedWalBuilder.java	New WAL builder implementation for compacted log format
KvTablet.java	Adds case handling for COMPACTED log format
MemoryLogRecordsCompactedBuilder.java	New builder extending common row-based builder for compacted records
CompactedLogRecord.java	Core log record implementation for compacted row format
AbstractRowMemoryLogRecordsBuilder.java	Extracted common functionality for row-based log builders
LogFormat.java	Adds COMPACTED enum value and updates documentation
CompactedLogWriteBatch.java	Client-side batch writer for compacted format
WriteRecord.java	Adds factory method for compacted append operations

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

fluss-common/src/main/java/org/apache/fluss/record/LogRecordReadContext.java

polyzos · 2025-11-10T07:45:45Z

I'm closing this as #85 addresses that.. Gonna check though if there are any possible improvements that can benefit the current implementation and create a separate PR if required.

polyzos · 2025-11-11T07:45:04Z

@swuferhong, before leaving this, since as you already addressed this, I wanted to check with you if you think there is anything we can reuse in case I missed something.
I only see this check https://github.com/apache/fluss/pull/1605/files#diff-74b13bc1eb78378f01a154188cdb7942d9b67dff4bbe2c1354a92617ae829349R148
And maybe there is any test we can reuse? Otherwise seems to me you have already covered everything, and my implementation doesn't add anything more, really.

swuferhong · 2025-11-11T13:37:29Z

@swuferhong, before leaving this, since as you already addressed this, I wanted to check with you if you think there is anything we can reuse in case I missed something. I only see this check https://github.com/apache/fluss/pull/1605/files#diff-74b13bc1eb78378f01a154188cdb7942d9b67dff4bbe2c1354a92617ae829349R148 And maybe there is any test we can reuse? Otherwise seems to me you have already covered everything, and my implementation doesn't add anything more, really.

Hi, @polyzos. I have wrongly closed #85 while organizing the PR. The work for "introduce compacted log row" hasn't been done yet. You can proceed with it in your current PR, but it's probably not high priority since there's currently no clear use case requiring this format.

polyzos · 2025-11-11T13:49:13Z

@swuferhong So what happens with all the work that got merged? because I see there is lot of similarity in both PRs
Otherwise, maybe you can take a look at my PR?

swuferhong · 2025-11-12T07:59:52Z

@swuferhong So what happens with all the work that got merged? because I see there is lot of similarity in both PRs Otherwise, maybe you can take a look at my PR?

Hi, @polyzos I don't quite understand this 'So what happens with all the work that got merged?' as this work wasn't merged yet?

I will try to review it. However, I'm not quite sure if it's necessary to introduce this format.

xx789633 · 2025-12-03T08:13:52Z

This feature is useful when projection is not required. it may not be a high priority.

luoyuxia · 2025-12-04T02:20:25Z

@polyzos Hi, I'm thinking to push forward this pr. We have a user case that big embeding vector write to fluss , and flusss tier to lance when column prune is not required. Then use compacted change log format can much reduce the cost of server rebuilding arrow change log. Could you please rebase main branh?

polyzos · 2025-12-04T08:33:00Z

@luoyuxia done 🫡

luoyuxia · 2025-12-04T09:14:51Z

@polyzos Hi, thanks for your greate work. I verify it in my local, it works.
I see you pr support:

write compacted row to log table
generate change log with compacted row

But for me, the real highest priority is generate change log with compacted row, so could you please split this pr into two, one is to support write compacted row to log table, another one is support change log with compacted row? I'll review the pr for support change log with compacted row?

…classes # Conflicts: # fluss-common/src/main/java/org/apache/fluss/record/MemoryLogRecordsIndexedBuilder.java

…cordsCompactedBuilder.java Co-authored-by: Copilot <[email protected]>

wuchong

Thanks @polyzos , this looks good to me. I only improved some code, like merge tests to reduce test time.

wuchong · 2025-12-25T09:34:54Z

fluss-client/src/main/java/org/apache/fluss/client/write/WriteRecord.java

+            CompactedRow row,
+            @Nullable byte[] bucketKey) {
+        checkNotNull(row);
+        int estimatedSizeInBytes = CompactedLogRecord.sizeOf(row) + RECORD_BATCH_HEADER_SIZE;


This RECORD_BATCH_HEADER_SIZE is for kv record batch, we should use LogRecordBatchFormat.recordBatchHeaderSize(CURRENT_LOG_MAGIC_VALUE)

polyzos marked this pull request as ready for review August 31, 2025 14:07

polyzos requested a review from Copilot August 31, 2025 14:07

This comment was marked as outdated.

Sign in to view

polyzos force-pushed the main branch 3 times, most recently from d88c76c to 434a4f4 Compare August 31, 2025 15:13

polyzos requested a review from Copilot September 1, 2025 13:23

This comment was marked as outdated.

Sign in to view

polyzos requested review from luoyuxia, swuferhong and wuchong September 3, 2025 05:56

polyzos mentioned this pull request Sep 9, 2025

Bump LogRecordBatch's CURRENT_LOG_MAGIC_VALUE to V1 to support leaderEpoch #778

Merged

polyzos force-pushed the support-compacted-log-format branch 4 times, most recently from e919a85 to 4639453 Compare September 10, 2025 09:41

polyzos requested a review from Copilot September 10, 2025 13:20

Copilot AI reviewed Sep 10, 2025

View reviewed changes

fluss-common/src/main/java/org/apache/fluss/record/LogRecordReadContext.java Outdated Show resolved Hide resolved

polyzos closed this Nov 10, 2025

polyzos reopened this Nov 11, 2025

polyzos force-pushed the support-compacted-log-format branch from a6974fd to 6b6fda0 Compare December 4, 2025 07:06

polyzos and others added 27 commits December 25, 2025 17:36

Add Compacted in LogFormat

1165438

Introduce CompactedLogRecord and related functionality

d0698ae

Add tests for CompactedLogRecord

8ced8f5

fix vcs.xml

88a06f8

apply spotless

5396cc7

revert vcs changes

b362b4b

update AppendWriterImpl to support compactedRow

6caa7a1

apply spotless

8cad339

CompactedLogWriteBatch and MemoryLogRecordsCompactedBuilder with tests

40db54b

refactor to minimize duplicate code between indexed and compactedrow …

e6d4567

…classes # Conflicts: # fluss-common/src/main/java/org/apache/fluss/record/MemoryLogRecordsIndexedBuilder.java

allow kv to use compacted log format

8286065

add end2end functionality for compact log write and tests

372818d

throw exception if projection is used in compacted log format

7c4ef28

small improvements

ac1ca45

Update fluss-common/src/main/java/org/apache/fluss/record/MemoryLogRe…

f1eb795

…cordsCompactedBuilder.java Co-authored-by: Copilot <[email protected]>

fix violations

f887182

fix compilation issues after rebase

f6fdc58

fix import issue

bbf0511

fix import issue

8843745

fix checkstyle issue

a2ec69f

fix remaining import issues

a79d9ac

fix tests

5a91d49

fix failing tests

c17103c

rebase branch

9efb9d2

fix rebase compilation issues

d7dd776

fix rebase issues

c63180c

improve some code

cf23fc3

wuchong approved these changes Dec 25, 2025

View reviewed changes

wuchong force-pushed the support-compacted-log-format branch from f9e7526 to cf23fc3 Compare December 25, 2025 09:37

wuchong merged commit a908690 into apache:main Dec 25, 2025
5 checks passed

Support COMPACTED format for log tables #1605

Support COMPACTED format for log tables #1605

Uh oh!

Conversation

polyzos commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

polyzos commented Nov 10, 2025

Uh oh!

polyzos commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swuferhong commented Nov 11, 2025

Uh oh!

polyzos commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

swuferhong commented Nov 12, 2025

Uh oh!

xx789633 commented Dec 3, 2025

Uh oh!

luoyuxia commented Dec 4, 2025

Uh oh!

polyzos commented Dec 4, 2025

Uh oh!

luoyuxia commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wuchong left a comment

Choose a reason for hiding this comment

Uh oh!

wuchong Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

polyzos commented Aug 28, 2025 •

edited

Loading

polyzos commented Nov 11, 2025 •

edited

Loading

polyzos commented Nov 11, 2025 •

edited

Loading

luoyuxia commented Dec 4, 2025 •

edited

Loading