[log] Follower fetch log already move to remote need to return RemoteLogFetchInfo to rebuild WriterState instead of throwing LogOffsetOutOfRangeException #728
Conversation
|
The test |
swuferhong
force-pushed
the
follower-fetch-remote-log
branch
from
8beca97 to
e972059
Compare
|
| log.writerStateManager().truncateFullyAndReloadSnapshots(); | ||
| File snapshotFile = FlussPaths.writerSnapshotFile(log.getLogDir(), nextFetchOffset); | ||
| buildWriterIdSnapshotFile(snapshotFile, remoteLogSegmentWithMaxStartOffset, rlm); | ||
| log.writerStateManager().reloadSnapshots(); |
There was a problem hiding this comment.
Why separate the original truncateFullyAndReloadSnapshots into truncateFullyAndStartAt(0) and reloadSnapshots()? It seems it is not necessary. If not necessary, we can avoid introducing the WriterStateManager#reloadSnapshots() method.
There was a problem hiding this comment.
Adding some comments to identify why do this.
fluss-server/src/main/java/com/alibaba/fluss/server/replica/ReplicaManager.java
Show resolved
Hide resolved
| lastOffset, | ||
| false); | ||
| // TODO, set to logStartOffset instead of 0 if we introduce logStartOffset. | ||
| rebuildWriterState(writerStateManager, localLog.getSegments(), 0, lastOffset, false); |
There was a problem hiding this comment.
Why the original localLog.getLocalLogStartOffset() is incorrect? And what's the differences between localLog.getLocalLogStartOffset() and the new logStartOffset ?
There was a problem hiding this comment.
Here, I use 0 as the logStartOffset passed into rebuildWriterState(). The reason is that the current implementation of logStartOffset in Fluss is not yet fully refined, and there may be cases where logStartOffset is not updated. As a result, logStartOffset is not yet reliable. Once the issue with correctly updating logStartOffset is resolved in issue #744 , we can use logStartOffset here.
Additionally, using 0 versus using logStartOffset does not affect correctness—they both can restore the complete WriterState. The only difference is that using logStartOffset can potentially skip over more segments.
fluss-server/src/main/java/com/alibaba/fluss/server/replica/fetcher/RemoteLeaderEndpoint.java
Outdated
Show resolved
Hide resolved
swuferhong
force-pushed
the
follower-fetch-remote-log
branch
from
e972059 to
cdd67ba
Compare
…LogFetchInfo to rebuild WriterState instead of throwing LogOffsetOutOfRangeException
swuferhong
force-pushed
the
follower-fetch-remote-log
branch
from
cdd67ba to
63d08c0
Compare
Purpose
Linked issue: #703
This pr is aims to change the way if follower fetch log already move to remote, in this situation, we need to return RemoteLogFetchInfo to rebuild WriterState instead of throwing LogOffsetOutOfRangeException. The reason why doing that see: #703
Brief change log
Tests
API and Format
Documentation