Skip to content

Fix Downgrade test fails on master #14028

@linliu-code

Description

@linliu-code

Bug Description

What happened:
Flaky test:

Error:  testUpgradeOrDowngrade{HoodieTableVersion, HoodieTableVersion}[4]  Time elapsed: 0.932 s  <<< ERROR!
org.apache.hudi.exception.HoodieException: Failed to downgrade LSM timeline to old archived format
	at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgradeFromLSMTimeline(EightToSevenDowngradeHandler.java:233)
	at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgrade(EightToSevenDowngradeHandler.java:132)
	at org.apache.hudi.table.upgrade.UpgradeDowngrade.downgrade(UpgradeDowngrade.java:328)
	at org.apache.hudi.table.upgrade.UpgradeDowngrade.run(UpgradeDowngrade.java:230)
	at org.apache.hudi.table.upgrade.TestUpgradeDowngrade.testUpgradeOrDowngrade(TestUpgradeDowngrade.java:112)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.util.Optional.ifPresent(Optional.java:159)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:272)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
	at java.util.ArrayList.forEach(ArrayList.java:1259)
Caused by: org.apache.hudi.exception.HoodieCommitException: Failed to archive commits
	at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.flushArchiveEntries(TimelineArchiverV1.java:173)
	at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler$ArchiveEntryFlusher.close(EightToSevenDowngradeHandler.java:274)
	at org.apache.hudi.table.upgrade.EightToSevenDowngradeHandler.downgradeFromLSMTimeline(EightToSevenDowngradeHandler.java:229)
	... 45 more
Caused by: java.lang.NullPointerException
	at org.apache.hudi.avro.HoodieAvroUtils.indexedRecordToBytesStream(HoodieAvroUtils.java:148)
	at org.apache.hudi.avro.HoodieAvroUtils.avroToBytesStream(HoodieAvroUtils.java:144)
	at org.apache.hudi.common.model.HoodieAvroIndexedRecord.getAvroBytes(HoodieAvroIndexedRecord.java:325)
	at org.apache.hudi.common.table.log.block.HoodieAvroDataBlock.serializeRecords(HoodieAvroDataBlock.java:120)
	at org.apache.hudi.common.table.log.block.HoodieDataBlock.getContentBytes(HoodieDataBlock.java:132)
	at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlocks(HoodieLogFormatWriter.java:155)
	at org.apache.hudi.common.table.log.HoodieLogFormatWriter.appendBlock(HoodieLogFormatWriter.java:132)
	at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.writeToFile(TimelineArchiverV1.java:447)
	at org.apache.hudi.client.timeline.versioning.v1.TimelineArchiverV1.flushArchiveEntries(TimelineArchiverV1.java:171)
	... 47 more

What you expected:
No NPE

Steps to reproduce:

  1. Run test TestUpgradeDowngrade.testUpgradeOrDowngrade many times

Environment

Hudi version:
Query engine: (Spark/Flink/Trino etc)
Relevant configs:

Logs and Stack Trace

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    type:bugFor Issues and PRs that fix bugs

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions