[Java] Class with missing field failed to deserialize on new version

[orisgarno]

Search before asking

• I had searched in the issues and found no similar issues.

Version

java v0.9.0

Component(s)

Java

Minimal reproduce step

BaseFury s = FuryFury.builder()
.withLanguage(Language.JAVA)
.withRefTracking(true)
.withIntCompressed(true)
.withLongCompressed(true)
.withStringCompressed(true)
.withCompatibleMode(CompatibleMode.COMPATIBLE)
.requireClassRegistration(false)
.withAsyncCompilation(true)
.serializeEnumByName(true)
.buildThreadSafeFury()

public class PrivateFliedClassNumberOne {
  private boolean privateBoolean = true;
  private int privateInt = 10;
  private String privateString = "notNull";
  private Map<String, String> privateMap = Map.of("a", "b");
  private List<String> privateList = List.of("l");
  private PrivateFieldSubClass privateFieldSubClass = new PrivateFieldSubClass();
}

public class PrivateFliedClassNumberTwoWithMissingField {

  private Map<String, String> privateMap = Map.of("a", "b");
  private int privateInt = 10;
  private PrivateFieldSubClass privateFieldSubClass = new PrivateFieldSubClass();
  private List<String> privateList = List.of("l");
  private boolean privateBoolean = true;
}

code to repro

    PrivateFliedClassNumberOne privateField = new PrivateFliedClassNumberOne();
    byte[] serialized = s.serializeJavaObject(privateField);


    PrivateFliedClassNumberTwoWithMissingField privateField2 = s.deserializeJavaObject(
        serialized, 
        PrivateFliedClassNumberTwoWithMissingField.class
    );

What did you expect to see?

Should be able to deserialize successfully

What did you see instead?

exception

Deserialize failed, read objects are: [PrivateFliedClassNumberTwoWithMissingField{privateMap={a=b}, privateInt=10, privateFieldSubClass=true, privateList=[l], privateBoolean=true}]

	at org.apache.fury.util.ExceptionUtils.handleReadFailed(ExceptionUtils.java:63)
	at org.apache.fury.Fury.deserializeJavaObject(Fury.java:1124)
	at org.apache.fury.Fury.deserializeJavaObject(Fury.java:1101)
	at org.apache.fury.ThreadLocalFury.deserializeJavaObject(ThreadLocalFury.java:202)
	at serde.SerdeTestBase.testBasicPrivateField_diffClassScrambledField(SerdeTestBase.java:89)
	at serde.SerdeTestBase.runAll(SerdeTestBase.java:35)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
	at org.testng.TestRunner.privateRun(TestRunner.java:756)
	at org.testng.TestRunner.run(TestRunner.java:610)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:387)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:382)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1293)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1218)
	at org.testng.TestNG.runSuites(TestNG.java:1133)
	at org.testng.TestNG.run(TestNG.java:1104)
	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:65)
	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:105)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 111 out of bounds for length 6
	at org.apache.fury.collection.ObjectArray.set(ObjectArray.java:58)
	at org.apache.fury.resolver.MapRefResolver.setReadObject(MapRefResolver.java:209)
	at org.apache.fury.Fury.readRef(Fury.java:875)
	at org.apache.fury.serializer.ObjectSerializer.readContainerFieldValue(ObjectSerializer.java:385)
	at org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:318)
	at org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:246)
	at org.apache.fury.Fury.readDataInternal(Fury.java:959)
	at org.apache.fury.Fury.deserializeJavaObject(Fury.java:1118)
	... 30 more

Anything Else?

this is the first time i encounter this issue.
i was using fury v0.5.1 and doesnt have this kind of problem. i am also tried to repro this on 0.5.1, but its not happening on that version.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[chaokunyang]

Hi @orisgarno , thanks for reporting this bug.

Could you use serialize/deserialize API instead. serializeJavaObject deserializeJavaObject will skip write class meta, so the dserialization side don't know how to handle schema inconsistency.

We should write class meta in serializeJavaObject deserializeJavaObject if CompatibleMode is used. Would you like to submit a PR?

[orisgarno]

Thank you to take a look at this issue @chaokunyang.
sure, will check what can i do.

Wondering though, does it mean that the class meta is already implemented on v0.5.1, but removed on v0.9.0?
Since I had no issue when using v0.5.1.

[chaokunyang]

0.5.1 didn't enable meta share mode, so it go to another schema compatible protocol, which write type meta in a KV format. This format is not efficient, and has been replaced by scoped meta share mode in later versions. But we forget to write shared type meta for root class for serializeJavaObject API

[orisgarno]

after further read, seems serializeJavaObject API already write class def. which meta are you refering to? maybe if you can help to share some context or pointer it will be much help

• fury/java/fury-core/src/main/java/org/apache/fury/Fury.java

  Line 1072 in b43d521

  classResolver.writeClassDefs(buffer);

• fury/java/fury-core/src/main/java/org/apache/fury/Fury.java

  Line 1078 in b43d521

  writeData(buffer, classInfo, obj);

thank you

[orisgarno]

Hi @chaokunyang.
I've tried several approaches but am currently stuck.
I would greatly appreciate any pointers or suggestions you might have.
Thanks

[chaokunyang]

Hi @orisgarno which fury version are you using when you try those approaches?

[orisgarno]

@chaokunyang the working one was v0.5.1, the failing one is v0.9.0.

[chaokunyang]

after further read, seems serializeJavaObject API already write class def. which meta are you refering to? maybe if you can help to share some context or pointer it will be much help

• fury/java/fury-core/src/main/java/org/apache/fury/Fury.java

  Line 1072 in b43d521

  classResolver.writeClassDefs(buffer);

• fury/java/fury-core/src/main/java/org/apache/fury/Fury.java

  Line 1078 in b43d521

  writeData(buffer, classInfo, obj);

thank you

It didn't write class def, the correct code should looks like following:

  public void serializeJavaObject(MemoryBuffer buffer, Object obj) {
    try {
      jitContext.lock();
      if (depth != 0) {
        throwDepthSerializationException();
      }
      if (config.isMetaShareEnabled()) {
        int startOffset = buffer.writerIndex();
        buffer.writeInt32(-1); // preserve 4-byte for meta start offsets.
        if (!refResolver.writeRefOrNull(buffer, obj)) {
          ClassInfo classInfo = classResolver.getOrUpdateClassInfo(obj.getClass());
          classResolver.writeClass(buffer, classInfo);
          writeData(buffer, classInfo, obj);

  public <T> T deserializeJavaObject(MemoryBuffer buffer, Class<T> cls) {
    try {
      jitContext.lock();
      if (depth != 0) {
        throwDepthDeserializationException();
      }
      if (shareMeta) {
        readClassDefs(buffer);
      }
      T obj;
      int nextReadRefId = refResolver.tryPreserveRefId(buffer);
      if (nextReadRefId >= NOT_NULL_VALUE_FLAG) {
        ClassInfo classInfo;
        if (shareMeta) {
          classInfo = classResolver.readClassInfo(buffer);
        } else {
          classInfo = classResolver.getClassInfo(cls);
        }
        obj = (T) readDataInternal(buffer, classInfo);
        return obj;
      } else {
        return null;
      }

And when deserializing, you must register class by id to setup class mapping:

  static BaseFury s = Fury.builder()
    .withRefTracking(true)
    .withCompatibleMode(CompatibleMode.COMPATIBLE)
    .requireClassRegistration(false)
    .serializeEnumByName(true)
    .buildThreadSafeFury();
  static BaseFury s1 = Fury.builder()
    .withRefTracking(true)
    .withCompatibleMode(CompatibleMode.COMPATIBLE)
    .requireClassRegistration(false)
    .serializeEnumByName(true)
    .buildThreadSafeFury();

  public static class PrivateFliedClassNumberOne {
    private boolean privateBoolean = true;
    private int privateInt = 10;
    private String privateString = "notNull";
    private Map<String, String> privateMap = ofHashMap("a", "b");
    private List<String> privateList = ofArrayList("l");
    private PrivateFieldSubClass privateFieldSubClass = new PrivateFieldSubClass();
  }

  public static class PrivateFliedClassNumberTwoWithMissingField {

    private Map<String, String> privateMap = ofHashMap("a", "b");
    private int privateInt = 10;
    private PrivateFieldSubClass privateFieldSubClass = new PrivateFieldSubClass();
    private List<String> privateList = ofArrayList("l");
    private boolean privateBoolean = true;
  }

  private static class PrivateFieldSubClass{}

  @Test
  public void test() {
    PrivateFliedClassNumberOne privateField = new PrivateFliedClassNumberOne();
    s.register(PrivateFliedClassNumberOne.class);
    byte[] serialized = s.serializeJavaObject(privateField);

    s1.register(PrivateFliedClassNumberTwoWithMissingField.class);
    PrivateFliedClassNumberTwoWithMissingField privateField2 = s1.deserializeJavaObject(
      serialized,
      PrivateFliedClassNumberTwoWithMissingField.class
    );
  }

[chaokunyang]

If you want to avoid class registeration, this will need more work, you need to extend org.apache.fury.meta.ClassDef and replace the original class with new passed class for all fields and the class. Then you need to extend ClassResolver with a new readClassInfoWithMetaShare method with passed deserialized type

[orisgarno]

Thanks for the code and the example @chaokunyang .
After adding some extensive unit test(still on my local), it works. will make a pr for the changes and also the documentation.
Wondering though, do we need to register only the top most parent class?

not so sure for the solution to avoid class registration. seems quite some works. if it doesn't affect the performance, will try to take a look.

[chaokunyang]

Looking forward to your PR.

For registration, if nested field classes are not registered, fury will skip those fields when there is a inconsistency. Currently fury will skip all incompatible fields if theiir declared types are not compatible. There is a PR #1870 which is working on this