Order of XML Properties trigger different behaviors with polymorphic nested objects

[vitorpamplona]

This very specific structure fails to parse a correct XML when the xsi:type property is placed after other fields in the XML. That's the only difference between the test cases.

See below:

import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import com.fasterxml.jackson.module.jaxb.JaxbAnnotationModule;
import org.testng.annotations.Test;
import javax.xml.bind.annotation.*;
import java.io.IOException;
import java.util.List;

import static org.junit.Assert.assertEquals;

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "ModelInfo")
class ModelInfo {
    protected List<TypeInfo> typeInfo;
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "TypeInfo")
@XmlSeeAlso({ ClassInfo.class })
abstract class TypeInfo {

}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "ClassInfo")
class ClassInfo extends TypeInfo {
    protected List<ClassInfoElement> element;
    @XmlAttribute(required = true)
    protected String name;
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "ClassInfoElement")
class ClassInfoElement {
    protected BindingInfo binding;
}

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "BindingInfo")
class BindingInfo {
    protected String description;
}

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
        @JsonSubTypes.Type(value = ClassInfo.class)
})
interface TypeInfoMixIn {}

public class JacksonXMLTests {

    XmlMapper mapper = new XmlMapper().builder()
            .defaultUseWrapper(false)
            .addMixIn(TypeInfo.class, TypeInfoMixIn.class)
            .addModule(new JaxbAnnotationModule())
            .build();

    @Test
    public void testTypeAfterOtherProperties() throws IOException {
        String xml =
                "<modelInfo xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n" +
                "  <typeInfo name=\"MyName\" xsi:type=\"ClassInfo\">\n" +
                "    <element>\n" +
                "      <binding description=\"Test\"/>\n" +
                "    </element>\n" +
                "  </typeInfo>\n" +
                "</modelInfo>";

        ModelInfo m = mapper.readValue(xml, ModelInfo.class);
        assertEquals("MyName", ((ClassInfo)m.typeInfo.get(0)).name);
        assertEquals("Test", ((ClassInfo)m.typeInfo.get(0)).element.get(0).binding.description);
    }

    @Test
    public void testTypeBeforeOtherProperties() throws IOException {
        String xml =
                "<modelInfo xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n" +
                "  <typeInfo xsi:type=\"ClassInfo\" name=\"MyName\">\n" +
                "    <element>\n" +
                "      <binding description=\"Test\"/>\n" +
                "    </element>\n" +
                "  </typeInfo>\n" +
                "</modelInfo>";

        ModelInfo m = mapper.readValue(xml, ModelInfo.class);
        assertEquals("MyName", ((ClassInfo)m.typeInfo.get(0)).name);
        assertEquals("Test", ((ClassInfo)m.typeInfo.get(0)).element.get(0).binding.description);
    }
}

Running this with:

        compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.13.2'
        compile group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-xml', version: '2.13.2'
        compile group: 'com.fasterxml.jackson.core', name: 'jackson-annotations', version: '2.13.2'
        compile group: 'com.fasterxml.jackson.module', name: 'jackson-module-jaxb-annotations', version: '2.13.2'

Yields a pass on testTypeBeforeOtherProperties and a fail on testTypeAfterOtherProperties with the following stack:

Unrecognized field "description" (class org.cqframework.cql.cql2elm.ClassInfoElement), not marked as ignorable (one known property: "binding"])
 at [Source: (StringReader); line: 4, column: 36] (through reference chain: org.cqframework.cql.cql2elm.ModelInfo["typeInfo"]->java.util.ArrayList[0]->org.cqframework.cql.cql2elm.ClassInfo["element"]->java.util.ArrayList[0]->org.cqframework.cql.cql2elm.ClassInfoElement["description"])
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "description" (class org.cqframework.cql.cql2elm.ClassInfoElement), not marked as ignorable (one known property: "binding"])
 at [Source: (StringReader); line: 4, column: 36] (through reference chain: org.cqframework.cql.cql2elm.ModelInfo["typeInfo"]->java.util.ArrayList[0]->org.cqframework.cql.cql2elm.ClassInfo["element"]->java.util.ArrayList[0]->org.cqframework.cql.cql2elm.ClassInfoElement["description"])
	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1127)
	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1989)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1700)
	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1678)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:319)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:313)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:214)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:186)
	at com.fasterxml.jackson.dataformat.xml.deser.WrapperHandlingDeserializer.deserialize(WrapperHandlingDeserializer.java:122)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:144)
	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:110)
	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:357)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:313)
	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176)
	at com.fasterxml.jackson.dataformat.xml.deser.WrapperHandlingDeserializer.deserialize(WrapperHandlingDeserializer.java:122)
	at com.fasterxml.jackson.dataformat.xml.deser.XmlDeserializationContext.readRootValue(XmlDeserializationContext.java:91)
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4674)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3629)
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3597)
	at org.cqframework.cql.cql2elm.JacksonXMLTests.testTypeAfterOtherProperties(JacksonXMLTests.java:72)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
	at org.testng.TestRunner.privateRun(TestRunner.java:767)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:343)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:305)
	at org.testng.SuiteRunner.run(SuiteRunner.java:254)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
	at org.testng.TestNG.run(TestNG.java:1057)
	at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.runTests(TestNGTestClassProcessor.java:141)
	at org.gradle.api.internal.tasks.testing.testng.TestNGTestClassProcessor.stop(TestNGTestClassProcessor.java:90)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy5.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:132)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:412)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at java.base/java.lang.Thread.run(Thread.java:834)

I am not sure if this is a JAXB Module issue or a DataFormat issue. I hope this is the right place.

[cowtowncoder]

Quick note: almost certainly XML dataformat issue, not JAXB.

Thank you for providing tests, that's great!

Would be great to avoid using JAXB annotations if possible (that is, use Jackson ones) but this works as is too.

[vitorpamplona]

Would be great to avoid using JAXB annotations if possible (that is, use Jackson ones) but this works as is too.

I don't have a choice. :(

The class structure + annotations are frozen. That's why I am using that MixIn as well.

[cowtowncoder]

Sorry, what I meant was that only the unit test (reproduction) used different annotations, triggering the same issue. Not that your production code would have to change.
This has to do with project dependencies where JAXB/XML coupling is slightly tricky (we were able to eliminate it for non-test code to avoid forcing JAXB dependency for users who do not need it).
Test dependency is ok if not ideal; just means that when releasing versions there is a dependency if so (JAXB annotations module has to be released first).

[vitorpamplona]

ohhh! makes sense.

Here's the class without JAXB, running into the same problem.

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import org.testng.annotations.Test;
import java.io.IOException;
import java.util.List;

import static org.junit.Assert.assertEquals;

class ModelInfo {
    @JsonProperty List<TypeInfo> typeInfo;
}

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
        @JsonSubTypes.Type(value = ClassInfo.class)
})
abstract class TypeInfo {

}

class ClassInfo extends TypeInfo {
    @JsonProperty List<ClassInfoElement> element;
    @JsonProperty String name;
}

class ClassInfoElement {
    @JsonProperty BindingInfo binding;
}

class BindingInfo {
    @JsonProperty String description;
}

public class JacksonXMLTests {
    XmlMapper mapper = new XmlMapper().builder()
            .defaultUseWrapper(false)
            .build();

    @Test
    public void testTypeAfterOtherProperties() throws IOException {
        String xml =
                "<modelInfo xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n" +
                "  <typeInfo name=\"MyName\" xsi:type=\"ClassInfo\">\n" +
                "    <element>\n" +
                "      <binding description=\"Test\"/>\n" +
                "    </element>\n" +
                "  </typeInfo>\n" +
                "</modelInfo>";

        ModelInfo m = mapper.readValue(xml, ModelInfo.class);
        assertEquals("MyName", ((ClassInfo)m.typeInfo.get(0)).name);
        assertEquals("Test", ((ClassInfo)m.typeInfo.get(0)).element.get(0).binding.description);
    }

    @Test
    public void testTypeBeforeOtherProperties() throws IOException {
        String xml =
                "<modelInfo xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\">\n" +
                "  <typeInfo xsi:type=\"ClassInfo\" name=\"MyName\">\n" +
                "    <element>\n" +
                "      <binding description=\"Test\"/>\n" +
                "    </element>\n" +
                "  </typeInfo>\n" +
                "</modelInfo>";

        ModelInfo m = mapper.readValue(xml, ModelInfo.class);
        assertEquals("MyName", ((ClassInfo)m.typeInfo.get(0)).name);
        assertEquals("Test", ((ClassInfo)m.typeInfo.get(0)).element.get(0).binding.description);
    }
}

[vitorpamplona]

Simplifying further to:

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import org.testng.annotations.Test;
import java.io.IOException;
import java.util.List;

import static org.junit.Assert.assertEquals;

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
        @JsonSubTypes.Type(value = ClassInfo.class)
})
abstract class TypeInfo {

}

class ClassInfo extends TypeInfo {
    @JsonProperty List<ClassInfoElement> element;
    @JsonProperty String name;
}

class ClassInfoElement {
    @JsonProperty BindingInfo binding;
}

class BindingInfo {
    @JsonProperty String description;
}

public class JacksonXMLTests {
    XmlMapper mapper = new XmlMapper().builder().defaultUseWrapper(false).build();

    @Test
    public void testTypeAfterOtherProperties() throws IOException {
        String xml =
                "  <typeInfo name=\"MyName\" type=\"ClassInfo\">\n" +
                "    <element>\n" +
                "      <binding description=\"Test\"/>\n" +
                "    </element>\n" +
                "  </typeInfo>";

        TypeInfo m = mapper.readValue(xml, TypeInfo.class);
        assertEquals("MyName", ((ClassInfo)m).name);
        assertEquals("Test", ((ClassInfo)m).element.get(0).binding.description);
    }

    @Test
    public void testTypeBeforeOtherProperties() throws IOException {
        String xml =
                "  <typeInfo type=\"ClassInfo\" name=\"MyName\">\n" +
                "    <element>\n" +
                "      <binding description=\"Test\"/>\n" +
                "    </element>\n" +
                "  </typeInfo>";

        TypeInfo m = mapper.readValue(xml, TypeInfo.class);
        assertEquals("MyName", ((ClassInfo)m).name);
        assertEquals("Test", ((ClassInfo)m).element.get(0).binding.description);
    }
}

[vitorpamplona]

Some additional learnings:

• Statically resolving the Type makes it pass: mapper.readValue(xml, ClassInfo.class)
• Turning the List<ClassInfoElement> element into ClassInfoElement element also makes it work
• Adding another Child class doesn't change the error: it tries to fill a property of ClassInfoElement and not the immediate parent.

With another child entity:

import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
import com.fasterxml.jackson.dataformat.xml.XmlMapper;
import org.testng.annotations.Test;
import java.io.IOException;
import java.util.List;

import static org.junit.Assert.assertEquals;

@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
@JsonSubTypes({
        @JsonSubTypes.Type(value = ClassInfo.class)
})
abstract class TypeInfo {

}

class ClassInfo extends TypeInfo {
    @JsonProperty List<ClassInfoElement> element;
    @JsonProperty String name;
}

class ClassInfoElement {
    @JsonProperty BindingInfo binding;
}

class BindingInfo {
    @JsonProperty BindingInfo2 binding2;
}

class BindingInfo2 {
    @JsonProperty String description;
}

public class JacksonXMLTests {
    XmlMapper mapper = new XmlMapper().builder().defaultUseWrapper(false).build();

    @Test
    public void testTypeAfterOtherProperties() throws IOException {
        String xml =
                "  <typeInfo name=\"MyName\" type=\"ClassInfo\">\n" +
                "    <element>\n" +
                "      <binding>\n" +
                "         <binding2 description=\"Test\"/>\n" +
                "      </binding>\n" +
                "    </element>\n" +
                "  </typeInfo>";

        TypeInfo m = mapper.readValue(xml, TypeInfo.class);
        assertEquals("MyName", ((ClassInfo)m).name);
        assertEquals("Test", ((ClassInfo)m).element.get(0).binding.binding2.description);
    }

    @Test
    public void testTypeBeforeOtherProperties() throws IOException {
        String xml =
                "  <typeInfo type=\"ClassInfo\" name=\"MyName\">\n" +
                "    <element>\n" +
                "      <binding>\n" +
                "         <binding2 description=\"Test\"/>\n" +
                "      </binding>\n" +
                "    </element>\n" +
                "  </typeInfo>";

        TypeInfo m = mapper.readValue(xml, TypeInfo.class);
        assertEquals("MyName", ((ClassInfo)m).name);
        assertEquals("Test", ((ClassInfo)m).element.get(0).binding.binding2.description);
    }
}

[cowtowncoder]

Ok, I was able to reproduce the failure and added the failing test case as reproduction.

From exception it does look like there is a structural mismatch, which I suspect is due to buffering that is needed when the type id is NOT the first property being found (that is, buffering of non-type-id properties until type id is encountered and handled). That should not matter, of course.

[cowtowncoder]

As to the root cause and possible fix... I think the problem has to do with handling of the "unwrapped" collections; the state is probably not retained by buffering via TokenBuffer (it being format-agnostic).

Jackson 2.13 does allow construction of format-specific TokenBuffers which might help solving the problem (instances are constructed via DeserializationContext and XML module already providers its own implementations for other reasons) but I don't remember if I ever figured out what information would need to be retained and how, unfortunately.
So on plus side I think this would be fixable, but downside is that I probably won't have time to spend on digging deep enough to do so.
But I'll keep the issue in mind in case something pops up.

The way I have used to dig deeper in the past has been to uncomment debug printing in FromXmlParser (and underlying XmlTokenStream) to see how XML elements and attributes get translated into equivalent JsonToken tokens; and how this varies across test cases. It's quite involved so I don't expect anyone else to do that, but it does show exactly what is happening, fwtw.