Skip to content

[BUG]: Extension method JsonSerDe SortProperties removes arrays with strings from metadata during serialization  #1082

Open
@TrondRo

Description

@TrondRo

Describe the bug
If you include an array of values in a StructField metadata instance they will not serialize when .Json is called on the root StructType. They will however show on the fields JsonValue.

To Reproduce

Steps to reproduce the behavior:

var values = new[] { "Test1", "Test2" };
var metadata = new JObject(new JProperty("allowedValues", values));
var field = new StructField("TestField", new StringType(), true, metadata);
var fields = new StructType(new[] { field });
var json = fields.Json;

json will contain {"fields":[{"metadata":{"allowedValues":[]},"name":"TestField","nullable":true,"type":"string"}],"type":"struct"}

Expected behavior
I would expect allowedValues in the example above to contain Test1 and Test2.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]
    Microsoft.Spark 2.1.1

Additional context
The code that sorts the array in the serialization when you call DataType.Json does not take into account if the item is a simple string and thus returns an empty array:

foreach (JToken item in jArray)
{
if (item is JObject elem)
{
sortedJArray.Add(elem.SortProperties());
}
else if (item is JArray arrayElem)
{
sortedJArray.Add(arrayElem.SortProperties());
}
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions