Newtonsoft JSON uses less ram then System.Text.Json on same large JSON

[vasicvuk]

Description

We have a scenario to load a large JSON file into .NET but noticed that the memory usage went high when using System.Text, we also tried to use Newtonsoft.JSON then and it shows way less memory usage for the same 200mb file JSON.

Code for Newtonsoft JSON:

DefaultContractResolver contractResolver = new DefaultContractResolver
{
    NamingStrategy = new CamelCaseNamingStrategy()
};
using StreamReader r = new(stream);
using JsonReader readerData = new JsonTextReader(r);
Newtonsoft.Json.JsonSerializer serializer = new()
{
    ContractResolver = contractResolver,
};
serializer.Converters.Add(new StringEnumConverter());
return serializer.Deserialize<ExecutionModelDTO>(readerData);

Code for System.Text.Json:

var options = new JsonSerializerOptions
{
    PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
    UnmappedMemberHandling = System.Text.Json.Serialization.JsonUnmappedMemberHandling.Skip,
    Converters =
{
    new JsonStringEnumMemberConverter(KebabCaseNamingPolicy),
    }
};
return  await System.Text.Json.JsonSerializer.DeserializeAsync<ExecutionModelDTO>(stream, options)
    .ConfigureAwait(false);

We tried also using Deserialize with ReadToEnd but this causes ram to go up to 1.2gb usage and it is not released.

Configuration

Dotnet: 8.0.116
Both on Windows and Linux Container

[eiriktsarpalis]

Your repro is missing information (including the data source and the DTO it's being deserialized to). That being said, this looks similar to #96559, have you tried running this using .NET 9?

[vasicvuk]

@eiriktsarpalis It is the same with .NET 9 as well—identical behavior.

Hi, this is the DTOs used, but they are very simple. I anonymized models

public class GenericDefinitionDto
{
    public string Field1 { get; set; }
    public string Field2 { get; set; }
    public string Field3 { get; set; }
    public string Field4 { get; set; }
    public string Field5 { get; set; }
    public object Field6 { get; set; }
    public List<string> Field7 { get; set; }
    public List<string> Field8 { get; set; }
}

public class GenericBenefitDto
{
    public string? Field1 { get; set; }
    public string Field2 { get; set; }
    public string Field3 { get; set; }
    public string? Field4 { get; set; }
    public string Field5 { get; set; }
    public object Field6 { get; set; }
    public object Field7 { get; set; }
    public object Field8 { get; set; }
    public decimal Field9 { get; set; }
    public object Field10 { get; set; }
    public bool Field11 { get; set; }
    public decimal? Field12 { get; set; }
    public decimal? Field13 { get; set; }
}

public class GenericConditionDto
{
    public string Field1 { get; set; }
    public string Field2 { get; set; }
    public string Field3 { get; set; }
    public object Field4 { get; set; }
    public string Field5 { get; set; }
    public object Field6 { get; set; }
    public decimal? Field7 { get; set; }
    public decimal? Field8 { get; set; }
    public decimal? Field9 { get; set; }
    public long Field10 { get; set; }
}

public class GenericVariationDto
{
    public string Field1 { get; set; }
    public string Field2 { get; set; }
    public string Field3 { get; set; }
    public string Field4 { get; set; }
    public object Field5 { get; set; }
    public object Field6 { get; set; }
    public object Field7 { get; set; }
    public decimal? Field8 { get; set; }
    public bool Field9 { get; set; }
    public decimal? Field10 { get; set; }
    public decimal? Field11 { get; set; }
    public string Field12 { get; set; }
}

public class GenericEligibilityDto
{
    public string Field1 { get; set; }
    public object Field2 { get; set; }
    public IEnumerable<string> Field3 { get; set; }
}

public class ExecutionModelDTO
{
    public string Field1 { get; set; }
    public IEnumerable<GenericDefinitionDto> Field2 { get; set; } = [];
    public IEnumerable<GenericConditionDto> Field3 { get; set; } = [];
    public IEnumerable<GenericVariationDto> Field4 { get; set; } = [];
    public IEnumerable<GenericBenefitDto> Field5 { get; set; } = [];
    public IEnumerable<GenericEligibilityDto> Field6 { get; set; } = [];
}

I do the deserialization of ExecutionModelDTO. All non-defined types are just simple enumerations.

[eiriktsarpalis]

And what data did you use to run both apps? We'd need a full repro of both to evaluate.

[vasicvuk]

I would have to create such mock with anonymized data since I cannot share real data here, I will update the issue once i do it.

[vasicvuk]

I've build a similar repro (not the same):

Newtonsoft

System.Text.Json

it is larger definitel,y but seems to depend on the model. Code for test:

// See https://aka.ms/new-console-template for more information
using Newtonsoft.Json.Converters;
using Newtonsoft.Json.Serialization;
using Newtonsoft.Json;
using ConsoleApp2;
using System.Text.Json;

public class Program
{

    public static async Task Main(string[] args)
    {
        var zipFilePath = "performance-testing-execution-model.zip";
        var jsonFileName = "execution-model.json";
        using var zip = System.IO.Compression.ZipFile.OpenRead(zipFilePath);
        var entry = zip.GetEntry(jsonFileName) ?? throw new FileNotFoundException($"Could not find {jsonFileName} in {zipFilePath}");
        using var stream = entry.Open();
        await ParseFromStreamSystemText(stream);

        Console.ReadLine();
    }
    public static ExecutionModelDTO ParseFromStream(Stream stream)
    {
        DefaultContractResolver contractResolver = new()
        {
            NamingStrategy = new CamelCaseNamingStrategy()
        };
        using StreamReader r = new(stream);
        using JsonReader readerData = new JsonTextReader(r);
        Newtonsoft.Json.JsonSerializer serializer = new()
        {
            ContractResolver = contractResolver,
        };
        serializer.Converters.Add(new StringEnumConverter());
        return serializer.Deserialize<ExecutionModelDTO>(readerData);

    }
    public static async Task<ExecutionModelDTO> ParseFromStreamSystemText(Stream stream)
    {
        var options = new JsonSerializerOptions
        {
            PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
            UnmappedMemberHandling = System.Text.Json.Serialization.JsonUnmappedMemberHandling.Skip,
            Converters =
            {
             }
        };
        return await System.Text.Json.JsonSerializer.DeserializeAsync<ExecutionModelDTO>(stream, options)
            .ConfigureAwait(false);

    }
}

The ZIP file used attached:
https://we.tl/t-bbPdKlcvpm

[vasicvuk]

.NET 9

Newtonsoft:

System Text:

[vasicvuk]

Hi @eiriktsarpalis, someone should download the file, because it will be removed from WeTransfer.

[eiriktsarpalis]

Hi, I don't have time to take a deeper look at this right now. If the download link is due to expire, please consider hosting it elsewhere or otherwise share code used for generating the data. Any changes you could make to narrow down the reproduction to something as simple as possible would also help.

Thanks

[vasicvuk]

Okey here is a generator in JS:

// generate.js

const fs = require('fs');
const path = require('path');

// Helpers
const randomEnum = (arr) => arr[Math.floor(Math.random() * arr.length)];
const randomString = () => Math.random().toString(36).substring(2, 10);
const randomDecimal = () => parseFloat((Math.random() * 100).toFixed(2));
const randomInt = (max) => Math.floor(Math.random() * max);

// Sample enum values (adjust as needed)
const attributeCategoryKinds = ['CategoryA','CategoryB','CategoryC'];
const conditionTypes         = ['ConditionX','ConditionY','ConditionZ'];
const booleanExpressions     = ['x > 5','y < z','a && b'];
const benefitTypes           = ['BenefitA','BenefitB','BenefitC'];
const adjustmentTypes        = ['Add','Subtract','Multiply'];
const priceTypes             = ['Fixed','Percentage'];
const planTypes              = ['PlanBasic','PlanPremium','PlanEnterprise'];

// Output file
const outPath = path.join(__dirname, 'execution-model.json');
const ws = fs.createWriteStream(outPath, { encoding: 'utf8' });

// Begin root object
ws.write('{');
ws.write(`"field1":"${randomString()}",`);

// GenericDefinitionDto[] (1 000 items)
ws.write(`"field2":[`);
for (let i = 0; i < 1000; i++) {
  const dto = {
    field1: randomString(),
    field2: randomString(),
    field3: randomString(),
    field4: randomString(),
    field5: randomString(),
    field6: randomEnum(attributeCategoryKinds),
    field7: [ randomString(), randomString(), randomString() ],
    field8: [ randomString(), randomString() ]
  };
  ws.write(JSON.stringify(dto) + (i < 999 ? ',' : ''));
}
ws.write('],');

// GenericConditionDto[] (1 000 items)
ws.write(`"field3":[`);
for (let i = 0; i < 1000; i++) {
  const maybePriceType = Math.random() < 0.8 ? randomEnum(priceTypes) : null;
  const dto = {
    field1: randomString(),
    field2: randomString(),
    field3: randomString(),
    field4: randomEnum(conditionTypes),
    field5: randomString(),
    field6: maybePriceType,
    field7: Math.random() < 0.7 ? randomDecimal() : null,
    field8: Math.random() < 0.7 ? randomDecimal() : null,
    field9: Math.random() < 0.7 ? randomDecimal() : null,
    field10: randomInt(1e6)
  };
  ws.write(JSON.stringify(dto) + (i < 999 ? ',' : ''));
}
ws.write('],');

// GenericVariationDto[] (1 000 items)
ws.write(`"field4":[`);
for (let i = 0; i < 1000; i++) {
  const dto = {
    field1: randomString(),
    field2: randomString(),
    field3: randomString(),
    field4: randomString(),
    field5: randomEnum(booleanExpressions),
    field6: Math.random() < 0.5 ? randomEnum(adjustmentTypes) : null,
    field7: Math.random() < 0.5 ? randomEnum(priceTypes) : null,
    field8: Math.random() < 0.5 ? randomDecimal() : null,
    field9: Math.random() < 0.5,
    field10: Math.random() < 0.5 ? randomDecimal() : null,
    field11: Math.random() < 0.5 ? randomDecimal() : null,
    field12: randomString()
  };
  ws.write(JSON.stringify(dto) + (i < 999 ? ',' : ''));
}
ws.write('],');

// GenericBenefitDto[] (500 000 items)
ws.write(`"field5":[`);
for (let i = 0; i < 1000000; i++) {
  const dto = {
    field1: Math.random() < 0.3 ? randomString() : null,
    field2: randomString(),
    field3: randomString(),
    field4: Math.random() < 0.3 ? randomString() : null,
    field5: randomString(),
    field6: randomEnum(booleanExpressions),
    field7: randomEnum(benefitTypes),
    field8: randomEnum(adjustmentTypes),
    field9: randomDecimal(),
    field10: randomEnum(priceTypes),
    field11: Math.random() < 0.5,
    field12: Math.random() < 0.5 ? randomDecimal() : null,
    field13: Math.random() < 0.5 ? randomDecimal() : null
  };
  ws.write(JSON.stringify(dto) + (i < 999999 ? ',' : ''));
}
ws.write('],');

// GenericEligibilityDto[] (20 000 items)
ws.write(`"field6":[`);
for (let i = 0; i < 20000; i++) {
  const dto = {
    field1: randomString(),
    field2: randomEnum(planTypes),
    field3: [ randomString(), randomString(), randomString() ]
  };
  ws.write(JSON.stringify(dto) + (i < 19999 ? ',' : ''));
}
ws.write(']');

// End root object
ws.write('}');
ws.end(() => console.log(`Wrote ${outPath}`));