How to process JSON data with 500,000+ lines

[RahulKesharwani353]

Hey Folks and @larsga,

I got an use case there I need to transform a json file with more than 500,000 lOC, It is consuming lot of memory.
Can anyone please suggest some way to optimize that.

Thanks in advance

[larsga]

The problem here is almost certainly the Jackson representation of the input (and possibly the output). JSLT itself probably isn't consuming much memory.

This means there isn't a whole lot of simple solutions. I have thought of abandoning Jackson in favour of a more compact representation, but since this doesn't seem to be a very common requirement I haven't gone further than just thinking.

Is the top level of your JSON an array? If so, you can use the streaming parser, then, for each object in the array, push the events into a JSON builder. That way you can build each sub-object, transform it, serialize that, then move on, and never actually keep the full representation of the JSON in memory.