s3 file reader cannot process JSON files over 512MB

[ciorg]

I think very large JSON files are not ideal for many reasons, but that being said there may be methods we could implement to accommodate large JSON files if we need to.

The 512MB string size limit is a hard limit in Node.js and when our processor reads the JSON file it accumulates all the chunks into a string to then parse to JSON, but with large files I get this error:

"error": "TSError: Cannot create a string longer than 0x1fffffe8 characters\n 

We could try something like - https://www.npmjs.com/package/stream-json to stream process json files, or there are other streaming options.

This tickets lists some more options - https://stackoverflow.com/questions/68230031/cannot-create-a-string-longer-than-0x1fffffe8-characters-in-json-parse.

At the very least we should add a better error explanation.