feat/ enable partition_json to partition any json

[Coniferish]

Currently partition_json is intended only for deserializing the unstructured JSON outputs/elements and is not included as a file format we accept for partitioning (see here).

The goal of this issue is to make partition_json work for any JSON file (probably similar to how partition_xml works)

[lironrisk]

any news?

[Coniferish]

any news?

Hey @lironrisk !
Apologies for the delay. We've been really busy with the launch of the paid api. @scanny has some preliminary code for this, but it is lower priority than the chunking improvements we have coming up. It has been added to our roadmap for Q2.

[orlandounstructured]

Over 180 days old but keeping open due to addition to roadmap, as mentioned by @Coniferish

[adrianruchti]

Hello unstructured Team. Would be interested in the partition_json.

[apmavrin]

We, so far, have implemented a helper function that parses the json, wraps it into a list and feeds it Unstructured.IO, so it could be parsed with the current version.

utility function

from typing import Dict, Optional

from google.protobuf.struct_pb2 import Struct


def struct_to_dict(struct: Struct, out: Optional[Dict] = None) -> Dict:
    if not out:
        out = {}
    for key, value in struct.items():
        if isinstance(value, Struct):
            out[key] = struct_to_dict(value)
        else:
            out[key] = value
    return out

the main function where we receive json and feed to to Unstructured

# TLDR: Unstructured can't handle a single JSON, but needs a List of JSON
parsed_list = [struct_to_dict(json_content_from_api)]
content_for_unstructured = json.dumps(parsed_list)

[adrianruchti]

thank you. And the content_for_unstructured can then be parsed by the unstructured partition function in the unstructured pipeline?

`runner = LocalRunner(
processor_config=ProcessorConfig(
verbose=True,
output_dir="local-ingest-output",
num_processes=2,
),
read_config=ReadConfig(),
partition_config=PartitionConfig(
partition_by_api=False
),

        connector_config=SimpleLocalConfig(
            input_path="data",
            recursive=True,
        ),
        chunking_config=ChunkingConfig(
            chunk_elements=True
        ),
        
    )
    #      writer=writer,
    #      writer_kwargs={},

     
    # # Run the DropboxRunner
    runner.run()`

[scanny]

@apmavrin @adrianruchti can you say a little more about your use case for this? It's not clear to me yet how a useful JSON partitioner would behave.

In particular:

• A JSON file, assumedly a JSON array or JSON object, is arguably a data file, not a document. How would we we want to discriminate between fields we want to partition and fields we want to ignore or perhaps add to metadata?
• A JSON "document" is often the response from an HTTP API and will often contain a collection of objects, say perhaps a series of blog posts between two dates. How would it make sense to handle that? Ignore "header" information and process array items only?
• How recurrent and stable are your use cases? Is it partitioning the same sort of JSON payload over and over (same schema, different day/data)? Or do you need to partition uncharacterized collections of JSON as encountered among a pile of other documents?

Any help you can give characterizing the use cases will help in developing a spec for something like this. Grateful for whatever help you can provide :)