Ability to customize _link_main values used

[sarahbehrens-cognite]

I am using flatterer to flatten a large quantity of json files that share the same schema to ingest into a database.
Is there a way to customize/specify the key (i.e. '_link' id column) used to link rows between the tables?

By default, flatterer uses a generic incrementing integer starting at zero, but I would like to use the filename or one of the top level fields within the files.
The problem I am encountering is that in the future there will be additional files to ingest to the same set of tables, but the new file ids will also start at zero and there will be duplicate keys. I can edit the csv files after flatterer generates them and create a unique key, but I'd rather avoid altering the data.

[kindly]

@sarahbehrens-cognite

The option to solve this issue was the --id-prefix cli option or id_prefix python option.
However, this was missing from the documentation, so sorry for that.

I have added docs now here: https://docs.flatterer.dev/options.html#id-prefix

I hope this covers your use case, it prepends whatever string you specify in this option to all _link fields ensuring if you put in a different value each time (like a date, or a unique filename) then all _link fields will be unique.

You might also want to look at the evolve option documentation and could consider this if inserting into a postgres or sqlite database. It will automatically add a random string if no id-prefix is specified but recommends adding your own.

https://docs.flatterer.dev/options.html#evolve-tables

[sarahbehrens-cognite]

Thanks for updating the docs. Unfortunately, I think that parameter would only solve the issue when processing the files one at a time. I'm using the array_item_generator data stream method to process a set of files together, so it would be the same prefix for all the files. Is there a way to specify the prefix to use for each file?

[kindly]

@sarahbehrens-cognite if using the streaming method, flatterer would not be aware of any filenames as you are only supplying it each JSON item, and no other context. For this case, for each entire stream, the _link values would start from 0 and not reset every file. So as long as for each stream (each use of array_item_generator) had a unique value for id-prefix then you would not get id conflicts. So something like making id-prefix be datetime.datetime.now().isoformat() + '_' would mean you would never have to worry about the link IDs ever clashing.

[sarahbehrens-cognite]

Thank you for the input, that does provide a good alternative. I think I will continue with the post-processing solution in order to make the id more descriptive to each file and align with other source systems.