Directories ingestion

[MatthMig]

In an old facility, we'll have million of files that could potentially be ingested by SciCat, this lead to several issues to adopt SciCat in a facility. One of the big feature that would ease it is to add the possibility to ingest directories. Therefore we should have strong selectors and also maybe the possibility to have a system to apply multiple schema files on a same data file. The purpose for it being to reduce the difficulty to maintain the ingestor for the facility.

The directories ingestion is a complex feature, I have done a nasty implementation in those commits:

• b60107b
• eaebfd4

For OrigDataBlocks:

• e72aeea
• 86458ea

For selector:

• Related issue: Extended selector possibilities for schemes #172
• 349d745

It lacks of several major features and use cases, as my ingestor takes one month to ingest 4 years of data, we should have strong mechanism to ensure connection with the SciCat backend. We should also severely optimize the whole thing to reduce run time. Also, it would be better to have another service to check what we have ingested or not, because with million of files it is impossible to know if some files have not been ingested by error.

Also, I have met an issue with OrigDataBlock to efficiently add new files to dataFilList, because sometime an OrigDataBlock can reach thousand of files, so it can be long if we do not have an endpoint to let the backend do the insertion of the new file, and doing the appropriate checks.

[YooSunYoung]

Max and I have talked about batch-ingestion for this purpose as well.

Also, I have met an issue with OrigDataBlock to efficiently add new files to dataFilList, because sometime an OrigDataBlock can reach thousand of files, so it can be long if we do not have an endpoint to let the backend do the insertion of the new file, and doing the appropriate checks.

However, using the scicat backend api directly via scicat-ingestor may not be so simple and efficient.

One can consider writing a script using scitacean instead in that case.

[MatthMig]

Max and I have talked about batch-ingestion for this purpose as well.

Also, I have met an issue with OrigDataBlock to efficiently add new files to dataFilList, because sometime an OrigDataBlock can reach thousand of files, so it can be long if we do not have an endpoint to let the backend do the insertion of the new file, and doing the appropriate checks.

However, using the scicat backend api directly via scicat-ingestor may not be so simple and efficient.

One can consider writing a script using scitacean instead in that case.

So the idea will be to redo an ingestor using scicatean ? If so it is greatly limiting ingestor interest to only the online purpose. Because to be fair, for 3 years I have already 1 million files that has been ingested in SciCat, I do not see who will one day want to ingest only a single file.

Directories ingestion is feasible with the ingestor, it requires some work and is not trivial, but I think that there is a great interest to have this possibility, as if one day a facility want to adopt SciCat, like ILL, the missing of this feature in SciCat make the adoption by one facility really harder.

If the idea is more to integrate scicatean in the ingestor to do the directories ingestion, I can look into how to do it