Direct online ingestion

[MatthMig]

Maybe we should have the possibility to directly ingest online data, instead of passing by the file management system. This is relevant at ILL because we have the possibility at the end of acquisition to send a message to scicat ingestor with the data. This is also relevant because we'd like to put as less as possible of stress on the file management system, so this could open the possibility to distribute the workload by making the instruments sending the data directly to the ingestor service, therefore the file management system would have no extra workload.

I have made a suggestion of implementation in this commit: a2aa789

[YooSunYoung]

Assuming file manager system our file writer
and the message is writing done message,

The file done writing message doesn't have to come from file writer.
ScicatIngestor does not care where the messages are from.

So I'm a bit lost here what you mean by the directly ingest online data.
Can you please elaborate a bit more...?

[MatthMig]

Actually when I was talking about the file management system, I was referring to the facility's system to manage access to its data , all of this is completely exterior to SciCat. The thing is that currently when ingesting data online in SciCat, what is actually done is that it receive a message with a path, and the ingestor will go get this data in the facility's file system.

Therefore we have 2 works done on the facility's file management system: first a file stored by the one that produce (for ILL it is the instrument), and second the ingestor will go into the facility's file management system to ingest the file's metadata.

What I say it that it can be too much on our facility's file management system, and so I suggest to not do the second work.
To avoid doing the second data, I suggest that we directly send the data in the message that the one that produce send to the ingestor. This way, SciCat online ingestion will put less stress on the facility's file management system.

I think that it can be too much, because I know that the offline ingestion of whole directories raised alerts at our IT, but at least offline ingestion of previous files is temporary, the online ingestor should ideally run in the long term, therefore I think it would be better to reduce as much as possible the traffic it generates on the facility services.

[nitrosx]

IMHO we could do the following:

• refactor the online ingestor to pass the message received to the offline ingestor as it is.
• refactor the offline ingestor so it can extract variables values from the message and not from the matching file

This should achieve the intended effect to reduce the number of accesses to the storage.
We should be good, as long as:

• the metadata passed in the message is not extensive
• these new features are configurable and backward compatible

Thoughts?

[MatthMig]

IMHO we could do the following:

• refactor the online ingestor to pass the message received to the offline ingestor as it is.
• refactor the offline ingestor so it can extract variables values from the message and not from the matching file

This should achieve the intended effect to reduce the number of accesses to the storage. We should be good, as long as:

• the metadata passed in the message is not extensive
• these new features are configurable and backward compatible

Thoughts?

That is more or less what I had in mind, just I think that as the variable values extraction from a message is only relevant for the online ingestor if the mode is activated, I think maybe we should decouple in another file the ingestion process itself, and the online and offline ingestor will in fact call the ingestor module script by passing in parameter the information needed (the medata), therefore we could arrange things like that:

Online ingestion mode with message containing variable values:

Online ingestion mode with message containing path only:

Offline ingestion

Therefore we'll have multiple flexible entry points leading to a single strict module, making it easy to maintain each mode.
Additionally it already open the door for implementation of directories ingestion.

In the end, offline and online ingestor module themselves will be entry points that will extract needed information and then pass it to a common module, the core of the ingestor.

[YooSunYoung]

One more type of source in the variables in the schema definition file could fix this issue I think.

For example, we can use MSG source type in the variables definition instead of NXS.

"variables": {
    "pid": {
      "source": "NXS",
      "path": "/entry/entry_identifier_uuid",
      "value_type": "string"
    }
}

will be

"variables": {
    "pid": {
      "source": "MSG",
      "path": "pid",
      "value_type": "string"
    }
}

In this way, the ingestor process doesn't have to know if it is going to open a file or not.
(It still needs a fix that it should not open a file if not necessary)

I'm assuming the message, MSG source type, will be a nested dictionary.

[YooSunYoung]

I'm still not entirely sure why it is too much of workload for file management system since I don't know how ILL is operating file servers though...
Ingestor is meant to extract some statistics from files, such as minimum value of certain fields.
So regardless of adding MSG source type, it'll be nice to have easier access to the file system from the ingestor.

[MatthMig]

I'm still not entirely sure why it is too much of workload for file management system since I don't know how ILL is operating file servers though... Ingestor is meant to extract some statistics from files, such as minimum value of certain fields. So regardless of adding MSG source type, it'll be nice to have easier access to the file system from the ingestor.

Basically when opening a file on the ILL's file management system, the file management system is locking it for us, and if the ingestor system is too active it could be rate limited by the file management system, or it could slow down access for users to files. This is why if we have the possibility to not charge it more, it is nice.

Globally, I do not know exactly what is the capacity of the ILL's management system, I just know that for example an access every 200 ms begins to be concerning for our IT service. So with 52 instruments and the possibility to go as fast as 1 second per acquisition, we can see that in the worst case it can put the file management system at risk (in practice we should probably never reach acquisition rate as fast as it would cause problem, especially knowing that the average acquisition runs during 3 minutes, and it is unlikely that all the instruments do the acquisition concurrently).

To be fair, this issue is more a long term one to anticipate potential issues in the future and give the possibility to further optimizations by avoiding to having to scale directly our facility files management system.

For short term, can you please take a look at:
PR: #178
Issues:

• Automatic creation of sample, proposal, instrument #174
• Directories ingestion #173
• Extended selector possibilities for schemes #172