Add possibility to listen on a directory

Add a parameter to DirReader.java that enables the Reader to listen without ending if a new file is put to the directory. If a new file comes in the file is read and passed on, just like the Reader works on files it finds in the directory right now.

Would be useful for e.g. https://github.com/hbz/rpb/ where at the moment a new Flux instance is loaded and executed, i.e. a Fix and all lookup-tables coming with it is loaded anew. If this happens (atm sometimes) 5 times a minute => 5 whole Flux instances are started in parallel. i.e. i.a. all maps are loaded 5 times in parallel . With the "directory-listener" option the Flux and Fix would also be loaded once.

I am not sure if that is conceptionally possible since the transformation result is always written in the same output/file except if you are adding a batch-reset. Is the output file then usable for further processing in RPB?

If that would be so: how is it overwriting prevented atm?

Overwriting is by default. I think there is no way to do it, but introducing a new variable for the filename.

but introducing a new variable for the filename.

So we will do this.
However, a bit offtopic:
So, as I wrote: there are sometimes multiple documents catalogued. I think that that can happen the same time by different people. What happens if, say, by coincidence, two records are catalogued nearly at the same time? But let us discuss this in the rpb repo - if my interpretation is correct and may result in the loss of documents.

@dr0i

@dr0i isn't this the issue that you wanted to implement next? can we move this to ready?

Yep, even Working. Moved it there now.

@fsteeg

We discussed this internally on Friday and today with @fsteeg & @Phu2: With regard to the solution to RPB's high RAM consumption, @fsteeg sees simpler options for addressing the problem. He will open an issue in the RPB Jira for the problem. As a result, this issue might not have so much priority anymore. I am moving this back to ready for re-assessment.

@fsteeg

@fsteeg sees simpler options for addressing the problem. He will open an issue in the RPB Jira

Opened https://jira.hbz-nrw.de/browse/RPB-350. The basic idea is to set up a custom Fix function (metafacture/metafacture-fix@40eee0b) to do something like the javamap from Metamorph (1054fc7), and caching those Java maps in the RPB apps.

I think this is simpler than the listener proposed here, but maybe I just don't fully understand the idea. It seems to me that the listener would introduce a very different way of running a Flux, requiring some kind of demon mode to keep the process running in the background or something like that? With the javamap approach we'd use the running RPB processes to manage the maps instead.