Support for multi-threads flat file item reader [BATCH-1613]

[spring-projects-issues]

Stéphane Nicoll opened BATCH-1613 and commented

The attached classes implement a scenario where a singe file is read by a partition. It's limited to one item = one line (only the method that does count the number of items in the input file though).

Attached also is a sample Job configuration file demoing the functionality (not runnable as is though)

---

Affects: 2.1.0

Reference URL: http://forum.springsource.org/showthread.php?t=93151

Attachments:

• sb-multi-threads-reader.zip (8.59 kB)

Referenced from: pull request #147

2 votes, 4 watchers

[spring-projects-issues]

Stéphane Nicoll commented

Hi Dave, do you consider adding this to the standard release?

[spring-projects-issues]

Dave Syer commented

I think it would make a good feature for 2.2 (starting work soon). Optional: can you move your partitioner into an org.sfw package and submit as a pull request on Github?

[spring-projects-issues]

Stéphane Nicoll commented

sure, will do.

[spring-projects-issues]

Sergey Shcherbakov commented

Hello all,
We have faced a situation at the customer recently where a multi-thread flat file reader would improve importing a huge file.
Here is a somewhat improved version of the code attached by Stephane:
#147

[spring-projects-issues]

Dominik Bartholdi commented

is there still any discussion ongoing to this topic?

[spring-projects-issues]

Michael Minella commented

Sergey Shcherbakov's pull request is still open and I know people have used the code. The reason we haven't merged it is that it doesn't provide universal improvement and we haven't had the time to nail down exactly what the conditions are that it does improve performance (to be able to document it). The fear is that if we merge it without either of those conditions met, that users will assume more threads == better which we know isn't the case in all situations. Dominik Bartholdi, is there something related to the topic you can add?

[spring-projects-issues]

Dominik Bartholdi commented

The reason why I was looking for this is the fact that on some platforms its quite hard to split a file (http://stackoverflow.com/a/25022841/187950) to be able to use something like the org.springframework.batch.core.partition.support.MultiResourcePartitioner. e.g. Google AppEngine (File Api disabled.) or when you run in Cloudfoundry you need to safe the files somewhere like S3, in Cloudfoundry you only have limited space available to write files and when you have to deal with very large files (even just temporarily) you get into problems.

Where the later even opens another problem: how can I use org.springframework.batch.core.partition.support.MultiResourcePartitioner to read files from S3? (http://stackoverflow.com/questions/27797133/spring-batch-to-read-a-file-from-s3)

So in my case I would not talk about to many threads/partitions, but I don't even get the option to play with it because of the current limitations.

But maybe I'm just to confused about the difference of ItemReader and ItemStream, because reading is not really my bottleneck, but processing.

[spring-projects-issues]

Michael Minella commented

Dominik Bartholdi, your last point is key. This issue is really only about getting the bytes of a file off the disk faster, which is not your issue. Feel free to post a question on StackOverflow and tag it with spring-batch. I monitor that tag and we can have a discussion about your exact issue there.