Adjusting number of splits of data set in biopig

[saranpons3]

Dear Biopig developers,
I have run Biopig in a hadoop cluster of 20 nodes(each node is of 8 cores) on the data set "F Vesca"(data set size is 10.2GB). I have set split size = 32 MB while uploading data set to HDFS. I wanted to have 160 splits for my data set. So I have chosen 32 MB for each split. But When I ran Kmer counter of Biopig on my data set in the cluster, the number of splits generated is only 41. So I would like to know that how to adjust split size in Biopig. It would be helpful if you give me the answer. Thanks in advance.

[xmeng]

Hello,
I am sorry to know the problem. Please send us the exact commands you used to run the kmer counter app. We will take a look from there.

Thanks,
xmeng

[saranpons3]

Hello xmeng, Thanks for replying. The following is the command i used to run biopig's kmer counter
saravanan@node81:~/biopig$ pig apps/kmerCount.pig

register /home/hduser/biopig/lib/biopig-job.jar A = load '/srr_fasta_ip/srr_fasta.fasta' using gov.jgi.meta.pig.storage.FastaStorage as (id: chararray, d: int, seq: bytearray); B = foreach A generate id, FLATTEN(gov.jgi.meta.pig.eval.KmerGenerator(seq, 25)) as (kmer:bytearray); C = group B by kmer; D = foreach C generate group, COUNT(B); store D into '/srr_biopig_out';

The above code I have used to run kmercount of biopig. If the data type of "seq" field is chararray then kmercount was not working. so I have changed to bytearray data type then it worked.

Please let me know how to change number of splits and no of reducers in biopig.

Thanks, Saravanan

[xmeng]

Hi Saravanan,

In Pig, map parallelism is determined by the input file, one map for each HDFS block.

Pig sets the number of reducers using a heuristic based on the size of the input data. You may use parallel features to change the the number of reducers. However, the parallel features only affect the number of reduce tasks.

For example, you may add "SET default_parallel 100" to set the number of reducers at the beginning of your Pig script. For the above case, it sets 100 reducers. Alternatively, use the PARALLEL clause to set the number of reducers at the operator level. Details can be found at https://pig.apache.org/docs/r0.11.1/perf.html

There are many factors which affect Apache Pig job performance in Apache Hadoop from hardware to software. It may take some iteration to achieve tangible improvement.

I hope information above is helpful.

Regards,
xmeng

[saranpons3]

Hi Xmeng,
Actually, I divided 10.2 GB into blocks where each block would be of size 32 MB. In my case, HDFS block size is 32 MB. I had 160 cores available in my cluster so I wanted to use all of them for map taks so divided data set into blocks of 32 MB each. So I will get 160 splits. but actual number of splits created is only 41 when i started running biopig. I want to know that why less number of splits were created.

[xmeng]

Hi Saravanan,
It looks like the HDFS block configuration is 256MB at your side if you get 41 splits of the input file.
To control the mappers in Pig, you may try to specify the input split size in your code.
e.g.,
SET mapreduce.input.fileinputformat.split.minsize 33554432
Thus in order to set the value to 32MB, you will specify 33554432 as the value for this property in Hadoop 2.x.

Please give a try and let us know if it works.
Regards,
xmeng

[saranpons3]

Hi Xmeng, Thanks for the suggestions and I tried with the configurations which you have mentioned in your answer and it is working as it is expected. But now I have one more question is that downloaded kmerCount program from github has chararray as a data type for "seq". But when I ran with chararray data type for "seq" field it did not work. But kmerCount runs when bytearray data type is chosen as the data type for "seq" field. Using bytearray is correct instead of chararray? When I looked into the output of kmerCount, all kmers are in encoded format. Have you written kmerCount to write the output in encoded form? Please answer my questions.
Regards,
Saravanan

[xmeng]

Hi Saravanan,

Yes. kmer compression is implemented to reduce the size of intermediate results in BioPig. Regarding the datatype, can you show error messages if chararray is used?

Regards,
xmeng

[lanhin]

Hi Saravanan,
I'm afraid it's a bug. You should use bytearray instead of chararray. Since the fasta reader has already compressed reads, the chararray type will make kmer generator work in a wrong way. So please just use bytearray.
BTW, I've fixed this.

[saranpons3]

Hi Xmeng, Thanks for the reply. I will get back to you if i have any other questions.

…

On Thu, Aug 24, 2017 at 12:40 PM, lanhin ***@***.***> wrote: Hi Saravanan, I'm afraid it's a bug. You should use bytearray instead of chararray. Since the fasta reader has already compressed reads, the chararray type will make kmer generator work in a wrong way. So please just use bytearray . BTW, I've fixed this. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AH_7c4WbU6urD39KqfAXOOYDh0dVH3ZNks5sbSHcgaJpZM4O1a1w> .