BUG : too many open files when splitting on mac

[amnona]

Got this from a user:
ANOTHER ERROR ( ONLY solution seems to be to divide the file up into pieces)

(deblurenv)MacQIIME dhcp238­043:~ $ deblur workflow ­­seqs­fp

/Volumes/Samsung_T1/MadaRaw/Mada_AS13JF14ND14_Splitlib_r3n0p.90q5/Mada_AS13JF14ND14_Splitlib_r3n0p.90q5_seqs250_254new.fasta

­­output­dir Madajul16_deblurtest ­O 3

Traceback (most recent call last):

File "/macqiime/anaconda/envs/deblurenv/bin/deblur", line 621, in

deblur_cmds()

File "/macqiime/anaconda/envs/deblurenv/lib/python3.5/site­packages/click/core.py", line 716, in call

return self.main(_args, *_kwargs)

File "/macqiime/anaconda/envs/deblurenv/lib/python3.5/site­packages/click/core.py", line 696, in main

rv = self.invoke(ctx)

File "/macqiime/anaconda/envs/deblurenv/lib/python3.5/site­packages/click/core.py", line 1060, in invoke

return _process_result(sub_ctx.command.invoke(sub_ctx))

File "/macqiime/anaconda/envs/deblurenv/lib/python3.5/site­packages/click/core.py", line 889, in invoke

return ctx.invoke(self.callback, **ctx.params)

File "/macqiime/anaconda/envs/deblurenv/lib/python3.5/site­packages/click/core.py", line 534, in invoke

return callback(_args, *_kwargs)

File "/macqiime/anaconda/envs/deblurenv/bin/deblur", line 561, in workflow

out_dir_split)

File "/macqiime/anaconda/envs/deblurenv/lib/python3.5/site­packages/deblur/workflow.py", line 445, in

split_sequence_file_on_sample_ids_to_files

outputs[sample] = open(join(outdir, sample + '.fasta'), 'w')

OSError: [Errno 24] Too many open files: '/Users/venceslab/Madajul16_deblurtest/split/J8.3.fasta'

[wasade]

Or a smarter algorithm for partitioning a single fasta file in which fewer handles are open at a time. Probably could be done with a heap where the files which haven't been written too recently get closed, and reopened if needed

[josenavas]

Probably using a "pool" of file handles that we open/close as needed.

[wasade]

This still appears to be an issue. @amnona is there a trajectory toward a resolution here?

Note: this is not an issue when driven from qiime2, and so one strategy we can undertake is just actively recommend users leverage the plugin.

[amnona]

i think this will be left for version 2. currently mac users can split using qiime or qiime2?

[wasade]

Please note in the README

[amnona]

added in troubleshooting section