Decrease paste time by inserting line breaks

[jyn514]

When pasting bitshuffle-encoded data, large files can take a long time to paste, since the application must read the text into memory. This is exacerbated by large line lengths.

The current standard is to add one line break after each packet. However, with large chuck sizes (>~1000) this decreases load times significantly.

We should add an option, enabled by default, to insert line breaks approximately every 100 characters.

[charlesdaniels]

When pasting bitshuffle-encoded data, large files can take a long time to paste, since the application must read the text into memory. This is exacerbated by large line lengths. The current standard is to add one line break after each packet. However, with large chuck sizes (>~1000) this decreases load times significantly.

This is based on my own unscientific testing with vim, which appeared to be slower with longer line lengths (I assume the in-memory data structure used to represent vim buffers probably wasn’t designed for ingesting hundreds of multi-thousand character lines at high speed). This may vary with different editors.

We should add an option, enabled by default, to insert line breaks approximately every 100 characters.

I would suggest 79 characters (80 if you count the line break), as this is the standard width of a terminal.

[jyn514]

On the topic of reducing paste time - I'm considering not opening an editor unless specified. Even vim adds a considerable amount of time for no benefit I'm aware of, since there's no need to edit the pasted packet. Will try to get some numbers on this.

[charlesdaniels]

Of course using $EDITOR is less performant. The purpose is as a quality of life feature. Namely, if someone sends you a BitShuffle message, viewing it's contents just takes bitshuffle -d  Followed by a control-v :wq. This is much more convienient, especially very large message where pasting inside of an echo "..." | bitshufle ... would be difficult.  If we want to cut this as a default behaviour, we should add an rc file (~/.config/bitshuffle/bitshuffle.yaml) where the user can specify this behaviour explicitly (namely because I'm one of the two total users of BitShuffle, and I use this behaviour nearly every time I use Bitshuffle).  I can code up the config file handling easily enough if desired. 

…

On Sun, 2018-02-18 at 08:02 -0800, Joshua Nelson wrote: On the topic of reducing paste time - I'm considering not opening an editor unless specified. Even vim adds a considerable amount of time for no benefit I'm aware of, since there's no need to edit the pasted packet. Will try to get some numbers on this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[jyn514]

Possible alternate usage:

cat << EOF | bitshuffle -d
<packet>
EOF