ability to work with captured stream "on the fly"

[yp-isi]

When using tcpdump to capture packets and writing to a file, we need to be able to compress the stream before writing. At high packet rates disk seems to be the bottleneck and using an ultra-fast compressor like snappy can significantly reduce disk bandwidth.

I've submitted a patch that adds an option --pipeoutput which inserts a piped "command" between raw capture and output file. For example:
tcpdump -w "pcap_%F-%T.gz" -G 3 -W 60 --pipeoutput gzip
will compress files on the fly.

Since the child (gzip in the example above) is running as a separate process, it can run on another core without robbing main tcpdump of cpu cycles.

Patch submitted: 3e08e65

[stevekay]

Silently fails if a bad pipeoutput command is given. Any way to trap this ?

$ sudo ./tcpdump  -w "pcap_%F-%T.gz" -G 3 -W 60 --pipeoutput foo
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
419 packets captured
424 packets received by filter
0 packets dropped by kernel
$ ls -l pcap_*
-rw-r--r--. 1 root  root     0 Jun 14 12:14 pcap_2015-06-14-12:14:12.gz
-rw-r--r--. 1 root  root     0 Jun 14 12:14 pcap_2015-06-14-12:14:17.gz
$

Using -z rather than --pipeoutput does give some indication of trouble.

$ sudo ./tcpdump  -w "pcap_%F-%T.gz" -G 3 -W 60 -z foo
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
compress_savefile:execlp(foo, pcap_2015-06-14-12:16:10.gz): No such file or directory
compress_savefile:execlp(foo, pcap_2015-06-14-12:16:17.gz): No such file or directory
compress_savefile:execlp(foo, pcap_2015-06-14-12:16:21.gz): No such file or directory

[yp-isi]

@stevekay, thanks for looking at this. I've updated the code to print a meaningful error message: 9f5d96f

[yp-isi]

@stevekay & other devs:

OK, reworked some issues with the code:

1. making sure buffered data is written through pipe when exiting on signal
2. making --pipeoutput work with -C (previously had ftell on a pipe issue) as in:
   sudo tcpdump -w /tmp/b1xx4cc-gz- -C 1 -s 1600 -Z user -i eth0 --pipeoutput /usr/bin/gzip
3. making --pipeoutput work with simple writing to a file without any rollovers:
   sudo tcpdump -w /tmp/b1xx4cs.gz -s 1600 -Z user -i eth0 --pipeoutput /usr/bin/gzip

Please review and pull if OK, if not please give feedback. Thanks!

Latest: 14cb024

[yp-isi]

Waiting for developers to comment/pull. Please note this is a new feature that does not break any existing functionality.

[infrastation]

This issue is basically a duplicate of the pull request #458.

[guyharris]

Duplicate of #458