Changes related to new option added: --pipeoutput.

[yp-isi]

This is a patch implementing a newoption --pipeoutput
This option allows you to pipe captured stream and process it in-line, before saving it to a file.
A use case is saving disk bandwidth using a fast compressor e.g. snappy before saving.
Compare this with postrotate:

-z snzip:

1. write full file to disk
2. read full file from disk
3. write compressed version back to disk
   ##--pipeoutput
4. write compressed version to disk

This is compelling since it saves one full read/one full write, while potentially utilizing other cores

[infrastation]

Could you please explain what advantage this change provides compared to the existing tcpdump -w - | gzip > file.pcap.gz way of having a pipeline?

[yp-isi]

@infrastation Hi, thanks for looking into this. The difference becomes apparent if you're using tcpdump for a long running high-rate capture. Piping output to gzip and saving to a file works but the file will keep growing indefinitely. You really want to keep writing multiple files and compressing them on the fly, to save the disk bandwidth.

[infrastation]

Now it is easier to understand, thank you.

[yp-isi]

@infrastation Hi Denis, have you had a chance to review this patch? Can we pull it or discuss why it's a bad idea?

[yp-isi]

OK, I'm not sure why Travis clang check failed; I tried clang and it all worked. Any ideas? I'm using a different version of clang though (3.5.0)

[yp-isi]

@infrastation fixed the bug that failed tests. Ready for a pull!

[yp-isi]

Folks, can you please pull or give a reason not to?

[infrastation]

Everybody has been plainly backlogged with other work, please excuse us. Could you rebase on the current master branch and make it one clean commit to make it easier to review?

[javedshakeel]

Just want to check if this has been already merged. I was looking for exactly this feature to reduce the amount of disk I/O.

[jmakov]

This is an important feature for enterprise users. How can I help to merge this ASAP?

[jmakov]

@yp-isi in the meantime I think this could be a hacky workaround. Let me know what you think:
tcpdump -w - | zstd | split -d -b 1G --filter="cat - > ${FILE}.pcap.zstd" - traffic_dump

What I think is problematic with my approach is that it's not clear:

1. what happens when this command receives e.g. SIGTERM - do all the buffers get flushed in the right order?
2. since zstd as gzip are stream compressors and do not know what a packet is, would ${FILE}.pcap.zstd contain whole packets or would a packet be cut off into 2 parts, the second part of the packet being at the beginning of a new file (created by split)

So having an option that's proposed in this PR would be helpful.

[yp-isi]

@yp-isi in the meantime I think this could be a hacky workaround. Let me know what you think:

It works on some level, but if your goal is to be able to process splits individually, this is not going to work, not only do you need to split before the compression, but also:

• you need to split on packet's boundary
• you need to have pcap header duplicated/adjusted for each split

Since the patch didn't find any traction here, we've created our own package to do this build around
libtrace:

https://ant.isi.edu/software/mtracecap/index.html

[jmakov]

@yp-isi thanks. Would be also great if that was on e.g. GitHub with a README.md etc.

[jmakov]

@yp-isi Also since I don't find any other channel, I cannot compile your source (v0.1 nor v0.4) on Ubuntu 19.04. Libtrace was installed as in https://github.com/LibtraceTeam/libtrace/wiki/Installing-Libtrace and whereis libtrace finds /usr/include/libtrace.h.
After running ./configure and then make, I get this make output (for v0.4):

$ make
make all-am
make[1]: Entering directory '/tmp/mtracecap-0.4'
g++ -DHAVE_CONFIG_H -I. -Wall -Werror -std=gnu++0x -g -O2 -MT mtracecap.o -MD -MP -MF .deps/mtracecap.Tpo -c -o mtracecap.o mtracecap.cc
mv -f .deps/mtracecap.Tpo .deps/mtracecap.Po
g++ -DHAVE_CONFIG_H -I. -Wall -Werror -std=gnu++0x -g -O2 -MT mtc_output.o -MD -MP -MF .deps/mtc_output.Tpo -c -o mtc_output.o mtc_output.cc
mv -f .deps/mtc_output.Tpo .deps/mtc_output.Po
g++ -Wall -Werror -std=gnu++0x -g -O2 -ltrace -lpthread -o mtracecap mtracecap.o mtc_output.o
/usr/bin/ld: mtracecap.o: in function cleanup_signal(int)': /tmp/mtracecap-0.4/mtracecap.cc:96: undefined reference to trace_interrupt'
/usr/bin/ld: mtracecap.o: in function main': /tmp/mtracecap-0.4/mtracecap.cc:306: undefined reference to trace_create_filter'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:319: undefined reference to trace_set_snaplen' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:321: undefined reference to trace_config'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:329: undefined reference to trace_start' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:333: undefined reference to trace_get_statistics'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:311: undefined reference to trace_create' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:312: undefined reference to trace_is_err'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:316: undefined reference to trace_set_event_realtime' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:317: undefined reference to trace_get_err'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:410: undefined reference to trace_event' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:463: undefined reference to trace_get_layer3'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:472: undefined reference to trace_get_erf_timestamp' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:510: undefined reference to trace_destroy_packet'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:517: undefined reference to trace_get_timeval' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:527: undefined reference to trace_destroy_packet'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:540: undefined reference to trace_get_statistics' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:543: undefined reference to trace_destroy'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:400: undefined reference to trace_get_erf_timestamp' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:449: undefined reference to trace_event'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:409: undefined reference to trace_create_packet' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:499: undefined reference to trace_perror'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:172: undefined reference to trace_help' /usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:330: undefined reference to trace_perror'
/usr/bin/ld: /tmp/mtracecap-0.4/mtracecap.cc:322: undefined reference to trace_perror' /usr/bin/ld: mtc_output.o: in function MTC_Output::close_trace(void*)':
/tmp/mtracecap-0.4/mtc_output.cc:180: undefined reference to trace_destroy_output' /usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:180: undefined reference to trace_destroy_output'
/usr/bin/ld: mtc_output.o: in function MTC_Output::dump_seg_stats() const': /tmp/mtracecap-0.4/mtc_output.cc:392: undefined reference to trace_get_statistics'
/usr/bin/ld: mtc_output.o: in function MTC_Output::open_trace(timeval const&)': /tmp/mtracecap-0.4/mtc_output.cc:308: undefined reference to trace_create_output'
/usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:309: undefined reference to trace_is_err_output' /usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:321: undefined reference to trace_config_output'
/usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:328: undefined reference to trace_start_output' /usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:315: undefined reference to trace_config_output'
/usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:317: undefined reference to trace_perror_output' /usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:310: undefined reference to trace_perror_output'
/usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:329: undefined reference to trace_perror_output' /usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:323: undefined reference to trace_perror_output'
/usr/bin/ld: mtc_output.o: in function MTC_Output::write_packet(libtrace_packet_t*)': /tmp/mtracecap-0.4/mtc_output.cc:205: undefined reference to trace_get_timeval'
/usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:211: undefined reference to trace_get_capture_length' /usr/bin/ld: /tmp/mtracecap-0.4/mtc_output.cc:227: undefined reference to trace_write_packet'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:398: mtracecap] Error 1
make[1]: Leaving directory '/tmp/mtracecap-0.4'
make: *** [Makefile:299: all] Error 2

[jmakov]

@yp-isi I've created a min mtracecap CMake project (and fixed the issue above) here https://github.com/jmakov/mtracecap. Please copy it and publish (so I can fork from you) on e.g. Github so we can send pull requests to you. Let me know when you do so I can delete my Github mtracecap repository.

[mcr]

do you have time/energy to rebase this?