GitHub - assafmo/joincap: Merge multiple pcap files together, gracefully. (original) (raw)
joincap
Merge multiple pcap files together, gracefully.
Installation
- Download a precompiled binary from https://github.com/assafmo/joincap/releases
- Or... Use
go get:
go get -u github.com/assafmo/joincap - Or use Ubuntu PPA:
curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add -
sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list
sudo apt update
sudo apt install joincap
Basic Usage
Usage: joincap [OPTIONS] InFiles...
Application Options: -v, --verbose Explain when skipping packets or entire input files -V, --version Print the version and exit -w= Sets the output filename. If the name is '-', stdout will be used (default: -) -c= An integer argument for limiting the pcap size (default: 9223372036854775807) -p=[micros|nanos] Sets timestamp precision (default: micros)
Help Options: -h, --help Show this help message
Why?
I believe skipping corrupt packets is better than failing the entire merge job.
When using tcpslice or mergecap sometimes pcapfix is needed to fix bad input pcap files.
- One option is to try and run merge (
mergecap/tcpslice), if we get errors then runpcapfixon the bad pcaps and then run merge again.- Adds complexity (run -> check errors -> fix -> rerun)
- (If errors) Demands more resources (
pcapfixprocesses) - (If errors) Extends the total run time
- Another option is to run
pcapfixon the input pcap files and then merge.- Extends the total run time by a lot (read and write each pcap twice instead of once)
- Demands more storage (for the fixed pcaps)
- Demands more resources (
pcapfixprocesses)
- We can use
pcapfix"in memory" with process substitution:mergecap -w out.pcap <(pcapfix -o /dev/stdout 1.pcap) <(pcapfix -o /dev/stdout 2.pcap).- Adds complexity (build a complex command line)
- Demands more resources (
pcapfixprocesses) - Harder for us to use pathname expansion (e.g.
tcpslice -w out.pcap *.pcap) - We have to mind the command line character limit (in case of long pathnames)
- Doesn't work for
tcpslice(seeks the last packets to calculate time ranges - cannot do this with pipes)
Error handling: joincap vs mergecap vs tcpslice
Results
| Use case | joincap | mergecap v2.4.5 | tcpslice v1.2a3 |
|---|---|---|---|
| Corrupt input global header | ✔️ | ❌ | ❌ |
| Corrupt input packet header | ✔️ | ❌ | ❌ |
| Unexpectd EOF(last packet data is truncated) | ✔️ | ✔️ | ✔️ |
| Input pcap has no packets(global header is ok, no first packet header) | ✔️ | ✔️ | ❌ |
| Input file size is smaller than 24 bytes(global header is truncated) | ✔️ | ✔️ | ❌ |
| Input file size is between 24 and 40 bytes(global header is ok, first packet header is truncated) | ✔️ | ❌ | ❌ |
| Input file doesn't exists | ✔️ | ❌ | ❌ |
| Input file is a directory | ✔️ | ❌ | ❌ |
| Input file end is garbage | ✔️ | ✔️ | ❌ |
| Input file is gzipped (.pcap.gz) | ✔️ | ✔️ | ❌ |
Error outputs
| Use case | Error outputs |
|---|---|
| Corrupt input global header | tcpslice: bad tcpdump file test_pcaps/bad_global.pcap: archaic pcap savefile formatmergecap: The file "test_pcaps/bad_global.pcap" contains record data that mergecap doesn't support. (pcap: major version 0 unsupported) |
| Corrupt input packet header | tcpslice: Infinite loop?mergecap: The file "test_pcaps/bad_first_header.pcap" appears to be damaged or corrupt. (pcap: File has 2368110654-byte packet, bigger than maximum of 262144) |
| Unexpectd EOF(last packet data is truncated) | |
| Input pcap has no packets(global header is ok, no first packet header) | tcpslice: Outputs empty pcap (Only global header) |
| Input file size is smaller than 24 bytes(global header is truncated) | tcpslice: bad tcpdump file test_pcaps/empty: truncated dump file; tried to read 4 file header bytes, only got 0 |
| Input file size is between 24 and 40 bytes(global header is ok, first packet header is truncated) | tcpslice: bad status reading first packet in test_pcaps/partial_first_header.pcap: truncated dump file; tried to read 16 header bytes, only got 11mergecap: The file "test_pcaps/partial_first_header.pcap" appears to have been cut short in the middle of a paket. |
| Input file doesn't exists | tcpslice: bad tcpdump file ./not_here: ./not_here: No such file or directorymergecap: The file "./not_here" doesn't exist. |
| Input file is a directory | tcpslice: bad tcpdump file examples: error reading dump file: Is a directorymergecap: "examples" is a directory (folder), not a file. |
| Input file end is garbage | tcpslice: problems finding end packet of file test_pcaps/bad_end.pcap |
| Input file is gzipped (.pcap.gz) | tcpslice: bad tcpdump file test_pcaps/ok.pcap.gz: unknown file format |
How to reproduce
| Use case | How to reproduce |
|---|---|
| Corrupt input global header | joincap -w out_joincap.pcap test_pcaps/bad_global.pcapmergecap -w out_mergecap.pcap test_pcaps/bad_global.pcaptcpslice -D -w out_tcpslice.pcap test_pcaps/bad_global.pcap |
| Corrupt input packet header | joincap -w out_joincap.pcap test_pcaps/bad_first_header.pcapmergecap -w out_mergecap.pcap test_pcaps/bad_first_header.pcaptcpslice -D -w out_tcpslice.pcap test_pcaps/bad_first_header.pcap |
| Unexpectd EOF(last packet data is truncated) | joincap -w out_joincap.pcap test_pcaps/unexpected_eof_on_first_packet.pcapmergecap -w out_mergecap.pcap test_pcaps/unexpected_eof_on_first_packet.pcaptcpslice -D -w out_tcpslice.pcap test_pcaps/unexpected_eof_on_first_packet.pcapjoincap -w out_joincap.pcap test_pcaps/unexpected_eof_on_second_packet.pcapmergecap -w out_mergecap.pcap test_pcaps/unexpected_eof_on_second_packet.pcaptcpslice -D -w out_tcpslice.pcap test_pcaps/unexpected_eof_on_second_packet.pcap |
| Input pcap has no packets(global header is ok, no first packet header) | joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcapmergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcaptcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/no_packets.pcap |
| Input file size is smaller than 24 bytes(global header is truncated) | joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/emptymergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/emptytcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/emptyjoincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcapmergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcaptcpslic -De -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/partial_global_header.pcap |
| Input file size is between 24 and 40 bytes(global header is ok, first packet header is truncated) | joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcapmergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcaptcpslic -De -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/partial_first_header.pcap |
| Input file doesn't exists | joincap -w out_joincap.pcap test_pcaps/ok.pcap ./not_heremergecap -w out_mergecap.pcap test_pcaps/ok.pcap ./not_heretcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap ./not_here |
| Input file is a directory | joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/mergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/tcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/ |
| Input file end is garbage | joincap -w out_joincap.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcapmergecap -w out_mergecap.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcaptcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap test_pcaps/bad_end.pcap |
| Input file is gzipped (.pcap.gz) | joincap -w out_joincap.pcap test_pcaps/ok.pcap.gzmergecap -w out_mergecap.pcap test_pcaps/ok.pcap.gztcpslice -D -w out_tcpslice.pcap test_pcaps/ok.pcap.gz |
Benchmarks
| Version | Speed | Time | |
|---|---|---|---|
| mergecap | 3.2.2 | 590MiB/s | 0m5.632s |
| tcpslice | 1.2a3 | 838MiB/s | 0m3.666s |
| joincap | 0.10.2 | 562MiB/s | 0m5.462s |
- Merging 3 files with total size of 2.99994GiB.
- Running on Linux 5.4.0-21-generic, with Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (with SSE4.2), with 31765 MB of physical memory, with locale C, with zlib 1.2.11.