netsniff-ng - a high performant packet sniffer

Update: 11-03-2012 - Added toolkit's components "Big Picture"
Update: 09-24-2012 - Added installation procedure
Update: 07-16-2012 - Formatting, mention Jumbo frame support, revised drop stat explanations
Update:07-30-2012-Code change, Stats were moved to new file:netsniff-ng/src/xutils.c, see [4]reformatted with tags, added Persistent Logging and Stat section along with awk scripts. Update: 08-01-2012 - Added section on timestamps and script to convert them Update: 08-16-2012 - Added section on replaying traffic & modified script at bottom Update: 08-28-2012 - Added comments on pcap file version

Synopsis:

Part of the netsniff-ng suite: http://netsniff-ng.org/- netsniff-ng should be compiled from source.Many package repositories are using old binaries that lack many of new and useful features. Here animage of the toolkit's components:
The first sniffer that invoked both, the zero-copy RX_RING as well as the zero-copyTX_RING for high-performance network I/O and scatter/gather or mmaped I/O. The shared memory RINGs are new additions to the PF_PACKET socket.PF_PACKET is also referred to as AF_PACKET. [1] Socket performance is said to be comparable to that of PF_RING.See comment from Daniel Miller, the Linux kernel networking maintainer. [2] For traffic analysis, I recommend using netsniff-ng for full-content packet capture to disk, and then using other tools toread and analyze the capture file. Note: netsniff-ng only supports pcap file version 2.4
$ file foo.pcap
foo.pcap: tcpdump capture file (little-endian) - version 2.4
(Ethernet, capture length 65535)

Compilation & Installation Walkthrough:

Since netsniff-ng is a suite of programs, a missing dependency for one will cause thecompiler to skip its build but will continue onto the rest of the tools whose needs are met.

$ git clone https://github.com/gnumaniacs/netsniff-ng.git

First, review the necessary files, README & INSTALL.

$ cd netsniff-ng
$ ls
AUTHORS  CHANGELOG  CODING  COPYING  CREDITS  HACKING  INSTALL  netsniff-ng.8  README  src  TODO  VERSION
$ less INSTALL
$ less README

Let's try and obtain some of the dependencies listed in the README files, we'll come to a few others down the road asI'd like to show examples of problems you may encounter:

$ apt-get install tlsdate libcli-dev libgeoip-dev liburcu-dev libnetfilter-conntrack-dev
$ cd src/
$ mkdir build
$ cd build/
$ cmake ..
-- The C compiler identification is GNU
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
...
-- Found Threads: TRUE 
-- Found Libnl: /usr/lib/libnl.so 
-- Found FLEX: /usr/bin/flex 
-- Found BISON: /usr/bin/bison 
-- Found libcli: /usr/lib/libcli.so 
-- Found Curses: /usr/lib/x86_64-linux-gnu/libcurses.so 
-- Found LibGeoIP: /usr/lib/libGeoIP.so 
-- Found libnetfilter_conntrack: /usr/lib/libnetfilter_conntrack.so 
-- Found liburcu: /usr/lib/liburcu.so 
-- Could NOT find NaCl (missing:  NACL_LIBRARY NACL_INCLUDE_DIR) 
NaCl was not found. Check if NACL_INCLUDE_DIR and NACL_LIB_DIR are properly set in nacl_path.cmake. Skipping curvetun build.
-- Configuring done
-- Generating done
-- Build files have been written to: /root/netsniff-ng/src/build

In the above example we're missing the LibNaCL libraries required by curvetun.Conveniently, netsniff-ng includes a script to download and install the libraries for us.

$ cd ../curvetun
$ mkdir ~/test
$ /build_nacl.sh ~/test/
Building NaCl for arch amd64 on host nms (grab a coffee, this takes a while) ...
NaCl lib path /root/test//nacl-20110221/build/nms/lib/amd64
NaCl include path /root/test//nacl-20110221/build/nms/include/amd64
Done!

Let's try that again!

$ cd ~/netsniff-ng/src/build
$ cmake ..
-- System has SO_ATTACH_FILTER/SO_DETACH_FILTER support
-- System has PF_PACKET sockets
-- System has strict alignment
-- System has TX_RING support
-- System has no SOF_TIMESTAMPING_RAW_HARDWARE support
-- System has GeoIPv6 support
-- Found NaCl: /root/test/nacl-20110221/build/nms/lib/amd64/libnacl.a 
-- Configuring done
-- Generating done
-- Build files have been written to: /root/netsniff-ng/src/build

Now that cmake has detected that all the required components are available we will makean attempt to compile the software.

$ make
Scanning dependencies of target netsniff-ng
[  1%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/hash.c.o
[  2%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/dissector.c.o
[  2%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/dissector_eth.c.o
[  3%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/dissector_80211.c.o
[  3%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/proto_arp.c.o
...
cc1: some warnings being treated as errors
make[2]: *** [netsniff-ng/CMakeFiles/netsniff-ng.dir/__/mac80211.c.o] Error 1
make[1]: *** [netsniff-ng/CMakeFiles/netsniff-ng.dir/all] Error 2
make: *** [all] Error 2

We encountered another error, this messeage is an indication that the correct libnl library is not installed.

Another case, slightly different, with an older libnl version. Notice thatwe were able to pass building the mac80211 object whereas that failed in thepreceding example.

$ make
[  0%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/mac80211.c.o
[  1%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/pcap.c.o
[  1%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/pcap_rw.c.o
[  2%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/pcap_sg.c.o
[  2%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/pcap_mmap.c.o
/root/netsniff-ng/src/pcap_mmap.c: In function pcap_mmap_prepare_close_pcap:
/root/netsniff-ng/src/pcap_mmap.c:238:2: warning: ISO C90 forbids mixed declarations and code [-Wdeclaration-after-statement]
[  3%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/ring_rx.c.o
[  4%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/ring_tx.c.o
[  4%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/mtrand.c.o
[  5%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/tprintf.c.o
[  5%] Building C object netsniff-ng/CMakeFiles/netsniff-ng.dir/__/netsniff-ng.c.o
make[2]: *** No rule to make target `/usr/lib/libnl.so', needed by `netsniff-ng/netsniff-ng'.  Stop.
make[1]: *** [netsniff-ng/CMakeFiles/netsniff-ng.dir/all] Error 2

I'll remove the older libnl libraries and install the latest version to satisfy netsniff-ng's needs:

 $ apt-get remove --purge libnl1 libnl2
$ wget http://www.infradead.org/~tgr/libnl/files/libnl-3.2.13.tar.gz
$ tar -zxf libnl-3.2.13.tar.gz
$ cd libnl-3.2.13
$ ./configure
$ make && make install

If you have existing libnl libraries installed and don't want to remove the older ones, use a symbolic linkto point the older libnl.so files to the newer version of the library, libnl.so.3.0.0.

/usr/lib $ ln -s libnl.so.3 libnl.so
ls -l /usr/lib/libnl*
lrwxrwxrwx 1 root root     18 Oct  6  2011 libnl-cli.so.3 -> libnl-cli.so.3.0.0
-rw-r--r-- 1 root root  39880 Oct  6  2011 libnl-cli.so.3.0.0
lrwxrwxrwx 1 root root     19 Oct  6  2011 libnl-genl.so.3 -> libnl-genl.so.3.0.0
-rw-r--r-- 1 root root  19504 Oct  6  2011 libnl-genl.so.3.0.0
lrwxrwxrwx 1 root root     17 Oct  6  2011 libnl-nf.so.3 -> libnl-nf.so.3.0.0
-rw-r--r-- 1 root root  65752 Oct  6  2011 libnl-nf.so.3.0.0
lrwxrwxrwx 1 root root     20 Oct  6  2011 libnl-route.so.3 -> libnl-route.so.3.0.0
-rw-r--r-- 1 root root 260856 Oct  6  2011 libnl-route.so.3.0.0
lrwxrwxrwx 1 root root     10 Sep 17 16:39 libnl.so ->libnl.so.3
lrwxrwxrwx 1 root root     14 Oct  6  2011 libnl.so.3 -> libnl.so.3.0.0
-rw-r--r-- 1 root root  92784 Oct  6  2011 libnl.so.3.0.0
$ cd ~/netsniff-ng/src/build
$ make && make install

Packet Drop Stats:

The drop count output from the program is listed below. It's helpful to determinewhere these value come from as they may be useful in debugging excessive drops.

0 frames failed filter (out of space)
0.0000% frame droprate

Older Version: I traced it to this from netsniff-ng/src/xsys.c [3] :
Latest Version: Moved code to new file: netsniff-ng/src/xutils.c[4] :

568 ret = getsockopt(sock, SOL_PACKET, PACKET_STATISTICS, &kstats,&slen);
569     if (ret > -1) {

570         printf("\r%12ld frames incoming\n",

571            1UL * kstats.tp_packets);

572         printf("\r%12ld frames passed filter\n",

573            1UL * kstats.tp_packets - kstats.tp_drops - skipped);

574         printf("\r%12ld frames failed filter (out of space)\n",

575            1UL * kstats.tp_drops + skipped);

576         if (kstats.tp_packets > 0)

577             printf("\r%12.4f%% frame droprate\n", 1.f * 578 kstats.tp_drops / kstats.tp_packets * 100.f)

tp_drops from PF_PACKET's PACKET_STATISTICS holds the number of packetswhich were dropped due to a lack of buffer space in PF_PACKET.This value is used in calculating the drop percentage displayed in the above snippet at line 578. Drops reported by tp_drops are due to a high load.The skipped variable (line 575), shown in the "frames failed filter (out of space)" output, holds the number of packets received but were too large forthe allocated ring buffer's frame size and thus not processed. By default the ring buffer is divided into 2048 byte slots, if you have a frame that exceedsthis amount, say a Jumbo Frame of 9000 bytes, it will not fit into this slot and will be dropped. These drops are reported in the skipped variable. Ifyou need jumbo support start netsniff-ng with the Jumbo option ( -J ).

"frames failed filter (out of space)" is the sum of the value of tp_drops and thevalue of skipped.

Now, we can see that neither of the drop values reported are being dropped by the NIC card itself or the device driver.You will have to look elsewhere for these numbers e.g. ethtool ( for supported hardware ), /proc/net/dev (ifconfig, ifpps etc.)

Examples:

  1. Basic netsniff-ng usage, print one packet ( --num ), set RX_RING size to larger value ( --ring-size ), bind to CPU 0 ( -b ), and set process as high priority ( -H ).

    netsniff-ng --dev eth0 --num 1 --ring-size 50MB -b 0 -H 

    1netsniff-ng_detailed

  2. Write traffic from eth0 to disk ( --in ) and ( --out ), don't print to packets tostdout i.e. screen ( -s ), set high priority ( -H ), and bind to CPU 0 ( -b ).

    netsniff-ng --in eth0 --out test.pcap -s -H -b 0 

    2netsniff-ng_pcap

  3. Only print one packet ( --num ) and its header ( -N ).

    netsniff-ng --in eth0 --num 1 -N 

    3netsniff-ng_header_num

  4. Print one packet ( --num ) and dump its payload in ASCII ( -l )

    netsniff-ng --dev eth0 --num 1 -l 

    4netsniff-ng_payload

  5. Print one packet and its entire contents in hex ( -X ), use ( -x ) to print only the payload in hex.

    netsniff-ng --dev eth0 --num 1 -X 

    5netsniff-ng_hexdump

  6. Print packets in less verbose ( -q )

    netsniff-ng --dev eth0 --num 1 -q 

    6netsniff-ng_quite

  7. Add BPF filter to netsniff-ng ( -f )

    netsniff-ng --in eth0 --out test.pcap -f arp.bpf --num 1 

    7netsniff-ng_filter

  8. Create a BPF filter with tcpdump by dumping the filter to a file with tcpdump( -dd ) then load the file with the dumped filter into netsniff-ng for processing.Write a new pcap file every 60 seconds ( -F ), pcap files will be titled in unix epoch time.

    tcpdump -dd arp 1>arp.bpfc 2>/dev/null 
    netsniff-ng --in eth0 --out test.pcap -f arp.bpfc -F 60 --num 1 

    8netsniff-ng_filter2_multiple

  9. Write pcaps out to directory ( --out ), write a new pcap every 60 seconds ( the default - redundant in this example ), and bindto multiple CPU's ( -b ).To create a new daily capture file set seconds to 86400 ( -F ).

    netsniff-ng --in eth0 --out mypcaps/ -F 60 -s -H -b 2,3 

    9netsniff-ng_every_minute

Replay:

Netsniff-ng can replay traffic, much like tcpreplay, from one interface or pcap file to another interface. One situation in whichreplaying is useful iswhen you want to analyze a pcap file but a program such as iftop doesn't support reading from a file.

Have iftop listen on an interface such as the loopback and use netsniff-ng to replay packets to the loopback interface:

iftop -i lo

netsniff-ng_replay_iftop

netsniff-ng --in traffic.pcap --out lo

netsniff-ng_replay

Replaying is also good for testing intrusion detection systems by sending pcaps with malicious traffic.

Persistent Logging:

Here's a full-content capture example script for a sensor/NMS (Network Monitoring System) with multiple interfaces writing to disk.I pass the ( -Q ) option due to prevent netsniff-ng from changing my custom IRQ CPU2NIC bindings. Larger RX ring sizes are set forhigher throughput interfaces. I'm sending stdout and stderr to a file for each instance. netsniff-ng will print out stats to this file afterevery newly created file, in our case, each day. I print the date for my records each time this script is ran which should only be after a reboot.

1
2
3
4
5
6
7
8
9
10
#!/bin/bash
dir=/media/pcaps
date=$(date +"%x-%X")
stats=/var/log/stats/netsniff-ng
echo $date >> $stats/eth0.txt; /usr/sbin/netsniff-ng --bind-cpu 0 --in eth0 --out $dir/eth0/ --ring-size 1GB -Q -s --interval 86400 &>> $stats/eth0.txt &
echo $date >> $stats/eth1.txt; /usr/sbin/netsniff-ng --bind-cpu 1 --in eth1 --out $dir/eth1/ --ring-size 1GB -Q -s --interval 86400 &>> $stats/eth1.txt &
echo $date >> $stats/eth2.txt; /usr/sbin/netsniff-ng --bind-cpu 3 --in eth2 --out $dir/eth2/ --ring-size 128MB -Q -s --interval 86400 &>> $stats/eth2.txt &
echo $date >> $stats/eth3.txt; /usr/sbin/netsniff-ng --bind-cpu 2 --in eth3 --out $dir/eth3/ --ring-size 128MB -Q -s --interval 86400 &>> $stats/eth3.txt &
echo $date >> $stats/eth4.txt; /usr/sbin/netsniff-ng --bind-cpu 3 --in eth4 --out $dir/eth4/ --ring-size 128MB -Q -s --interval 86400 &>> $stats/eth4.txt &
echo $date >> $stats/eth4.txt; /usr/sbin/netsniff-ng --bind-cpu 3 --in eth5 --out $dir/eth5/ --ring-size 128MB -Q -s --interval 86400 &>> $stats/eth5.txt &

Stats:

It's important to know when netsniff-ng is unable to process packets in order to mitigate further losses and to understandnetwork conditions at the time of loss.

When in dump mode ( --out <dir>), which writes a new pcap in the specified directory based on the interval ( --interval ),the stats are printed to stdout after every new file. Stats are not printed like this in other modes e.g. writing a single pcap.Here's a code sample fromtheenter_mode_rx_only_or_dump sectionof netsniff.c [5]:

if (mode->dump && next_dump) {
struct tpacket_stats kstats;
socklen_t slen = sizeof(kstats);
fmemset(&kstats, 0, sizeof(kstats));
getsockopt(sock, SOL_PACKET, PACKET_STATISTICS,
&kstats, &slen);
fd = next_multi_pcap_file(mode, fd);
next_dump = false;
if (mode->print_mode == FNTTYPE_PRINT_NONE) {
printf(".(+%lu/-%lu)", 1UL * kstats.tp_packets - kstats.tp_drops - skipped, 1UL * kstats.tp_drops + skipped);
fflush(stdout);

By piping stdout into a file, like in the example script from the Persistent Logging section of this article, we can have a look at thestat data ex-ante.Output is very basic and takes the form of (+packets_processed/-packets_lost).

.(+163916/-0).(+73641/-0).(+584687/-0).(+677999/-0).(+319949/-0).(+363009/-0).(+959416/-0).(+156566/-0).(+22928/-0).(+49017/-0).(+25172/-0).(+22678/-0).(+34653/-0).(+53730/-0).(+2361501/-0).(+17782688/-0).(+16870388/-0).(+26805509/-0).(+15671915/-0).(+19167822/-58606).(+17028296/-0).(+20680989/-0).(+15228690/-0).(+3888172/-0).(+1428221/-0).(+398312/-0).(+336574/-0).(+1303433/-0).(+1834177/-0).(+1035831/-0).(+876504/-0).(+518982/-0).(+407991/-0).(+156001/-0).(+1363033/-0).(+1361576/-0).(+81246/-0).(+51907/-0).(+1522237/-0).(+8194604/-0).(+1003

Each time a new file is created ( -F / --interval ) stats for the previous file will be written to stdout.

netsniff-ng_dump_stats

The default value is60 seconds.We can use an awk scriptto organize the data so it isn't so hard on the eyes.

awk 'BEGIN { RS="."; FS="/"; ORS="\n" } { print }' eth0.txt | tail
(+6986726/-0)
(+3995027/-0)
(+3580777/-0)
(+8869777/-0)
(+10382901/-0)
(+3081145/-0)
(+528693/-0)
(+458763/-0)
(+615587/-0)
(+127489/-0)

To only print lines that have a loss count greater than 0, try this:

awk 'BEGIN { RS="."; FS="/"; ORS="\n" } { if( $0 !~ /netsniff/ && substr( $2,2,length($2)-2 ) > 0 ) print }' eth0.txt
(+4715313/-147096)
(+8343155/-9325)

Dump Mode - Timestamps:

When in dump mode ( --out <dir> ) capture files are written to the specified directory at an interval ( --interval ).In addition to this, filenames are named with the creation time in seconds since Unix Epoch[6].

$ date +"%s"
1343872303

The timestamps aren't really human readable so you'll probably need to convert them to something easier.To convert the time in the following pcap filename, 1343872303.pcap, use the date command with its outputnotation formats. I like this granular expression:

$ date -d @1343872303 +"%m-%d-%Y_%R:%S"
08-01-2012_21:51:43

netsniff-ng_date_conversion

Here's a script you can modify/use to automatically change the filenames to another date format. The scriptalso calculates a hash of the pcap file for validating its integrity, a good practice.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/bin/bash
# directories under /mnt/pcaps are like this: eth0/ eth1/ eth2/ eth3/ eth4/ etc.
ext_hd=/mnt/pcaps

if ! [[ "$1" ]]; then
  printf "Usage: $0 /mnt/pcaps/<int_dir>\n"
  exit
fi

cd $ext_hd/$1

for file in $(ls -1)
do
  # get length of string, filename
  ln=${#file}
  fn=${file:0:10}
  # if length is 15, meaning epoch time without extension AND
  # the date of the file is not today, go ahead and rename and hash file.
  # we need to skip today's file because it's in use
  if [[ $ln -eq 15 ]] && [[ "$(date @$fn +"%Y-%m-%d")" != "$(date +"%Y-%m-%d")" ]]; then
    # copy string excluding the .pcap extension
    nf=$(date -d @$fn +"%m-%d-%Y_%R:%S")
    mv $file $nf.pcap
    sha1sum $nf.pcap | cut -d " " -f1 > $nf.sha1
  fi
done

References:

[1] http://www.allinterview.com/showanswers/60403.html
[2] http://www.spinics.net/lists/netfilter-devel/msg20212 .html
[3] https://github.com/gnumaniacs/netsniff-ng/blob/master/src/xsys.c
[4]https://github.com/gnumaniacs/netsniff-ng/blob/master/src/xutils.c
[5]
https://github.com/gnumaniacs/netsniff-ng/blob/master/src/netsniff-ng.c
[6]
http://en.wikipedia.org/wiki/Unix_time