nfcollect

Collect Netfilter NFLOG log entries and commit them to stable storage in binary (compressed) format.

The project contains two binaries: nfcollect and nfextract:

`nfcollect`

Collect packets from Netfilter netlink kernel interface. Packets are aggregated onto a memory region (we call it a trunk), until the trunk is full. A full trunk will be committed to disk by configurable means (currently zstd compression and no compression is implemented). Trunks will be stored in a specific directory, which will be scanned by nfextract to extract all trunks.

Due to communication with the kernel, this program requires root privilege.
The maximum size of raw collection is configured by storage_size. When we received that many packets, further packets will overwrite starting from the begining, mimicing a rotation-based storage. Unfortunately, if compression is enabled, the reduced size will not be reflected because currently we calculate size only in term of raw size. Thus, user has to manually scale storage_size by compression ratio of their workload.

Dependencies Installation

Fedora

sudo dnf install libnetfilter_log libzstd-devel

Ubuntu

sudo apt install libnetfilter-log1 libnetfilter-log-dev libzstd1 libzstd1-dev

Build

./bootstrap.sh
./configure
make

Run ./configure --enable-debug to enable debug output.

Usage

$ ./nfcollect --help
Usage: nfcollect [OPTION]

Options:
  -c --compression=<algo>      compression algorithm to use (default: no compression)
  -d --storage_file=<filename> sqlite database storage file
  -h --help                    print this help
  -g --nflog-group=<id>        the group id to collect
  -s --storage_size=<dirsize>  log files maximum total size in MiB
  -v --version                 print version information

$ ./nfextract -h     
Usage: nfextract [OPTION]

Options:
  -d --storage=<dirname>     sqlite storage file
  -h --help                  print this help
  -v --version               print version information
  -s --since                 start showing entries on or newer than the specified date (format: YYYY-MM-DD [HH:MM][:SS])
  -u --until                 stop showing entries on or older than the specified date (format: YYYY-MM-DD [HH:MM][:SS])

Examples

# Send all packets destined for localhost to the nflog group #5
sudo iptables -A OUTPUT -p tcp -d 127.0.0.1  -j NFLOG --nflog-group 5

# Receive the packets from nfnetlink
sudo ./nfcollect -d packets.db -g 5 -s 100 -c zstd

# Let it collect for a while ...

# Dump the collected packets
./nfextract -d packets.db

References

libnetfilter_log: https://www.icir.org/gregor/tools/files/doc.libnetfilter_log/html/libnetfilter__log.html
zstd: https://facebook.github.io/zstd/zstd_manual.html
lz4: https://github.com/lz4/lz4
sqlite: https://www.sqlite.org