Dnsanon: extract DNS traffic from pcap to text with optionally anonymization

dnsanon

Dnsanon reads pcap files, extracts DNS traffic, and writes it to several plain-text tables (in Fsdb format). Optionally it anonymizes the IP addresses and queries.

Pre-built versions for RPM (Fedora, CentOS, REHL): see https://copr.fedorainfracloud.org/coprs/johnh/dnsanon/

  • dnsanon-1.11-2.tar.gz [SRPM] (2019-05-23); current release

    • Bugfix (some files are empty if inline compression is enabled) (bugfix)
    • Added new option to force creation of emptyfiles, even if DNS packet data is absent (improvement)
    • Update DSC TCP reassembly code to the latest release v.2.8.1 (improvement)
    • Bugfix in limiting tcp list size when running in tcp-relax mode (bugfix)
    • Code restructure and cleanup (improvement)

  • dnsanon-1.9.tar.gz [SRPM] (2018-06-20)

    support install_static for a static libtrace and (default) install now is dynamic libaries

  • dnsanon-1.7.tar.gz [SRPM] (2017-09-19)

    add new -T option to report TTLs; bug fix: -o and -p; improve docs; ported to openssl-1.1

  • dnsanon-1.6.tar.gz [SRPM] (2016-09-13)

    fix to install process to correct install problem when not built in chroot

  • dnsanon-1.5.tar.gz (2016-07-18)

    adds functionality to limit and clean up idle TCP connections

  • dnsanon-1.4.tar.gz [SRPM] (2016-06-02)

    fixes a bug where -p Q failed to output messages with '_qr==1'

  • dnsanon-1.3.tar.gz [SRPM] (2016-05-20)

    Add option to write to stdout (with -o -)

  • dnsanon-1.2.tar.gz [SRPM] (2016-04-17)

    Fix build problem against ldns-devel-1.6.16 (such as for centos7).

  • dnsanon-1.1.tar.gz [SRPM] (2016-04-16)

    SRPM update.

  • dnsanon-1.1.tar.gz [SRPM] (2016-01-21)

    First public release.

Packages and libraries

We package dnsanon for Fedora/REHL/CentOS:

  • see COPR for dnsanon or install with dnf copr enable johnh/dnsanon; dnf install dnsanon

Building it under Fedora and CentOS requires libtrace, which provide as an external package at our libtrace package page.

Documentation

% dnsanon(1) % Liang Zhu liangzhu@isi.edu, John Heidemann johnh@isi.edu % October 22, 2016

NAME

dnsanon - extract DNS queries and responses from network trace file to text files with optional anonymization

SYNOPSIS

dnsanon [-p OPTION] [-a OPTION] [-k KEYFILE] [–pass4 BITS] [–pass6 BITS] [-i PATH] [-o PATH] [-f PREFIX] [-c CMP] [-t NUMBER] [-d NUMBER] [-l NUMBER] [-v] [-x] [-q] [-T]

DESCRIPTION

Dnsanon reads network trace files (any format accepted by libtrace, including pcap and ERF), extracts DNS traffic, and writes it to several plain-text tables (in Fsdb format). Optionally it anonymizes the IP addresses, query names, or both.

About parsing:

  • Output format is tab-separated text with a one-line header.

  • Both UDP and TCP are supported.

About anonymization:

  • One might hope to write anonymized queries back to pcap, but unfortunately one can’t do that easily, because of label compression.

  • IP anonymization is done with the Cryptopan algorithm (prefix-preserving). Optionally, high bits of addresses can passed and low bits anonymized. Both IPv4 and IPv6 are supported.

  • query name anonymization is done component-by-component

OPTIONS

-p/--parse OPTION
output format, availeable options are mqreQR: report DNS m)essages, and q)uestion and r)esponses sections, each as separate tables. Capital Q and R includes messages information to the question and response tables. Default is mqr (all three, non-unified)

Note that these options select what sections are reported, and not the status of the QR bit.

-a/--anon OPTION
anonymization, availeable options are id,addr,server-addr,client-addr,name,all (separated by comma)

id: DNS id in packet header

name: query and response content

addr: server address (server-addr) and client address (client-addr). Address with port 53 is identified as server address. If both src and dst port are 53, both of the address are anonymized (anonymized addresses are suffixed with *).

all: all of the above

-k/--key KEYFILE
specify the key file. KEYFILE is created if it doesn’t exist
--pass4 BITS
preserve BITS high bits of IPv4 addrsss as un-anonymized; default is 0, anonymizing all bits if anon is set
--pass6 BITS
similar to –pass4, but for IPv6 address
-i/--input PATH
specify input file path; required
-o/--output-path PATH
specify output path which should be directory; default is current directory. If given “-“ and the output is only one file, it will go to standard output. stdout (-) is incompatible with multi-file output. With stdout, -p is required and you can only specify one option for -p
-c/--compressor COMP
compressor, default is ‘/usr/bin/xz’, use “none” to disable
-f/--output-prefix PREFIX
prefix of the output file, default is timestamp of the first packet
-v/--verbose
verbose log; default is none
-x/--tcp-relax
optimistically try TCP reassembly even without seeing TCP SYN
-t/--tcp-timeout NUMBER
specify timeout for idle TCP connections used in TCP reassembly, default is 60s. Longer idle TCP stream will be discarded. 0 for no timeout.
-l/--tcp-list-limit NUMBER
specify the size limit of internal list to keep track of tcp states for TCP reassembly, used to control memory consumption
-T/--ip-ttl
output ‘Time To Live’ in IPv4 header or ‘Hop Limit’ in IPv6 header
-d/--tcp-check-period NUMBER
specify the time period to clean up idle TCP connections, default is 60s
-q/--show-empty-query
show empty query as ‘-‘ in unified query message (option -Q); otherwise, messages with empty query won’t show in the output

EXAMPLES

Assume test.pcap contains one DNS query and one response for “www.isi.edu A”.

Extracting Queries from a Trace to Text

  1. output query, responses and message sections (default)

     ./dnsanon -i test.pcap
    

    output files:

     1461102303.357871.message.fsdb.xz
     1461102303.357871.question.fsdb.xz
     1461102303.357871.rr.fsdb.xz
    
     1461102303.357871 is the timestamp for the first packet in the pcap file
    
  2. default output but with output file prefix and no compression

     ./dnsanon -i ./test.pcap -c none -f test
    

    output files:

     test.message.fsdb
     test.question.fsdb
     test.rr.fsdb
    
  3. output only query sections

     ./dnsanon -i ./test.pcap -c none -f test -p q
    

    output file:

     test.question.fsdb
    

    with content:

     #fsdb -F t msgid id name type class
     1	  10888	     www.isi.edu. A	IN
     2	  10888	     www.isi.edu. A	IN
     # ./dnsanon -i ./test.pcap -c none -f test -p q
    

    output fsdb schema:

     msgid: DNS message id in the network trace file
     id: the id in DNS header
     name: query name
     type: query type
     class: query class
    
  4. output only response sections

     ./dnsanon -i ./test.pcap -c none -f test -p r
    

    output file:

     test.rr.fsdb
    

    with content:

     #fsdb -F t msgid id section_name name type class ttl rdf_num rdata
     2     10888	 ANS		 www.isi.edu.	 A   IN	     1746	1	128.9.176.20
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	nitro.isi.edu.
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	vapor.isi.edu.
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	ns.east.isi.edu.
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	ns.isi.edu.
     2     10888	 ADD		 ns.isi.edu.	 A   IN	     28434	1	128.9.128.127
     2     10888	 ADD		 ns.east.isi.edu.    A	     IN		28434	1	65.114.168.20
     2     10888	 ADD		 nitro.isi.edu.	     A	     IN		128236	1	128.9.208.207
     2     10888	 ADD		 vapor.isi.edu.	     A	     IN		128236	1	128.9.64.64
     # ./dnsanon -i ./test.pcap -c none -f test -p r
    

    the schema of that output (the meanings of each field):

     msgid: DNS message id in the pcap file
     id: the id in DNS header
     section_name: AUS=>ANSWER SECTION; AUT=>AUTHORITY SECTION; ADD=>ADDITIONAL SECTION
     name: name of the resource records 
     type: type of the resource records
     class: class of the resource records
     ttl: ttl of the resource records
     rdf_num: number of fields in rdata
     rdata: specific data of the resource records
    
  5. output only message information

     ./dnsanon -i ./test.pcap -c none -f test -p m
    

    output file:

     test.message.fsdb
    

    with content:

     #fsdb -F t msgid time srcip srcport dstip dstport protocol id qr opcode aa tc rd ra z ad cd rcode qdcount ancount nscount arcount edns_present edns_udp_size edns_extended_rcode edns_version edns_z msglen
     1     1461102303.357871     192.168.1.2   59008   192.168.1.1 53 udp	 10888 0  0  0 0  1  001   0	   0	   1	   0	   0		0	      1			  4096	       0      0	0	48
     2     1461102303.362905     192.168.1.1   53	   192.168.1.2 59008	 udp   10888 1 0  0  0	   1	   100	   0	   0	   1		1	      4			  4	       1      4096	0	0	0	207
     # ./dnsanon -i ./test.pcap -c none -f test -p m
    

    The schema here:

     msgid: DNS message id in the pcap file
     time: time of the message in seconds with fractoins since the Unix epoch
     srcip: source IP address
     srcport: source port
     dstip: destination IP address
     dstport: destination port
     protocol:  the transport protocol: udp or tcp
     id: the id in DNS header
     qr: the query/repsonse bit from the DNS header, query is 0, response is 1
     opcode: the opcode field from the DNS header, 0: standard query, 1: invers, 2: status
     aa: the Authoritative Answer from the DNS header
     tc: the TrunCation bit from the DNS header
     rd: the Recursion Desired bit from the DNS header
     ra: the Recursion Availablebit from the DNS header
     z: the unused Z bit from the DNS header
     ad: the ad bit from the DNS header
     cd: the cd bit from the DNS header
     rcode: respons code
     qdcount: qd count from the DNS header
     ancount: an count from the DNS header
     nscount: ns count from the DNS header
     arcount: ar count from the DNS header
     edns_present: 1 if the packet has EDNS 
     edns_udp_size: edns udp size
     edns_extended_rcode: edns extended rcode
     edns_version: edns version
     edns_z: edns z value
     msglen: dns message size
    
  6. output query sections with combined message information

     ./dnsanon -i ./test.pcap -c none -f test -p Q
    

    output file:

     test.message_question.fsdb
    

    with content:

     #fsdb -F t msgid time srcip srcport dstip dstport protocol id qr opcode aa tc rd ra z ad cd rcode qdcount ancount nscount arcount edns_present edns_udp_size edns_extended_rcode edns_version edns_z msglen name type class
     1	1461102303.357871	192.168.1.2	59008	192.168.1.1	53	udp	10888	0	0	0	0	1	0	0	1	0	0	1	0	0	0	1	4096	0	0	0	48	www.isi.edu.	A	IN
     2	1461102303.362905	192.168.1.1	53	192.168.1.2	59008	udp	10888	1	0	0	0	1	1	0	0	0	0	1	1	4	4	1	4096	0	0	0	207	www.isi.edu.	A	IN
     # ../dnsanon -i ./test.pcap -c none -f test -p Q
    

Anonymization

  1. anonymization

     ./dnsanon -i ./test.pcap -c none -f test -p m -a all -k keyfile
    

    Full anonymization:

    IP is prefix-preserve anonymized and suffixed with *.

    ID in DNS header is encrpyted and output is base64 string.

    Each part of the names in resource records is encrpyted and output is base64 string.

    sample output files:

    test.message.fsdb:

     #fsdb -F s msgid time srcip srcport dstip dstport protocol id qr opcode aa tc rd ra z ad cd rcode qdcount ancount nscount arcount edns_present edns_udp_size edns_extended_rcode edns_version edns_z msglen
     1 1461102303.357871 210.91.230.223* 59008 210.91.230.220* 53 udp GMgZKqG94x3W7FJ0yrcJZg== 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 4096 0 0 0 48
     2 1461102303.362905 210.91.230.220* 53 210.91.230.223* 59008 udp GMgZKqG94x3W7FJ0yrcJZg== 1 0 0 0 1 1 0 0 0 0 1 1 4 4 1 4096 0 0 0 207
     # ./dnsanon -i ./test.pcap -c none -f test -p m -a all -k keyfile
    

    test.question.fsdb:

     #fsdb -F s msgid id name type class
     1 GMgZKqG94x3W7FJ0yrcJZg== hiALV6+e9T9ni1ZJyCD35Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN
     2 GMgZKqG94x3W7FJ0yrcJZg== hiALV6+e9T9ni1ZJyCD35Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN
     # ./dnsanon -i ./test.pcap -c none -f test -a all -k keyfile
    

    test.rr.fsdb:

     #fsdb -F s msgid id section_name name type class ttl rdf_num rdata
     2 GMgZKqG94x3W7FJ0yrcJZg== ANS hiALV6+e9T9ni1ZJyCD35Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 1746 1 128.215.201.242*
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 PoQD5IM57swzKjlFkC+OEw==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 Zw+bRwIuYctDLbA1SDVYGA==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 DswjzZnRq9/c5+jPctKr2Q==.ofRFtyXN6AeIHlBAb3SFCQ==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 DswjzZnRq9/c5+jPctKr2Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD DswjzZnRq9/c5+jPctKr2Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 28434 1 128.215.225.32*
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD DswjzZnRq9/c5+jPctKr2Q==.ofRFtyXN6AeIHlBAb3SFCQ==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 28434 1 82.117.24.134*
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD PoQD5IM57swzKjlFkC+OEw==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 128236 1 128.215.190.161*
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD Zw+bRwIuYctDLbA1SDVYGA==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 128236 1 128.215.63.252*
     # ./dnsanon -i ./test.pcap -c none -f test -a all -k keyfile
    

CHANGES

  • 1.3, 2016-05-20: adds option to write to standard output with -o -
  • 1.4, 2016-06-02: fixes a bug where -p Q failed to output messages with ‘_qr==1’
  • 1.5, 2016-07-18: adds functionality to limit and clean up idle TCP connections (not publically released)
  • 1.6, 2016-09-13: fix to install process to correct install problem when not built in chroot
  • 1.7, 2017-09-19: add new -T option to report TTLs; bug fix: -o and -p; improve docs; ported to openssl-1.1
  • 1.8, 2018-05-14: BUG FIX: fix segamentation fault caused by ldns_wire2pkt with NULL pointer as payload