Dnsanon: extract DNS traffic from pcap to text with optionally anonymization

dnsanon

Dnsanon reads pcap files, extracts DNS traffic, and writes it to several plain-text tables (in Fsdb format). Optionally it anonymizes the IP addresses and queries.

Pre-built versions for RPM (Fedora, CentOS, REHL): see https://copr.fedorainfracloud.org/coprs/johnh/dnsanon/

Packages and libraries

We package dnsanon for Fedora/REHL/CentOS:

Building it under Fedora and CentOS requires libtrace, which provide as an external package at our libtrace package page.

Documentation

% dnsanon(1) % Liang Zhu liangzhu@isi.edu, John Heidemann johnh@isi.edu % October 22, 2016

NAME

dnsanon - extract DNS queries and responses from network trace file to text files with optional anonymization

SYNOPSIS

dnsanon [-p OPTION] [-a OPTION] [-k KEYFILE] [–pass4 BITS] [–pass6 BITS] [-i PATH] [-o PATH] [-f PREFIX] [-c CMP] [-t NUMBER] [-d NUMBER] [-l NUMBER] [-v] [-x] [-q]

DESCRIPTION

Dnsanon reads network trace files (any format accepted by libtrace), extracts DNS traffic, and writes it to several plain-text tables (in Fsdb format). Optionally it anonymizes the IP addresses and queries.

OPTIONS

-p/--parse OPTION
output format, availeable options are mqreQR: report DNS m)essages, and q)uestion and r)esponses sections, each as separate tables. Capital Q and R includes messages information to the question and response tables. Default is mqr (all three, non-unified)

Note that these options select what sections are reported, and not the status of the QR bit.

-a/--anon OPTION
anonymization, availeable options are id,addr,server-addr,client-addr,name,all (separated by comma)

id: DNS id in packet header

name: query and response content

addr: server address (server-addr) and client address (client-addr). Address with port 53 is identified as server address. If both src and dst port are 53, both of the address are anonymized (anonymized addresses are suffixed with *).

all: all of the above

-k/--key KEYFILE
specify the key file. KEYFILE is created if it doesn’t exist
--pass4 BITS
preserve BITS high bits of IPv4 addrsss as un-anonymized; default is 0, anonymizing all bits if anon is set
--pass6 BITS
similar to –pass4, but for IPv6 address
-i/--input PATH
specify input file path; required
-o/--output-path PATH
specify output path which should be directory; default is current directory. If given “-“ and the output is only one file, it will go to standard output. stdout (-) is incompatible with multi-file output. With stdout, -p is required and you can only specify one option for -p
-c/--compressor COMP
compressor, default is ‘/usr/bin/xz’, use “none” to disable
-f/--output-prefix PREFIX
prefix of the output file, default is timestamp of the first packet
-v/--verbose
verbose log; default is none
-x/--tcp-relax
optimistically try TCP reassembly even without seeing TCP SYN
-t/--tcp-timeout NUMBER
specify timeout for idle TCP connections used in TCP reassembly, default is 60s. Longer idle TCP stream will be discarded. 0 for no timeout.
-l/--tcp-list-limit NUMBER
specify the size limit of internal list to keep track of tcp states for TCP reassembly, used to control memory consumption
-d/--tcp-check-period NUMBER
specify the time period to clean up idle TCP connections, default is 60s

EXAMPLES

Assume test.pcap contains one DNS query and one response for “www.isi.edu A”.

  1. output query, responses and message sections (default)

     ./dnsanon -i test.pcap
    

    output files:

     1461102303.357871.message.fsdb.xz
     1461102303.357871.question.fsdb.xz
     1461102303.357871.rr.fsdb.xz
    
     1461102303.357871 is the timestamp for the first packet in the pcap file
    
  2. default output but with output file prefix and no compression

     ./dnsanon -i ./test.pcap -c none -f test
    

    output files:

     test.message.fsdb
     test.question.fsdb
     test.rr.fsdb
    
  3. output only query sections

     ./dnsanon -i ./test.pcap -c none -f test -p q
    

    output file:

     test.question.fsdb
    

    with content:

     #fsdb -F t msgid id name type class
     1	  10888	     www.isi.edu. A	IN
     2	  10888	     www.isi.edu. A	IN
     # ./dnsanon -i ./test.pcap -c none -f test -p q
    

    output fsdb schema:

     msgid: DNS message id in the network trace file
     id: the id in DNS header
     name: query name
     type: query type
     class: query class
    
  4. output only response sections

     ./dnsanon -i ./test.pcap -c none -f test -p r
    

    output file:

     test.rr.fsdb
    

    with content:

     #fsdb -F t msgid id section_name name type class ttl rdf_num rdata
     2     10888	 ANS		 www.isi.edu.	 A   IN	     1746	1	128.9.176.20
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	nitro.isi.edu.
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	vapor.isi.edu.
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	ns.east.isi.edu.
     2     10888	 AUT		 isi.edu.	 NS  IN	     7143	1	ns.isi.edu.
     2     10888	 ADD		 ns.isi.edu.	 A   IN	     28434	1	128.9.128.127
     2     10888	 ADD		 ns.east.isi.edu.    A	     IN		28434	1	65.114.168.20
     2     10888	 ADD		 nitro.isi.edu.	     A	     IN		128236	1	128.9.208.207
     2     10888	 ADD		 vapor.isi.edu.	     A	     IN		128236	1	128.9.64.64
     # ./dnsanon -i ./test.pcap -c none -f test -p r
    

    output fsdb schema:

     msgid: DNS message id in the pcap file
     id: the id in DNS header
     section_name: AUS=>ANSWER SECTION; AUT=>AUTHORITY SECTION; ADD=>ADDITIONAL SECTION
     name: name of the resource records 
     type: type of the resource records
     class: class of the resource records
     ttl: ttl of the resource records
     rdf_num: number of fields in rdata
     rdata: specific data of the resource records
    
  5. output only message information

     ./dnsanon -i ./test.pcap -c none -f test -p m
    

    output file:

     test.message.fsdb
    

    with content:

     #fsdb -F t msgid time srcip srcport dstip dstport protocol id qr opcode aa tc rd ra z ad cd rcode qdcount ancount nscount arcount edns_present edns_udp_size edns_extended_rcode edns_version edns_z msglen
     1     1461102303.357871     192.168.1.2   59008   192.168.1.1 53 udp	 10888 0  0  0 0  1  001   0	   0	   1	   0	   0		0	      1			  4096	       0      0	0	48
     2     1461102303.362905     192.168.1.1   53	   192.168.1.2 59008	 udp   10888 1 0  0  0	   1	   100	   0	   0	   1		1	      4			  4	       1      4096	0	0	0	207
     # ./dnsanon -i ./test.pcap -c none -f test -p m
    
  6. output query sections with combined message information

     ./dnsanon -i ./test.pcap -c none -f test -p Q
    

    output file:

     test.message_question.fsdb
    

    with content:

     #fsdb -F t msgid time srcip srcport dstip dstport protocol id qr opcode aa tc rd ra z ad cd rcode qdcount ancount nscount arcount edns_present edns_udp_size edns_extended_rcode edns_version edns_z msglen name type class
     1	1461102303.357871	192.168.1.2	59008	192.168.1.1	53	udp	10888	0	0	0	0	1	0	0	1	0	0	1	0	0	0	1	4096	0	0	0	48	www.isi.edu.	A	IN
     2	1461102303.362905	192.168.1.1	53	192.168.1.2	59008	udp	10888	1	0	0	0	1	1	0	0	0	0	1	1	4	4	1	4096	0	0	0	207	www.isi.edu.	A	IN
     # ../dnsanon -i ./test.pcap -c none -f test -p Q
    
  7. anonymization

     ./dnsanon -i ./test.pcap -c none -f test -p m -a all -k keyfile
    

    Full anonymization:

    IP is prefix-preserve anonymized and suffixed with *.

    ID in DNS header is encrpyted and output is base64 string.

    Each part of the names in resource records is encrpyted and output is base64 string.

    sample output files:

    test.message.fsdb:

     #fsdb -F s msgid time srcip srcport dstip dstport protocol id qr opcode aa tc rd ra z ad cd rcode qdcount ancount nscount arcount edns_present edns_udp_size edns_extended_rcode edns_version edns_z msglen
     1 1461102303.357871 210.91.230.223* 59008 210.91.230.220* 53 udp GMgZKqG94x3W7FJ0yrcJZg== 0 0 0 0 1 0 0 1 0 0 1 0 0 0 1 4096 0 0 0 48
     2 1461102303.362905 210.91.230.220* 53 210.91.230.223* 59008 udp GMgZKqG94x3W7FJ0yrcJZg== 1 0 0 0 1 1 0 0 0 0 1 1 4 4 1 4096 0 0 0 207
     # ./dnsanon -i ./test.pcap -c none -f test -p m -a all -k keyfile
    

    test.question.fsdb:

     #fsdb -F s msgid id name type class
     1 GMgZKqG94x3W7FJ0yrcJZg== hiALV6+e9T9ni1ZJyCD35Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN
     2 GMgZKqG94x3W7FJ0yrcJZg== hiALV6+e9T9ni1ZJyCD35Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN
     # ./dnsanon -i ./test.pcap -c none -f test -a all -k keyfile
    

    test.rr.fsdb:

     #fsdb -F s msgid id section_name name type class ttl rdf_num rdata
     2 GMgZKqG94x3W7FJ0yrcJZg== ANS hiALV6+e9T9ni1ZJyCD35Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 1746 1 128.215.201.242*
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 PoQD5IM57swzKjlFkC+OEw==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 Zw+bRwIuYctDLbA1SDVYGA==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 DswjzZnRq9/c5+jPctKr2Q==.ofRFtyXN6AeIHlBAb3SFCQ==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== AUT kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. NS IN 7143 1 DswjzZnRq9/c5+jPctKr2Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==.
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD DswjzZnRq9/c5+jPctKr2Q==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 28434 1 128.215.225.32*
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD DswjzZnRq9/c5+jPctKr2Q==.ofRFtyXN6AeIHlBAb3SFCQ==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 28434 1 82.117.24.134*
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD PoQD5IM57swzKjlFkC+OEw==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 128236 1 128.215.190.161*
     2 GMgZKqG94x3W7FJ0yrcJZg== ADD Zw+bRwIuYctDLbA1SDVYGA==.kaWqOoja4H2SFZaeXVHdhg==.JkJ9ngv//NKYi1GIrfEY2A==. A IN 128236 1 128.215.63.252*
     # ./dnsanon -i ./test.pcap -c none -f test -a all -k keyfile
    

CHANGES