LANDER:Root DNS Event-20151130 From Predict README version: 5438, last modified: 2016-07-29. This file describes the trace dataset "Root_DNS_Event-20151130" provided by the LANDER project. Contents • 1 LANDER Metadata • 2 Dataset Contents • 2.1 November 30, 2015 Event • 3 Dataset Format • 3.1 Syntax • 3.2 Schema • 4 Collection Method • 4.1 Public Data • 4.2 Data Cleaning • 4.2.1 how to identify the hijacked probes • 5 How to parse the specific sites and servers • 6 Citation • 7 Results Using This Dataset • 8 User Annotations LANDER Metadata ┌───────────────────────────┬────────────────────────────────────────────────────────────────────────────────────┐ │ dataSetName │ Root_DNS_Event-20151130 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ status │ usc-web-and-predict │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ shortDesc │ DNS CHAOS data for all Root Letters │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ longDesc │ This dataset is the DNS CHAOS measurements towards all DNS Root Letters from Nov │ │ │ 30th to Dec 1th, 2015. The Internet's Root Domain Name Service sustained traffic │ │ │ at 100 times normal load on these two days. Raw data is originally from RIPE Atlas │ │ │ but converted from JSON to CSV and filtered out hijacked probes. │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ datasetClass │ Quasi-Restricted │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ commercialAllowed │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ requestReviewRequired │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ productReviewRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ ongoingMeasurement │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ submissionMethod │ Upload │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartDate │ 2015-11-30 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndDate │ 2015-12-02 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartDate │ 2016-11-30 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndDate │ 2030-01-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ anonymization │ none │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ archivingAllowed │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ keywords │ category:internet-topology-data, subcategory:anycast-enumeration, DNS, CHAOS, │ │ │ Anycast, DDos, internet measurement │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ format │ text │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ access │ https │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostName │ USC-LANDER │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ providerName │ USC │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingId │ │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingSummaryFlag │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ retrievalInstructions │ download │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ byteSize │ 1365245952 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ expirationDays │ 14 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ uncompressedSize │ 6139986194 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ impactDoi │ 10.23721/109/1354160 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ useAgreement │ dua-ni-160816 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ irbRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ privateAccessInstructions │ See https://ant.isi.edu/datasets/#getting-datasets for information on obtaining │ │ │ this dataset. │ │ │ See │ └───────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘ Dataset Contents a.csv.gz     b.csv.gz     c.csv.gz    d.csv.gz e.csv.gz     f.csv.gz     g.csv.gz    h.csv.gz i.csv.gz     j.csv.gz     k.csv.gz    l.csv.gz m.csv.gz Each starting letter of the file name stands for the corresponding RootDNS, e.g. the starting letter "a" in file "a.csv.gz" stands for the A-root server. The dataset after decompression is plain-text files in csv format. Besides the above dataset, we also provide 13 tables(named *rootCHAOSmapping.fsdb) to parse each string(which contains the information about a certain site and a certain server), to reveal the specific site and server that respond to a CHAOS query. Same as above, the starting letter "A" in "ArootCHAOSmapping.fsdb" stands for A-root server. ArootCHAOSmapping.fsdb  BrootCHAOSmapping.fsdb  CrootCHAOSmapping.fsdb DrootCHAOSmapping.fsdb  ErootCHAOSmapping.fsdb  FrootCHAOSmapping.fsdb GrootCHAOSmapping.fsdb  HrootCHAOSmapping.fsdb  IrootCHAOSmapping.fsdb JrootCHAOSmapping.fsdb  KrootCHAOSmapping.fsdb  LrootCHAOSmapping.fsdb MrootCHAOSmapping.fsdb November 30, 2015 Event During two intervals on November 30, 2015 and December 1, 2015, several of the root name servers received up to 5 million queries per second each, for a single undisclosed domain name. Source addresses were spread throughout IPv4 space, however these may have been spoofed. Some root server networks became saturated, resulting in timeouts, however redundancy among the root servers prevented downstream issues from occurring during this incident. The Internet’s Root Domain Name Service, made up of 13 independently designed services, and 11 of them are deployed with IP anycast running at more than 500 sites. This DDoS attack happened at November 30, 2015 and December 1, 2015 provides study materials to research on how anycast service will behave when stressed by sustained traffic at 100 times normal load. The event is recorded at this link Dataset Format Syntax The format for all files is compressed ascii format. The uncompressed file is in csv format composed of columns. Schema Below is an example fragment of the csv files: #src_ip,dst_ip,proto, dst_name, rtt,prb_id, timestamp,rcode 134.197.42.163,198.41.0.4,UDP,[u'nnn1-lax2'],41.385,10008,1448841608,0 84.106.139.124,198.41.0.4,UDP,[u'nnn1-lon3'],16.331,10016,1448841645,0 Each file is a simple database with 8 comma-separated columns in total, which are: ┌───────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ src_addr │ The IP-address of the RIPE Atlas Probe. │ ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ dst_addr │ The IP-address of the DNS-server │ ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ proto │ The transport protocol (UDP or TCP) │ ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Rdata │ The reply to the CHAOS query (hostname.bind), a proof from which we can use to determine the │ │ │ anycast instance the query ended up with(e.g. "lax" =>Los Angeles and "lon" => London). │ ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ rt │ The RTT (units: ms) │ ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ prb_id │ The ID of the Ripe Atlas Probe │ ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ timestamp │ The timestamp in unix time │ ├───────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ rcode │ The DNS response code. 0 (a successful response), or -1 (no response). │ └───────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘ Collection Method Public Data All measurements are originally from RIPE Atlas and the raw data is publicly available at the Atlas website. Data Cleaning • To convert the file format from JSON to CSV, with major fields filtered, and decoding the binary field (answer) to retrieve the hostname.bind for each measurement. • To filter out hijacked probes. how to identify the hijacked probes CHAOS replies contain the meaningless information, so this process requires some manual process. (1)To pull out all the unique strings in the dst_name position, and calculate how many times they each appears. Below is an example for A-root: dst_name times nevarem0 336 nnn1-fra1 1436442 nnn1-hkg5 70904 nnn1-lax2 351207 nnn1-lon3 865092 nnn1-nyc3 228095 ns1 328 (2)Identify the meaningless and low-occurrence string as the garbage string that should be discarded. For example, "nevarem0" and "ns1" should be discarded. All the rest are valid A-root site information. • To split each measurement file first per RCODE, then per probe ID, and also per site and server. How to parse the specific sites and servers For each letter name server, we provide a file named *rootCHAOSmapping.fsdb to parse the dst_name information. The easy-to-parse file is simply a plain-text file with columes delimited by tab, commented by＃. (The file can also be used in FSDB format, if you are interested to use FSDB, see this README Below is an example of an easy-to-parse file ArootCHAOSmapping.fsdb: #fsdb -F t letter dst_name site server count #letter dst_name site server count A nnn1-fra1 fra 1 1436442 A nnn1-lon3 lon 3 865092 A nnn1-lax2 lax 2 351207 A nnn1-nyc3 nyc 3 228095 A nnn1-hkg5 hkg 5 70904 A n - - 2333 A - - - 1254 A cadmium.la.net.ua - - 672 The first colume stands for the letter of the name server, in this case it stands for the A-root name server. The second colume is the dst_name information contained in the csv file, we can parse it to colume three, the site and colume four, the server. For example, the dst_name "nnn1-fra1" can be parsed to the site "fra" and server "1". In most letters, the site can be parsed as a standard aircode. But in some cases, the site string is a bit complicated, such as in the file DrootCHAOSmapping.fsdb, #fsdb -F t letter dst_name site server count #letter dst_name site server count D mcva1.droot mcva 1 958310 D mcva3.droot mcva 3 868489 D mcva2.droot mcva 2 864808 the mcva1.droot, mcva2.droot, mcva3.droot can be recognized as response from the same site "mcva" but different servers"1" ,"2" ,"3". The fifth colume in our easy-to-parse data is traces left from the procedures we fill out the hijacked probes, counting how many times the site information appears. All items are sorted reserve-numerically so the top ones are real site information. Citation If you use this trace to conduct additional research, please cite it as: Emulated attacks in enterprise traffic, PREDICT ID: USC-LANDER/Root_DNS_Event-20151130. Provided by the USC/LANDER project http://www.isi.edu/ant/lander. Results Using This Dataset Some published and submitted work is based on this dataset. • Giovane C. M. Moura, Ricardo de O. Schmidt, John Heidemann, Wouter B. de Vries, Moritz Müller, Lan Wei and Christian Hesselman. "Anycast vs. DDoS: Evaluating the November 2015 Root DNS Event (extended)," submitted for review to the ACM IMC, 2016. • Giovane C. M. Moura, Ricardo de O. Schmidt, John Heidemann, Wouter B. de Vries, Moritz Müller, Lan Wei and Christian Hesselman. "Anycast vs. DDoS: Evaluating the November 2015 Root DNS Event (extended)," Technical Report ISI-TR-2016-709. USC/Information Sciences Institute, May 2016 http://www.isi.edu/~johnh/PAPERS/Moura16a.pdf User Annotations Currently no annotations. Categories: • Datasets • LANDER • LANDER:Datasets • LANDER:Datasets:AddressSpace:Adaptive Probing • LANDER:Datasets:AddressSpace