LANDER:internet outage adaptive a23c-20160101 From Predict README version: 5375, last modified: 2016-06-8. This file describes the trace dataset "internet_outage_adaptive_a23c-20160101" provided by the LANDER project. Contents • 1 LANDER Metadata • 2 Dataset Contents • 3 Data Format • 3.1 data • 3.2 blocklist • 3.3 iplist • 3.4 site_rate • 4 Collection Method • 4.1 Probing Location(s) • 4.2 Beginning/Ending Date and Time Zone • 5 Citation • 6 User Annotations LANDER Metadata ┌───────────────────────────┬────────────────────────────────────────────────────────────────────────────────────┐ │ dataSetName │ internet_outage_adaptive_a23c-20160101 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ status │ usc-web-and-predict │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ shortDesc │ reachability of responsive of IPv4 /24s │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ longDesc │ To collect this data, all allocated and responsive Internet IP address blocks were │ │ │ pinged by sending ICMP ECHO_REQUEST (PING) packet. The block status was recorded │ │ │ in this data set. Probe was repeated every 11 minutes, and more quickly when │ │ │ uncertain about block status. In all, approximately 3.5M /24 subnets were │ │ │ periodically probed. │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ datasetClass │ Quasi-Restricted │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ commercialAllowed │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ requestReviewRequired │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ productReviewRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ ongoingMeasurement │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ submissionMethod │ Upload │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartDate │ 2016-01-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartTime │ 01:26:46 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndDate │ 2016-04-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndTime │ 01:38:02 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartDate │ 2016-10-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndDate │ 2030-01-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ anonymization │ none │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ archivingAllowed │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ keywords │ category:address-space-status-data, subcategory:network-outage, ip-address, sweep, │ │ │ address-collection, ping, icmp │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ format │ text │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ access │ https │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostName │ USC-LANDER │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ providerName │ USC │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingId │ internet outages datasets │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingSummaryFlag │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ retrievalInstructions │ download │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ byteSize │ 591905423360 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ expirationDays │ 14 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ uncompressedSize │ 2852144801996 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ impactDoi │ 10.23721/109/1354153 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ useAgreement │ dua-ni-160816 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ irbRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ privateAccessInstructions │ See https://ant.isi.edu/datasets/#getting-datasets for information on obtaining │ │ │ this dataset. │ │ │ See │ └───────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘ Dataset Contents internet_outage_adaptive_a23c-20160101.README.txt     copy of this README data/     ...     .e.a..r..fsdb.bz2     data files     ...     .sha*sum     SHA checksum input/     _blacklist.a.bz2     blacklist (where available)     ips_all.full.blocklist.part..bz2     trinocular input block information     ips_all.full.iplist.part..bz2     trinocular input list of blocks (binary)     ips_all.full.text.part..bz2     trinocular input list of blocks (text)     .sha*sum     SHA checksum info/     site_rate-a.fsdb     hourly probe and response rates (where available) In the above: is the short hostname of the pinger is the standard unix seconds since the epoch timestamp of the first record in the file is the sequence number of the survey site, e.g. w=isi west, c=colorado state, g=greece, etc. index of the input "part" (we do 4 or 5-way probing, dividing all blocks into 4 or 5 parts) index of the 11-min "round" sequence number of the file Data Format data Data is in FSDB format with the following schema: #fsdb -F t block round_no round_start_epoch a_short a_oper status belief n_pos n_neg (A7 has a reduced schema, only: #fsdb block round ts A status belief n_pos n_neg) • block: hex format of /24 IP block, with trailing zeros (A7 omits trailing zeros) • round_no: round number in this batch (will reset each time we restart) • round_start_epoch: when the round began, in seconds since 1970 • a_short: the short term estimate of availability • a_oper: the operational estimate of A value (long term and reflecting variance) • status: status of this block • for A7, text UP/DOWN/UNKNOWN • for A12 and later: 0 for down, 1 for up, 2 for unknown • belief: our belief the block is down • n_pos: number of positive responses in this round • n_neg: number of negative responses in this round Latest adaptive probing datasets add additional fields: #fsdb -F t block round_no round_start_epoch a_short a_oper status belief n_pos n_neg probe_log rtt_us where • probe_log: uuencoded sequence of last octets of the block probed within this round • rtt_us: RTT to the last successful ping to this block within this round blocklist blocklist is a text format. FSDB for a15 and later. (before a15 space-separated and without a header) Sample line is: #fsdb prefix status belief_down nresp_census apos_s atot_s apos_l atot_l a_oper a_absdev_l Eb_cnt Eb_list 0104fd00 UP 0.05 128 1 1.99218749999891 1 1.99218749999891 0.250980392157 0.0 255 5a,f6,0f,71,53,96,e1,1e,e5,61,98,9e,7c,3d,87,fd,6a,68,e6,6b,f3,be,b6,af,a4,bf,08,3f,8f,95,ef,56,74,5e,1f,1b,73,1d,12,18,0c,ae,9d,f7,6f,19,0e,f9,11,dc,92,2b,16,57,0b,54,bc,81,a5,40,eb,24,77,ba,42,ff,44,fb,2c,a2,65,78,62,20,9f,46,a7,8c,5f,89,7d,55,bd,63,15,38,e8,6e,b5,ee,52,99,47,d2,27,f1,c3,b0,9c,2d,5d,90,05,ad,c8,7a,df,50,0a,06,70,28,c9,82,4d,fc,ed,51,ec,04,d9,93,07,a8,32,48,a1,21,4e,02,da,3c,c1,b4,36,de,9b,94,49,86,03,cb,c6,39,f4,e3,79,b2,8d,10,91,22,75,d8,db,d1,9a,3b,ac,c0,80,4b,23,8b,6d,25,1c,6c,c2,97,4c,e9,d3,d4,bb,4a,e4,dd,a9,00,e2,35,8a,01,4f,0d,2f,69,b8,d7,f5,ab,64,41,14,09,d0,85,60,84,aa,d6,88,b3,f2,f8,34,d5,b7,8e,7f,58,ca,43,17,2a,66,33,37,45,cf,31,26,f0,76,b1,83,13,cc,5c,2e,1a,b9,5b,ce,72,29,a0,fa,a3,cd,e0,3a,a6,e7,7e,3e,30,ea,c5,c7,c4,59,7b,67 where fields are • prefix, in hex, (before a15 without trailing zeros) • status of the block (UP or DOWN or UNKNOWN; initialized to UP) • belief_down - our estimated probability the block is down (Bayes) • nresp_census - number of responses in the last 4 censuses • apos_s - running (short) average of positive responses (initialized to 1) • atot_s - running (short) average of total probes sent (initialized to 1/historical_avalability) • apos_l - running (long) average of positive responses (initialized to 1) • atot_l - running (long) average of total probes sent (initialized to 1/historical_avalability) • a_oper - operational value of A for this block (initialized to 1/2 of historical_availability) • a_absdev_l - absolute deviation of A, long term for the block • Eb_cnt - count of responsive IPs • Eb_list - last octets of active IP addresses to probe, in hex, separated by whitespace or commas iplist Iplists are a binary format used internally by icmptrain, our probing program. It defines the probing order in which pings are scheduled. The blocklist includes more information than the iplist. See icmptrain for iplist format. site_rate To better diagnose site-wide problems we started to include site_rate files with our datasets. These files record hourly counts of pings sent and received. FSDB headers are: • sitebin_start_epoch - timestamp of the hour • n_pos_sum - count of positive replies • n_neg_sum - count of non-responses (or negative responses) • n_probe_sum - total count of probes (= n_pos_sum + n_neg_sum) • pos_probe_fraction - positive reply rate (= n_pos_sum / n_probe_sum) Collection Method Data collection involves pinging allocated and responsible address blocks. Full description of this method is in: > Lin Quan, John Heidemann and Yuri Pradkin. Trinocular: Understanding Internet Reliability Through Adaptive > Probing. SIGCOMM 2013. August, 2013 http://www.isi.edu/~johnh/PAPERS/Quan13c.html. > John Heidemann, Lin Quan and Yuri Pradkin. A Preliminary Analysis of Network Outages During Hurricane Sandy. > ISI-TR-685b. November, 2012 http://www.isi.edu/~johnh/PAPERS/Heidemann12d.html . > Lin Quan, John Heidemann and Yuri Pradkin. Detecting Internet Outages with Precise Active Probing (extended). > ISI-TR-678b. February, 2012 http://www.isi.edu/~johnh/PAPERS/Quan12a.html. Probing Location(s) The probing locations for outage surveys are indicated in their names. A "w" means west, which is from isi.edu in Marina del Rey, California; "c" means center, from colostate.edu, in Ft. Collins, Colorado; "e" means east which is from east.isi.edu, from Arlington, Virginia; "g" stands for Greece, hosted by AUEB, in Athens (datasets starting a17); "n" stands for Netherlands, hosted by surfnet.nl in Utrecht (datasets starting a28) See Dataset Contents for more information on the system who did the survey. Beginning/Ending Date and Time Zone Dates/Times specified in the metadata are in UTC. Citation If you use this trace to conduct additional research, please cite it as: Internet Outages dataset, IMPACT ID: USC-LANDER/internet_outage_adaptive_a23c-20160101/rev5375 . Traces taken 2016-01-01 to 2016-04-01. Provided by the USC/LANDER project (http://www.isi.edu/ant/lander). User Annotations Currently no annotations. Categories: • Datasets • LANDER • LANDER:Datasets • LANDER:Datasets:AddressSpace:Adaptive Probing • LANDER:Datasets:AddressSpace • LANDER:Datasets:AddressSpace:Maintenance