LANDER:internet address survey it1-20040621 From Predict README version: 1540, last modified: 2010-06-17. This file describes the trace dataset "internet_address_survey_it1-20040621" provided by the LANDER project. [IMG] Warning: This dataset has been marked as possibly damaged; see a full description in User Annotations Contents • 1 LANDER Metadata • 2 Dataset Contents • 3 Data Format • 4 Collection Method • 4.1 Probing Location(s) • 4.2 Coverage • 4.3 Beginning/Ending Date and Time Zone • 5 Citation • 6 Results Using This Dataset • 7 User Annotations • 7.1 A note on the dataset's content LANDER Metadata ┌───────────────────────────┬────────────────────────────────────────────────────────────────────────────────────┐ │ dataSetName │ internet_address_survey_it1-20040621 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ status │ usc-web-and-predict │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ shortDesc │ ping census of allocated IPv4 addresses │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ longDesc │ To collect this data, an Internet-wide IP address sweep was conducted. Every IP │ │ │ address in the ranges allocated by IANA was pinged once by sending ICMP │ │ │ ECHO_REQUEST (PING) packet. If the response (ICMP_ECHO_REPLY) came, its IP address │ │ │ was recorded in this data-set. In all, over 2.5 billion distinct IP addresses were │ │ │ probed during this experiment. │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ datasetClass │ Quasi-Restricted │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ commercialAllowed │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ requestReviewRequired │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ productReviewRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ ongoingMeasurement │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ submissionMethod │ Upload │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartDate │ 2004-06-21 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndDate │ 2004-08-29 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartDate │ 2012-01-27 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartTime │ 17:05:24 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndDate │ 2030-01-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ anonymization │ none │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ archivingAllowed │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ keywords │ category:address-space-status-data, subcategory:internet-census-and-survey-data, │ │ │ ip-address, sweep, address-collection, ping, icmp, one-time │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ format │ binary │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ access │ https │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostName │ USC-LANDER │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ providerName │ USC │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingId │ internet address censuses │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingSummaryFlag │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ retrievalInstructions │ download │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ byteSize │ 768606208 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ expirationDays │ 14 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ uncompressedSize │ 2324179380 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ impactDoi │ 10.23721/109/1353571 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ useAgreement │ dua-ni-160816 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ irbRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ privateAccessInstructions │ See http://www.isi.edu/ant/traces/index.html#getting_datasets for information on │ │ │ obtaining this dataset. │ │ │ See │ └───────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘ Dataset Contents internet_address_survey_it1-20040621.README.txt      copy of this README iana--ipv4-address-space.{txt,fsdb} iana allocations used for probing data/     it.1.doe*.bz2 binary data files     it.1.nil*.bz2     it.1.ptr*.bz2     it.1.toe*.bz2     .sha1sum SHA-1 checksum pcap/ (where available)     it.1.doe*.pcap.bz2 pcap traces of "non-conformant" responses     it.1.nil*.pcap.bz2     it.1.ptr*.pcap.bz2     it.1.toe*.pcap.bz2     .sha1sum SHA-1 checksum info/ (where available)     subnet_stats.slash-16.fsdb stats computed over dataset in FSDB flat text format     subnet_stats.slash-16.png pretty picture of the above     summary.txt summary of IP address usage in human readable form "iana--ipv4-address-space.txt" is the iana allocations file used for probing. Note: we're in the process of transitioning from reporting the raw IANA allocations in text format (.txt) file to an FSDB-formatted file. The .fsdb file has several columns, most importantly iana_prefix (e.g. 008/8, despite leading zeroes the number is in decimal), and probing. The value of of that column true or false determines whether or not the corresponding prefix was probed or not. Subdirectory "data" contains four bzipped binary files containing probe records. Each file is named after probing machine that was collecting data. E.g. it.*.doe.bz2 was collected by machine "doe.isi.edu". These machines have statically assigned IP addresses: doe.isi.edu 128.9.160.251 nil.isi.edu 128.9.160.83 ptr.isi.edu 128.9.168.45 toe.isi.edu 128.9.160.82 The machine in the file name can be followed by a number. It means that the output data from the machine was broken up into several files and the sub-files are numbered sequentially, starting from 1. The address space was divided among probing machines in a mutual exclusive way. The division was such that each /24 subnet was probed by a single machine. The file ".sha1sum" contains SHA1 checksums of individual compressed files. The integrity of the distribution thus can be checked by independently calculating SHA1 sums of files and comparing them with those listed in the file. If you have the sha1sum utility installed on your system, you can do that by executing: sha1sum --check .sha1sum This has to be done before files are uncompressed. Subdirectory "pcap" contains corresponding bzipped pcap files containing ICMP packets received by the prober other than types 0 (ECHO_REPLY) and 3 (DEST_UNREACH). Subdirectory "info" contains stats computed over /16s in the dataset. Data Format Binary format of trace files is described in detail here: http://www.isi.edu/ant/traces/topology/address_surveys/binformat_description.html Collection Method Data collection involves pinging all allocated addresses. A full description of this method is in: > John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. > Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, p.169-182. > Vouliagmeni, Greece, ACM. October, 2008 http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html. Probing Location(s) The probing locations for censuses and surveys are indicated in their names. A "w" means west, which is from isi.edu in Marina del Rey, California; "c" means center, from colostate.edu, in Ft. Collins, Colorado; "e" means east which is from east.isi.edu, from Arlington, Virginia; "j" means Japan, which is from WIDE in Fujisawa-shi, Kanagawa, Japan; "g" stands for AUEB, located in Athens, Greece; "n" stands for Utrecht, The Netherlands. See Dataset Contents for more information on the system who did the survey. Coverage Earliest censuses started before June 2007, i.e. up to and including it17, did not attempt to probe addresses with the last octet either 0 or 255, e.g. x.y.z.0 (/24 subnet address) or x.y.z.255 (/24 broadcast). Censuses started between September 2007 and June 2008, i.e. it18-it21, probed all values of the last octet, except .255. Censuses started after September 2008, starting from it22, probed all values of the last octet, including both .0 and .255. Beginning/Ending Date and Time Zone Dates/Times specified in the metadata are in UTC. Earlier censuses (before it37) used local time in their metadata description. Their metadata will be updated to effectively switch to UTC in the near future. Citation If you use this trace to conduct additional research, please cite it as: Internet Addresses Census dataset, IMPACT ID: USC-LANDER/internet_address_survey_it1-20040621/rev1540 . Traces taken 2004-06-21 to 2004-08-29. Provided by the USC/LANDER project (http://www.isi.edu/ant/lander). Results Using This Dataset Traces similar to this one containing collections of "live" IP addresses have been used the following previously published work: • John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, p.169-182. Vouliagmeni, Greece, ACM. October, 2008 http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html. • Yuri Pryadkin, Robert Lindell, Joseph Bannister, and Ramesh Govindan An Empirical Evaluation of IP Address Space Occupancy Technical Report ISI-TR-2004-598, USC/Information Sciences Institute, November 2004 ftp://ftp.isi.edu/isi-pubs/tr-598.pdf. • Lin Quan, John Heidemann, Yuri Pradkin. Detecting Internet Outages with Precise Active Probing (extended). Technical Report ISI-TR-2012-678b, USC/Information Sciences Institute, May, 2012 ftp://ftp.isi.edu/isi-pubs/tr-678b.pdf. User Annotations A note on the dataset's content This is one of the early censuses which used text files to record information. Later these text files were converted to the 24-byte binary format. In this census only ICMP_ECHO_RESPONSEs were recorded. Hence, the only fields in the binary files that are valid are: Type, Length, ICMP reply type=0, ICMP reply code=0, Probe IP, and Response IP. For completeness, the original text files are included in the raw-subdirectory. It's content is: it.1.doe.txt.gz it.1.nil.txt.gz it.1.ptr.txt.gz it.1.toe.txt.gz Each gzipped text file consists of lines each having 1 to 2 fields. First field is the probed IP address. Second field (if present) records the source IP address of the response, if different from the probed IP. For example: 63.171.43.67 216.135.11.168 216.135.11.169 In the first case, the IP address 63.171.43.67 was pinged and responded; while in the second, 216.135.11.168 was pinged but response came from 216.135.11.169. Pcap data was not captured for this dataset --Yuri 18:42, 2 June 2010 (UTC) Categories: • Datasets • LANDER • LANDER:Datasets • LANDER:Datasets:AddressSpace:Census • LANDER:Datasets:AddressSpace