LANDER:internet address census it49w-20120731 From Predict README version: 3171, last modified: 2012-09-20. This file describes the trace dataset "internet_address_census_it49w-20120731" provided by the LANDER project. [IMG] Warning: This dataset has been marked as possibly damaged; see a full description in User Annotations Contents • 1 LANDER Metadata • 2 Dataset Contents • 3 Data Format • 4 Collection Method • 4.1 Probing Location(s) • 4.2 Coverage • 4.3 Beginning/Ending Date and Time Zone • 5 Citation • 6 Results Using This Dataset • 7 User Annotations • 7.1 Non-standard census LANDER Metadata ┌───────────────────────────┬────────────────────────────────────────────────────────────────────────────────────┐ │ dataSetName │ internet_address_census_it49w-20120731 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ status │ usc-web-and-predict │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ shortDesc │ ping census of allocated IPv4 addresses │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ longDesc │ To collect this data, an Internet-wide IP address sweep was conducted. Every IP │ │ │ address in the ranges allocated by IANA was pinged once by sending ICMP │ │ │ ECHO_REQUEST (PING) packet. If the response (ICMP_ECHO_REPLY) came, its IP address │ │ │ was recorded in this data-set. In all, over 2.5 billion distinct IP addresses were │ │ │ probed during this experiment. │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ datasetClass │ Quasi-Restricted │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ commercialAllowed │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ requestReviewRequired │ true │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ productReviewRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ ongoingMeasurement │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ submissionMethod │ Upload │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartDate │ 2012-07-31 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionStartTime │ 17:26:26 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndDate │ 2012-09-02 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ collectionEndTime │ 20:43:19 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartDate │ 2012-09-19 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityStartTime │ 19:14:12 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndDate │ 2030-01-01 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ availabilityEndTime │ 00:00:00 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ anonymization │ none │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ archivingAllowed │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ keywords │ category:address-space-status-data, subcategory:internet-census-and-survey-data, │ │ │ ip-address, sweep, address-collection, ping, icmp, one-time │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ format │ binary │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ access │ https │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ hostName │ USC-LANDER │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ providerName │ USC │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingId │ internet address censuses │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ groupingSummaryFlag │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ retrievalInstructions │ download │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ byteSize │ 17349738496 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ expirationDays │ 14 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ uncompressedSize │ 104289747014 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ impactDoi │ 10.23721/109/1353807 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ useAgreement │ dua-ni-160816 │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ irbRequired │ false │ ├───────────────────────────┼────────────────────────────────────────────────────────────────────────────────────┤ │ privateAccessInstructions │ See http://www.isi.edu/ant/traces/index.html#getting_datasets for information on │ │ │ obtaining this dataset. │ │ │ See │ └───────────────────────────┴────────────────────────────────────────────────────────────────────────────────────┘ Dataset Contents internet_address_census_it49w-20120731.README.txt      copy of this README iana--ipv4-address-space.{txt,fsdb} iana allocations used for probing data/     it.49.hostname.bz2 binary data file     .sha1sum SHA-1 checksum pcap/ (where available)     it.49.hostname.pcap.bz2 pcap traces of "non-conformant" responses     .sha1sum SHA-1 checksum info/ (where available)     subnet_stats.slash-16.fsdb stats computed over dataset in FSDB text format     subnet_stats.slash-16.png pretty picture of the above     summary.txt summary of IP address usage in human readable form "iana--ipv4-address-space.*" is the iana allocations file used for probing. Note: we're in the process of transitioning from reporting the raw IANA allocations in text format (.txt) file to an FSDB-formatted file. The .fsdb file has several columns, most importantly iana_prefix (e.g. 008/8, despite leading zeroes the number is in decimal), and probing. The value of of that column true or false determines whether or not the corresponding prefix was probed or not. Subdirectory "data" contains four bzipped binary files containing probe records. Each file is named after probing machine that was collecting data. These machines have statically assigned IP addresses: toe.isi.edu           128.9.160.251 pinger-w1.ant.isi.edu 206.117.25.88 The machine in the file name can be followed by a number. It means that the output data from the machine was broken up into several files and the sub-files are numbered sequentially, starting from 1. The address space was divided among probing machines in a mutual exclusive way. The division was such that each /24 subnet was probed by a single machine. The file ".sha1sum" contains SHA1 checksums of individual compressed files. The integrity of the distribution thus can be checked by independently calculating SHA1 sums of files and comparing them with those listed in the file. If you have the sha1sum utility installed on your system, you can do that by executing: sha1sum --check .sha1sum This has to be done before files are uncompressed. Subdirectory "pcap" (if present) contains corresponding bzipped pcap files containing ICMP packets received by the prober other than types 0 (ECHO_REPLY) and 3 (DEST_UNREACH). Subdirectory "info" contains stats computed over /16s in the dataset. Data Format Binary format of trace files is described in detail here: http://www.isi.edu/ant/traces/topology/address_surveys/binformat_description.html Collection Method Data collection involves pinging all allocated addresses. A full description of this method is in: > John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. > Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, p.169-182. > Vouliagmeni, Greece, ACM. October, 2008 http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html. Probing Location(s) The probing locations for censuses and surveys are indicated in their names. A "w" means west, which is from isi.edu in Marina del Rey, California; "c" means center, from colostate.edu, in Ft. Collins, Colorado; "e" means east which is from east.isi.edu, from Arlington, Virginia; "j" means Japan, which is from WIDE in Fujisawa-shi, Kanagawa, Japan; "g" stands for AUEB, located in Athens, Greece; "n" stands for Utrecht, The Netherlands. See Dataset Contents for more information on the system who did the survey. Coverage Earliest censuses started before June 2007, i.e. up to and including it17, did not attempt to probe addresses with the last octet either 0 or 255, e.g. x.y.z.0 (/24 subnet address) or x.y.z.255 (/24 broadcast). Censuses started between September 2007 and June 2008, i.e. it18-it21, probed all values of the last octet, except .255. Censuses started after September 2008, starting from it22, probed all values of the last octet, including both .0 and .255. Beginning/Ending Date and Time Zone Dates/Times specified in the metadata are in UTC. Earlier censuses (before it37) used local time in their metadata description. Their metadata will be updated to effectively switch to UTC in the near future. Citation If you use this trace to conduct additional research, please cite it as: Internet Addresses Census dataset, IMPACT ID: USC-LANDER/internet_address_census_it49w-20120731/rev3171 . Traces taken 2012-07-31 to 2012-09-02. Provided by the USC/LANDER project (http://www.isi.edu/ant/lander). Results Using This Dataset Traces similar to this one containing collections of "live" IP addresses have been used the following previously published work: • John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, p.169-182. Vouliagmeni, Greece, ACM. October, 2008 http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html. • Yuri Pryadkin, Robert Lindell, Joseph Bannister, and Ramesh Govindan An Empirical Evaluation of IP Address Space Occupancy Technical Report ISI-TR-2004-598, USC/Information Sciences Institute, November 2004 ftp://ftp.isi.edu/isi-pubs/tr-598.pdf. • Lin Quan, John Heidemann, Yuri Pradkin. Detecting Internet Outages with Precise Active Probing (extended). Technical Report ISI-TR-2012-678b, USC/Information Sciences Institute, May, 2012 ftp://ftp.isi.edu/isi-pubs/tr-678b.pdf. User Annotations Non-standard census This survey is different from previous ones in two respects: 1. Each ICMP echo_request packet sent had a 4 byte payload to see if the presense of a payload increases the chances of passing through firewalls. The answer was that it doesn't, since the fraction of positive responses (echo_replies) is pretty much the same as in LANDER:internet_address_census_it48w-20120615. 2. A BUG in the code discarded many icmp_unreachables as too short if the payload was not included in the original echo_request. Thus, this dataset contains significantly fewer icmp unreachables than other censuses. Yuri (talk) 07:56, 20 September 2012 (PDT) Categories: • Datasets • LANDER • LANDER:Datasets • LANDER:Datasets:AddressSpace:Census • LANDER:Datasets:AddressSpace