LANDER:internet address census it100g-20220808 From Predict Jump to navigation Jump to search The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead. README version: 13429, last modified: 2022-09-13. This file describes the trace dataset "internet_address_census_it100g-20220808" provided by the LANDER project. The most recent version of this file can be found on-line at https://wiki.isi.edu/predict/index.php?title=LANDER:internet_address_census_it100g-20220808. [ ] Contents • 1 LANDER Metadata • 2 Dataset Contents • 3 Data Format • 4 Collection Method • 4.1 Probing Location(s) • 4.2 Coverage • 4.3 Beginning/Ending Date and Time Zone • 5 Citation • 6 Results Using This Dataset • 7 User Annotations LANDER Metadata (https://wiki.isi.edu/predict/index.php?title=LANDER:internet_address_census_it100g-20220808/landermeta) ┌─────────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────┐ │dataSetName │internet_address_census_it100g-20220808 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │status │usc-web-and-predict │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │shortDesc │ping census of allocated IPv4 addresses │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │longDesc │To collect this data, an Internet-wide IP address sweep was conducted. Every IP address in the │ │ │ranges allocated by IANA was pinged once by sending ICMP ECHO_REQUEST (PING) packet. If the │ │ │response (ICMP_ECHO_REPLY) came, its IP address was recorded in this data-set. In all, over 2.5│ │ │billion distinct IP addresses were probed during this experiment. │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │datasetClass │Quasi-Restricted │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │commercialAllowed │true │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │requestReviewRequired │true │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │productReviewRequired │false │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │ongoingMeasurement │false │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │submissionMethod │Upload │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │collectionStartDate │2022-08-08 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │collectionStartTime │23:45:07 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │collectionEndDate │2022-09-09 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │collectionEndTime │21:02:27 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │availabilityStartDate │2022-09-14 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │availabilityStartTime │00:00:00 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │availabilityEndDate │2030-01-01 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │availabilityEndTime │00:00:00 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │anonymization │none │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │archivingAllowed │false │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │keywords │category:address-space-status-data, subcategory:internet-census-and-survey-data, ip-address, │ │ │sweep, address-collection, ping, icmp, one-time │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │format │binary │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │access │https │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │hostName │USC-LANDER │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │providerName │USC │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │groupingId │internet address censuses │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │groupingSummaryFlag │false │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │retrievalInstructions │download │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │byteSize │17147363328 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │expirationDays │14 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │uncompressedSize │90337499299 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │impactDoi │ │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │useAgreement │dua-ni-160816 │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │irbRequired │false │ ├─────────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────┤ │privateAccessInstructions│See http://www.isi.edu/ant/traces/index.html#getting_datasets for information on obtaining this│ │ │dataset. │ │ │See https://wiki.isi.edu/predict/index.php?title=LANDER:internet_address_census_it100g-20220808│ │ │for details on this dataset. │ └─────────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────┘ Dataset Contents internet_address_census_it100g-20220808.README.txt      copy of this README iana--ipv4-address-space.{txt,fsdb} iana allocations used for probing data/     it.100.hostname.bz2 binary data file     .sha1sum SHA-1 checksum pcap/ (where available)     it.100.hostname.pcap.bz2 pcap traces of "non-conformant" responses     .sha1sum SHA-1 checksum info/ (where available)     subnet_stats.slash-16.fsdb stats computed over dataset in FSDB text format     subnet_stats.slash-16.png pretty picture of the above     summary.txt summary of IP address usage in human readable form "iana--ipv4-address-space.*" is the iana allocations file used for probing. Note: we're in the process of transitioning from reporting the raw IANA allocations in text format (.txt) file to an FSDB-formatted file. The .fsdb file has several columns, most importantly iana_prefix (e.g. 008/8, despite leading zeroes the number is in decimal), and probing. The value of of that column true or false determines whether or not the corresponding prefix was probed or not. Subdirectory "data" contains four bzipped binary files containing probe records. Each file is named after probing machine that was collecting data. These machines have statically assigned IP addresses: pinger-g1.unnamed.aueb.gr 195.251.255.68 The machine in the file name can be followed by a number. It means that the output data from the machine was broken up into several files and the sub-files are numbered sequentially, starting from 1. The address space was divided among probing machines in a mutual exclusive way. The division was such that each /24 subnet was probed by a single machine. The file ".sha1sum" contains SHA1 checksums of individual compressed files. The integrity of the distribution thus can be checked by independently calculating SHA1 sums of files and comparing them with those listed in the file. If you have the sha1sum utility installed on your system, you can do that by executing: sha1sum --check .sha1sum This has to be done before files are uncompressed. Subdirectory "pcap" (if present) contains corresponding bzipped pcap files containing ICMP packets received by the prober other than types 0 (ECHO_REPLY) and 3 (DEST_UNREACH). Subdirectory "info" contains stats computed over /16s in the dataset. Data Format Binary format of trace files is described in detail here: http://www.isi.edu/ant/traces/topology/address_surveys/binformat_description.html Collection Method Data collection involves pinging all allocated addresses. A full description of this method is in: > John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. > Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, p.169-182. > Vouliagmeni, Greece, ACM. October, 2008 http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html. Probing Location(s) The probing locations for censuses and surveys are indicated in their names. A "w" means west, which is from isi.edu in Marina del Rey, California; "c" means center, from colostate.edu, in Ft. Collins, Colorado; "e" means east which is from east.isi.edu, from Arlington, Virginia; "j" means Japan, which is from WIDE in Fujisawa-shi, Kanagawa, Japan; "g" stands for AUEB, located in Athens, Greece; "n" stands for Utrecht, The Netherlands. See Dataset Contents for more information on the system who did the survey. Coverage Earliest censuses started before June 2007, i.e. up to and including it17, did not attempt to probe addresses with the last octet either 0 or 255, e.g. x.y.z.0 (/24 subnet address) or x.y.z.255 (/24 broadcast). Censuses started between September 2007 and June 2008, i.e. it18-it21, probed all values of the last octet, except .255. Censuses started after September 2008, starting from it22, probed all values of the last octet, including both .0 and .255. Beginning/Ending Date and Time Zone Dates/Times specified in the metadata are in UTC. Earlier censuses (before it37) used local time in their metadata description. Their metadata will be updated to effectively switch to UTC in the near future. Citation If you use this trace to conduct additional research, please cite it as: Internet Addresses Census dataset, IMPACT ID: USC-LANDER/internet_address_census_it100g-20220808/rev13429 . Traces taken 2022-08-08 to 2022-09-09. Provided by the USC/LANDER project (http://www.isi.edu/ant/lander). Results Using This Dataset Traces similar to this one containing collections of "live" IP addresses have been used the following previously published work: • John Heidemann, Yuri Pradkin, Ramesh Govindan, Christos Papadopoulos, Genevieve Bartlett, and Joseph Bannister. Census and Survey of the Visible Internet. In Proceedings of the ACM Internet Measurement Conference, p.169-182. Vouliagmeni, Greece, ACM. October, 2008 http://www.isi.edu/~johnh/PAPERS/Heidemann08c.html. • Yuri Pryadkin, Robert Lindell, Joseph Bannister, and Ramesh Govindan An Empirical Evaluation of IP Address Space Occupancy Technical Report ISI-TR-2004-598, USC/Information Sciences Institute, November 2004 ftp://ftp.isi.edu/isi-pubs/tr-598.pdf. • Lin Quan, John Heidemann, Yuri Pradkin. Detecting Internet Outages with Precise Active Probing (extended). Technical Report ISI-TR-2012-678b, USC/Information Sciences Institute, May, 2012 ftp://ftp.isi.edu/isi-pubs/tr-678b.pdf. User Annotations Suggestion: Edit the annotations at https://wiki.isi.edu/predict/index.php?title=LANDERNOTES:internet_address_census_it100g-20220808&action=edit Currently no annotations. Retrieved from "https://wiki.isi.edu/predict/index.php?title=LANDER:internet_address_census_it100g-20220808&oldid=13429" Categories: • Datasets • LANDER • LANDER:Datasets • LANDER:Datasets:AddressSpace:Census • LANDER:Datasets:AddressSpace Navigation menu Personal tools • Wikiexport • Talk • Preferences • Watchlist • Contributions • Log out Namespaces • LANDER • Discussion [ ] English Views • Read • Edit • View history • Watch [ ] More • Move _____________________ [ Search ] [ Go ] Navigation • Main page • Providers • Datasets • Results • Categories • Recent changes • Random page • Help Tools • What links here • Related changes • Upload file • Special pages • Permanent link • Page information • This page was last edited on 13 September 2022, at 17:19. • Content is available under Attribution-Share Alike 3.0 Unported unless otherwise noted. • Privacy policy • About Predict • Disclaimers • Attribution-Share Alike 3.0 Unported • Powered by MediaWiki