fighting bit rot in log-term data archives with babarchive

As part of research at ANT we generate a lot of data, and our goal is to keep it safe even in the face of an imperfect world of data storage.

When we say a lot, we mean hundreds of terabytes: As of May 2020, we have releasable 860 datasets making up 134 TB of storage (510TB if we uncompressed it). We provide this data at no cost to researchers, and since 2008 we’ve provided 2049 datasets (338 TB, or 1.1PB if uncompressed!) to 406 researchers!

These datasets range from packet captures of “normal” traffic, to curated captures of DDoS attacks, as well as dozens of research paper-specific datasets, 16 years of Internet censuses and 7 years of Internet outages, plus target lists for IPv4 that are regularly used for traffic studies and tools like Verfploeter anycast mapping.

As part of keeping this data, our goal is to keep this data. We want to fight bit rot and data loss. That means the RAID-6 for primary storage, with monitoring and timely disk replacement. It means off site backup (with a big thanks to our collaborators at Colorado State University, Christos Papadopoulos, Craig Partridge, and Dimitrios Kounalakis for their help). And it means watching bits to make sure they don’t spontaneously change.

One might think that bits at rest stay at rest, but… not always. We’ve seen three times when disks have spontaneously changed a byte over the last 20 years. In 2011 and 2012 I had bit flips on my personal files, and in 2020 we had a byte flip on a packet capture.

How do we know? We have application-level checksums of every file, and every day we take 10 minutes to check at least one dataset against its checksums. (Over time, we cover all datasets and then start all over.)

Our checksumming software is babarchive–our own wrapper around collecting SHA-256 checksums over a directory tree. We encourage other researchers interested in long-term data curation to carry out active content monitoring (in addition to backups and RAID).

A huge thanks to our research sponsors: DHS (through the LANDER, LACREND, and LACANIC projects), NSF (through the MADCAT, MR-Net), and DARPA (through GAWSEED).

Announcements Data Internet

ANT IPv4 census appears in Library of Congress Blog on Innovative Mapping

John Hessler, a member of the US Library of Congress’ Geography and Map Division wrote a nice blog post about our IPv4 Internet maps: “Computing Space V: Mapping the Web or Pinging your Way to Infinity“.  Check out his take on our IPv4 data!

You too can browse the IPv4 Internet at our website.  Or for detailed analysis, get the data from IMPACT or us.

Thanks to the DHS IMPACT program for supporting collection of this data.

Announcements Collaborations Data Internet Outages

welcoming Greece to the ANT Internet Census

We’re happy to welcome Greece to our browsable Internet map at !  Of course Greece has always been in our Internet censuses, but George Xylomenos and George Polyzos of the Athens University of Economics and Business (their lab) helped set up a new observation site.  Greece now provides a new vantage point for Internet censuses.

The differences in the census are small, as one would hope, since it’s a global Internet.  However, when we look at latency (the time it takes for an IP address to reply to our requests), Greece gives us a European view.

Compare the lower-left corner of the Internet, since that is European IPv4 address space:

it61g RTTs
Round-trip times from our Greek vantage point (in to the world. Observe that European IP addresses in the lower left corner are nearby (light colored).

it61w RTTs
Round-trip times from our Los Angeles-based vantage point (at to the world. Observe that European IP addresses in the lower left corner are distant (darker gray).

In addition to big thanks to George Xylomenos and George Polyzos of AUEB (σας ευχαριστώ!) and AUEB for institutional funding for this work.  We also thank Christos Papadopoulos (Colorado State) for helping with many details, and Colin Perkins (U. Glasgow) for discussions about potential European hosts.

Data from our Greece census is available to researchers at no cost on the same terms as our existing census data.  See our datasets page for details. Greek data starts with it61 as of 2014-08-29.

Announcements Data

Complete IPv4 geolocation dataset now available


We recently finished the work of geolocating all IPv4 addresses and plotted a “complete IP geolocation map“.

This work is based on our previous IMC paper “Towards Geolocation of Millions of IP Addresses“, joint work of Zi Hu, John Heidemann, and Yuri Pradkin.

Processed data from this work is visible on our browsable web map.  The raw data from this effort is available through PREDICT or from the authors.