Network outages vary in scope and cause, from the Egyptian Internet shutdown in Feb. 2011 and natural disasters such as October 2012 Hurricane Sandy and the Mar. 2011 Japanese earthquake, to the thousands of small, daily outages.
Outages matter because they allow us to judge the reliability of the Internet directly, and they sometimes allow us to infer things about the real world (like how widespread was a natural disaster).
We use Trinocular, our active probing system to track outages across the Internet, adaptively probing all /24 address blocks where at least 15 address reply to pings (as of 2016: that’s about 4.1M blocks!). We develop new algorithms to identify outages and cluster them to events, providing the first visualization of outages. Finally, we report on Internet stability as a whole, and the size and duration of typical outages, using core-to-edge observations. We find that about 0.3% of the Internet is likely to be unreachable at any time, suggesting the Internet provides only 2.5 ``nines’’ of availability.
We have studied outages in the January/February Internet outage in Egypt corresponding to the Egyptian revolution, the March 2011 Tohoku earthquake off the coast of Japan, and the October 2012 Hurricane Sandy on the east coast of the U.S. We have a technical report describing our analysis of Hurricane Sandy. We studied the 2017 Hurricane Harvey, and recently 2018 Hurricane Florence.
The full technical details are in the above papers.
A short, informal description: A Trinocular probers sends to each block (a /24 IPv4 prefix, that is, addresses in 1.2.3.*, where the first three parts are fixed), every 11 minutes (a “Trinocular round”) We probe multiple times, stopping when we get a positive reply, since that indicates the block is reachable and therefore up. We limit probing to 15 times per round, a traffic rate so low it does not have a noticeable effect on the target network.
We determine an outage has occurred after voting by 6 Trinocular probers running from different locations around the world. We use multiple locations to avoid interpreting a regional or local problem as an outage.
We add a number of other algorithms to the above basics: hole filling (to handle lost messages), precision improvement (to improve the timing of event start), gone-dark handling (to remove blocks that stop responding), full-block scanning (to avoid false outages in very sparse blocks), lone-address block recovery (to handle blocks with 2 or fewer addresses). These algorithms cover different conditions that might otherwise result in false outages.
We run two versions of our processing: batch processing includes all of our algorithms and happens every quarter. Near-real-time (NRT) processing happens continuously and includes only the basic algorithms. We report results in our outage website, with NRT data for recent results, and batch data for older results.