This morning, from about 2024-08-05t04:50Z (10:50am local time) to t07:40Z, Bangladesh had another very large Internet outage. Fortunately, unlike the outage that began on 2024-07-18, this one cleared up after about three hours. I presume this outage corresponds to the resignation of the prime minster.
We hope for calm for the people of Bangladesh.
We have released a new technical report: “Reasoning about Internet Connectivity”, available at https://arxiv.org/abs/2407.14427.
From the abstract:
![Figure 1 from [Baltra24b], showing the connected core (A, B and C) with B and C peninsulas, D and E islands, and X an outage.](https://ant.isi.edu/blog/wp-content/uploads/2024/07/Baltra24b_icon.png){kind=icon}
Innovation in the Internet requires a global Internet core to enable
communication between users in ISPs and services in the cloud. Today, this Internet core is challenged by partial reachability: political pressure
threatens fragmentation by nationality, architectural changes such as
carrier-grade NAT make connectivity conditional, and operational problems and commercial disputes make reachability incomplete for months. We assert that partial reachability is a fundamental part of the Internet core. While some systems paper over partial reachability, this paper is the first to provide a conceptual definition of the Internet core
so we can reason about reachability from first principles. Following
the Internet design, our definition is guided by reachability, not
authority. Its corollaries are peninsulas: persistent regions of
partial connectivity; and islands: when networks are partitioned
from the Internet core. We show that the concept of peninsulas and islands can improve existing measurement systems. In one example,
they show that RIPE’s DNSmon suffers misconfiguration and persistent
network problems that are important, but risk obscuring operationally
important connectivity changes because they are 5x to 9.7x larger. Our evaluation also informs policy questions, showing no single
country or organization can unilaterally control the Internet core.
This technical report is joint work of Guillermo Baltra, Tarang Saluja, Yuri Pradkin, John Heidemann done at USC/ISI. This work was supported by the NSF via the EIEIO and InternetMap projects.
Since around 2024-07-18t15:00Z (July 18,21:00 local time), Bangladesh has had a major,country-wide Internet outage. As of t17:30Z some regions see 97% unreachability. This country-wide outage seems to be in response to civil unrest and protests.
Here’s the view from Trinocular outage detection as of 17:30Z:
We wish the best for the people of Bangladesh during this unrest.
Update July 19 morning: A day after Bangladesh’s Internet connectivity first went down, it remains nearly completely stopped. Here is our view of Bangladeshi connectivity at 2024-07-19t14:40Z (20:40 local time there):
Update July 19 afternoon: USC/ISI posted an article about the Bangladeshi Internet outage and our work as ISI news, and an new NYT article about the protests.
The AP reports “A statement from the country’s Telecommunication Regulatory Commission said they were unable to ensure service after their data center was attacked Thursday by demonstrators, who set fire to some equipment. The Associated Press was not able to independently verify this.” However, the near-complete outage observed by Trinocular (as seen in the figures above) seems inconsistent with problems at a single datacenter.
Update July 19, 22:28Z: ISOC Pulse has a post about this outage, and reports that “In a press event on 18 July, Bangladesh minister for posts, telecommunications, and information technology, Zunaid Ahmed Palak confirmed that the government had ordered the shutdown. “
Update July 20: The country-wide outage continues.
Update July 21, 17:00Z: Although recent news reports suggest some government response to protests, the near-complete country-wide Internet outage continues.
Update July 22, 23:00Z: Another day with no externally visible change–all of Bangladesh remains inaccessible from outside.
Update July 23, 18:00Z: Beginning around 13:00Z (which 19:00 in Bangladesh), we see the first signs of Bangaldeshi networks coming back on-line! The figure below is as of 16:26Z and shows about half of the national networks reachable from outside the country.
To add about the root cause, the Deccan Herald published an article from Reuters quoting Zunaid Ahmed Palak, junior information technology minister, as saying to reporters: “Mobile internet has been temporarily suspended due to various rumors and the unstable situation created…. on social media” on July 18. Today, Reuters quoted Palak as saying that “broadband internet would be restored by Tuesday night but [he] did not comment on mobile internet”. This statement is consistent with the country-wide outage we observed, and the prior statement suggests the outage was a request of the government.
Update July 24, 13:00Z (19:00 in Bangladesh): It looks like nearly all Bangladeshi networks are now back online.
Update July 25: The July 25 episode of The Briefing, an Australian news podcast, discussed the Bangladeshi outage and its impact, interviewing us about what we saw.
Hurricane Beryl made landfall in Texas around 2024-07-08 at 3:17am local time (CDT) (8:17 UTC). We see a fair number of Internet outages in the Huston area, presumably as people lost power due to flooding.
Compared to our view of Hurricane Harvey in 2017 in our blog and web, Beryl looks much less severe–we see fewer areas where most Internet acccess is out (as shown by red circles).
Our most recent data, about 10 hours after landfall (1:33pm local time, or 2024-07-08t18:33Z):
Just before landfall, at 3:17am local time (2024-07-08t08:17Z):
We wish the best for Texas, and for the residents of the Caribbean who experienced Beryl last week.
For current status, please see our near-real-time outage site. Data about this outage will be released at the end of the quarter.
Starting on April 21, 2024, we observed a large Internet outage in the country Georgia. More than half the IP blocks in large parts of the country have become unreachable from the U.S., with the problem persisting for several days so far.
The timing of this outage is consistent with a recent resurgence of protests over the Georgian “Law on Transparency of Foreign Influence”.
On March 14, 2024, we observed a large outage in several West African countries. In Ivory Coast and Liberia, the outage was quite severe, affecting 93% of the active network blocks:
Fortunately some locations were able to partially recover from the problems, presumably by routing through different paths:
The root cause for these outages is likely a problems in multiple undersea telecommunication cables, as has been reported in the Washington Post and the Guardian, among other places.
Our new paper “Ebb and Flow: Implications of ISP Address Dynamics” will appear at the 2024 Conference on Passive and Active Measurements (PAM 2024).
From the abstract:
Address dynamics are changes in IP address occupation as users come and go, ISPs renumber them for privacy or for routing maintenance. Address dynamics affect address reputation services, IP geolocation, network measurement, and outage detection, with implications of Internet governance, e-commerce, and science. While prior work has identified diurnal trends in address use, we show the effectiveness of Multi-Seasonal-Trend using Loess decomposition to identify both daily and weekly trends. We use ISP-wide dynamics to develop IAS, a new algorithm that is the first to automatically detect ISP maintenance events that move users in the address space. We show that 20% of such events result in /24 IPv4 address blocks that become unused for days or more, and correcting nearly 41k false outages per quarter. Our analysis provides a new understanding about ISP address use: while only about 2.8% of ASes (1,730) are diurnal, some diurnal ASes show more than 20% changes each day. It also shows greater fragmentation in IPv4 address use compared to IPv6.
This paper is a joint work of Guillermo Baltra, Xiao Song, and John Heidemann. Datasets from this paper can be found at https://ant.isi.edu/datasets/outage. This work was supported by NSF (MINCEQ, NSF 2028279; EIEIO CNS-2007106.
I would like to congratulate Dr. Guillermo Baltra for defending his PhD at the University of Southern California in August 2023 and completing his doctoral dissertation “Improving network reliability using a formal definition of the Internet core”.
From the abstract:
After 50 years, the Internet is still defined as “a collection of interconnected networks”. Yet seamless, universal connectivity is challenged in several ways. Political pressure threatens fragmentation due to de-peering; architectural changes such as carrier-grade NAT, the cloud makes connectivity indirect; firewalls impede connectivity; and operational problems and commercial disputes all challenge the idea of a single set of “interconnected networks”. We propose that a new, conceptual definition of the Internet core helps disambiguate questions in analysis of network reliability and address space usage.
We prove this statement through three studies. First, we improve coverage of outage detection by dealing with sparse sections of the Internet, increasing from a nominal 67% responsive /24 blocks coverage to 96% of the responsive Internet. Second, we provide a new definition of the Internet core, and use it to resolve partial reachability ambiguities. We show that the Internet today has peninsulas of persistent, partial connectivity, and that some outages cause islands where the Internet at the site is up, but partitioned from the main Internet. Finally, we use our definition to identify ISP trends, with applications to policy and improving outage detection accuracy. We show how these studies together thoroughly prove our thesis statement. We provide a new conceptual definition of “the Internet core” in our second study about partial reachability. We use our definition in our first and second studies to disambiguate questions about network reliability and in our third study, to ISP address space usage dynamics.
Guillermo’s PhD work was supported by NSF grants CNS-1806785, CNS-2007106 and NSF-2028279 and DH S&T Cyber Security Division contract 70RSAT18CB0000014 and a DHS contract administred by AFRL as contract FA8750-18-2-0280, to USC Viterbi, the Armada de Chile, and the Agencia Nacional de Investigación y Desarrollo de Chile (ANID).
Please see his individual publications for what data is available from his research; his results are also in use in ongoing Trinocular outage detection datasets.
Recent news reports (for example, Reuters) state that Telecom Italia had a major outage on Sunday, February 5, 2023.
We see evidence for this outage in our Internet outage detection system.
It looks like there were two relatively brief outages, one at 2023-02-05t10:49Z (11:49 local time in Italy) and a smaller one at 11:33Z (12:33 local time). Our monitoring rounds time to about 11 minutes, so the actual events may have been at slightly different times.
These outages were nation-wide, apparently affecting most of Italy. However, it looks like they “only” affected 20-30% of networks, and not all Italian ISPs. We’re happy they were able to recover so quickly.
This event shows the importance of global network monitoring.
Asma Enayet will present her poster “Internet Outage Detection Using Passive Analysis” by Asma Enayet and John Heidemann at ACM Internet Measurement Conference, Nice, France from October 25-27th, 2022.
We expect the ACM poster abstract (without the poster) to appear at https://doi.org/10.1145/3517745.3563032 in October 2022.
We are making a report available now with the poster abstract and poster at https://doi.org/10.48550/arXiv.2209.13767 as a pre-print.
From the abstract:
Outages from natural disasters, political events, software or hardware issues, and human error place a huge cost on e-commerce ($66k per minute at Amazon). While several existing systems detect Internet outages, these systems are often too inflexible, with fixed parameters across the whole internet with CUSUM-like change detection. We instead propose a system using passive data, to cover both IPv4 and IPv6, customizing parameters for each block to optimize the performance of our Bayesian inference model. Our poster describes our three contributions: First, we show how customizing parameters allows us often to detect outages that are at both fine timescales (5 minutes) and fine spatial resolutions (/24 IPv4 and /48 IPv6 blocks). Our second contribution is to show that, by tuning parameters differently for different blocks, we can scale back temporal precision to cover more challenging blocks. Finally, we show our approach extends to IPv6 and provides the first reports of IPv6 outages.
This work was supported by NSF grant CNS-2007106 (EIEIO).