This post is presented in conjunction with The Internet Society.

February was a surprisingly quiet month for major Internet disruptions. In contrast to previous months, we observed few full outages or multi-day disruptions in the Oracle Internet Intelligence Map during the month. As always, there were a number of brief and unattributed disruptions observed over the course of the month, but the issues highlighted below were related to fiber cuts (and repairs) and likely problems with satellite connectivity. And while not yet a visible disruption, reportssurfaced in February that Russian authorities and major Internet providers are planning to disconnect the country from the global Internet as part of a planned experiment.

Fiber

Kicking off the month, Burkina Faso experienced brief partial disruptions to its Internet connectivity on February 1 & 2, as shown in the Country Statistics graphs below. The disruptions are also evident in the Traffic Shifts graphs below for AS25543 (Onatel), which is the country’s National Office of Telecommunications, holding a monopoly on fixed-line telecommunications there. Facebook posts from Onatel (February 1, 2) indicated that road work between the towns of Sabou and Boromo had resulted in a fiber cut, and subsequent posts made Continue reading

This post is presented in conjunction with The Internet Society.

During the second half of 2018, the causes of significant Internet disruptions observed through the Oracle Internet Intelligence Map could be clustered into a few overarching areas: government-directed, cable problems, power outages, distributed denial of service (DDoS) attacks, and general technical issues. Little changed heading into 2019, with two new government-directed Internet disruptions observed in Africa, alongside disruptions caused by fiber cuts and other network issues that impacted a number of countries around the world.

Government Directed

Initially covered in last month’s overview, the Internet disruption in the Democratic Republic of Congo continued into January, lasting through the third week of the month. Government authorities reportedly cut off Internet access in the country in December to prevent “rumor mongering” in the run-up to presidential elections.

An attempted military coup in Gabon led to a day-long Internet disruption in the country. The disruption started just after 07:00 UTC on January 7, as seen in the figure below, which shows clear declines in the Traceroute Completion Ratio and BGP Routes metrics, as well as a disruption to the usual diurnal pattern seen in the DNS Query Rate metric. Although the coup Continue reading

Closing out 2018, in December the Oracle Internet Intelligence team observed Internet disruptions in countries around the world due to power outages, government direction, technical faults, and possible issues relating to satellite connectivity. While these causes have become relatively common, it is interesting to note that other common reasons for Internet disruptions, including severe weather (such as typhoons and hurricanes), concerns over cheating on exams, and denial-of-service attacks did not appear to drive significant Internet disruptions observed in Oracle’s Internet Intelligence Map during the month. And while we tend to focus on Internet disruptions, it is also important to highlight that after several rounds of testing, nationwide mobile Internet access was finally activated across Cuba.

Cuba

In three tranches (based on the first two digits of a subscriber’s mobile phone number) over December 6, 7, and 8, ETECSA, Cuba’s national telecommunications company, enabled nationwide mobile Internet access. The rollout was reportedly stable, in contrast to the congestion experienced during the trials conducted several months prior. The figure below shows the gradual adoption of this newly available connectivity through changes in the DNS Query Rate. As seen in the graph, the query rate was comparatively low in the days ahead of Continue reading

In November, we saw fewer significant Internet disruptions in the Oracle Internet Intelligence Map as compared to prior months. As usual, there were hundreds of brief issues with limited impact and generally unknown causes, but the most notable issues last month were due to reported DDoS attacks, problems with terrestrial and submarine cables, and general network issues.

DDoS Attacks

On November 4 and 5, several Cambodian ISPs were targeted by DDoS attacks described as the “biggest attacks in Cambodian history.” Published reports indicated that ISPs including EZECOM, SINET, Telcotech, and Digi were targeted by DDoS attacks totaling nearly 150 Gbps, causing subscriber downtime lasting as much as half a day. Disruption from the attacks was visible in the Country Statistics view for Cambodia in the Internet Intelligence Map, as shown in the figure below. However, because Internet connectivity remained generally available (albeit impaired) across the country, the impact appears nominal in the graphs.

However, when viewed at a network level, the impact of the attacks appears to be more significant. SINET, one of the ISPs targeted by the DDoS attacks, posted a Tweet on November 5 letting users know that they were under attack, and followed up Continue reading

The level of significant Internet disruptions observed through the Oracle Internet Intelligence Map was lower in October, though the underlying reasons for these disruptions remained generally consistent compared to prior months. For enterprises, the importance of redundant Internet connectivity and regularly exercised failover plans is clear. Unfortunately, for state-imposed Internet outages, such planning and best practices may need to include failsafes for operations while periodically offline.

Directed disconnection

On October 10, Ethiopian Prime Minister Abiy Ahmed met with several hundred soldiers who had marched on his office to demand increased pay. The Ethiopian Broadcasting Corporation (formerly known as ETV) did not cover the soldiers marching but noted that Internet connectivity within the country had been shut off for several hours to prevent “fake news” from circulating on social media. This aligned with residents’ reports of a three-hour Internet outage. The figure below shows that the disruption began around 12:00 GMT, significantly impacting both traceroutes to, and DNS query traffic from, Ethiopia for several hours.

The impact of the Internet shutdown is also clearly evident in the figure below, which shows traceroutes into Ethio Telecom, the state-owned telecommunications service provider. Similar to the country-level graph shown above, the number of Continue reading

Over the course of a given month, hundreds of Internet-impacting “events” are visible within the Oracle Internet Intelligence Map. Many are extremely short-lived, lasting only minutes, while others last for hours or days; some have a minor impact on a single metric, while others significantly disrupt all three metrics. In addition, for some events, the root cause is publicly available/known, while for other events, digging into the underlying data helps us make an educated guess about what happened. Ultimately, this creates challenges in separating the signal from the noise, triaging and prioritizing that month’s events for review in this blog post.

Having said that, in September we observed Internet disruptions due to exams, power outages, extreme weather, and submarine cable issues, as well as a number of others with unknown causes. Additionally, a third test of nationwide mobile Internet connectivity took place in Cuba.

Cuba

As noted in our August post, ETECSA (the Cuban state telecommunications company) carried out two tests of nationwide mobile Internet connectivity, which were evident as spikes in the DNS query rates from Cuba. In a Facebook post, they noted, “On August 14th was a first test that measured levels of traffic congestion and Continue reading

During August 2018, the Oracle Internet Intelligence Map surfaced Internet disruptions around the world due to familiar causes including nationwide exams, elections, maintenance, and power outages. A more targeted disruption due to a DDoS attack was also evident, as were a number of issues that may have been related to submarine cable connectivity. In addition, in a bit of good news, the Internet Intelligence Map also provided evidence of two nationwide trials of mobile Internet services in Cuba.

Cuba

On August 15, the Oracle Internet Intelligence Twitter account highlighted that a surge in DNS queries observed the prior day was related to a nationwide test of mobile Internet service, marking the first time that Internet services were available nationwide in Cuba’s history. The figure below shows two marked peaks in DNS query rates from resolvers located in Cuba during the second half of the day (GMT) on the 14^th. Paul Calvano, a Web performance architect at Akamai, also observed a roughly 25% increase in their HTTP traffic to Cuba during the trial period.

This testing was reported by ETECSA (the Cuban state telecommunications company) in a Facebook post in which they noted:

The Telecommunications company of Cuba S.A. Continue reading

In June, we launched the Internet Intelligence microsite (home of this blog), featuring the new Internet Intelligence Map.  As the associated blog post noted, “This free site will help to democratize Internet analysis by exposing some of our internal capabilities to the general public in a single tool. …. And since major Internet outages (whether intentional or accidental) will be with us for the foreseeable future, we believe offering a self-serve capability for some of the insights we produce is a great way to move towards a healthier and more accountable Internet.”

While we will continue to share information about Internet disruptions and events as they occur via @InternetIntel, we also plan to provide a monthly roundup in a blog post, allowing readers to learn about Internet disruptions and events that they may have missed, while enabling us to provide additional context and insight beyond what fits within Twitter’s character limit.

Exams

In the past, countries including Iraq, Syria, and Ethiopia have implemented partial or complete national Internet shutdowns in an effort to prevent student cheating on exams. This past month saw Iraq implement yet another round of Internet shutdowns, and Algeria began Continue reading

On September 28, 2010, Vivek Kundra, Federal CIO at the time, issued a “Transition to IPv6” memorandum noting that “The Federal government is committed to the operational deployment and use of Internet Protocol version 6 (IPv6).” The memo described specific steps for agencies to take to “expedite the operational deployment and use of IPv6”, and laid out target deadlines for key milestones. Of specific note, it noted that agencies shall “Upgrade public/external facing servers and services (e.g. web, email, DNS, ISP services, etc) to operationally use native IPv6 by the end of FY 2012.”

For this sixth “launchiversary” of the World IPv6 Launch event, we used historical Internet Intelligence data collected from Oracle Dyn’s Internet Guide recursive DNS service to examine IPv6 adoption trends across federal agencies both ahead of the end of FY 2012 (September 2012) deadline, as well as after it.

Background

The data set used for this analysis is similar to the one used for the recent “Tracking CDN Usage Through Historical DNS Data” blog post, but in this case, it only includes .gov hostnames. While the memorandum calls out the use of IPv6 for ‘web, email, DNS, ISP services, etc.’, in order Continue reading

With Mother’s Day having just passed, some e-commerce sites likely saw an associated boost in traffic. While not as significant as the increased traffic levels seen around Black Friday and Cyber Monday, these additional visitors can potentially impact the site’s performance if it has not planned appropriately.  Some sites have extra infrastructure headroom and can absorb increased traffic without issue, but others turn to CDN providers to ensure that their sites remain fast and available, especially during holiday shopping periods.

To that end, I thought that it would be interesting to use historical Internet Intelligence data (going back to 2010) collected from Oracle Dyn’s Internet Guide recursive DNS service, to examine CDN usage. As a sample set, I chose the top 50 “shopping” sites listed on Alexa, and looked at which sites are being delivered through CDNs, which CDN providers are most popular, and whether sites change or add providers over time. Although not all of the listed sites would commonly be considered “shopping” sites, as a free and publicly available list from a well-known source, it was acceptable for the purposes of this post.

The historical research was done on the www hostname of the listed Continue reading

On Thursday, May 10 at approximately 02:00 UTC, the SeaMeWe-3 (SMW-3) subsea cable suffered yet another cable break. The break disrupted connectivity between Australia and Singapore, causing latencies to spike as illustrated below in our Internet Intelligence tool, because traffic had to take a more circuitous path.

The SMW-3 cable has had a history of outages, which we have reported on multiple times in the past, including August 2017, December 2014, and January 2013.

The ACE (African Coast to Europe) submarine cable runs along the west coast of Africa between France and South Africa, connecting 22 countries. It extends over 17,000 km, and has a potential capacity of 5.12 Tbps. The cable system is managed by a consortium of 19 telecommunications operators & administrations, and the first phase entered service in December 2012. While it may not have been completely problem-free over the last 5+ years, online searches do not return any published reports of significant outages caused by damage to the cable.

However, on March 30, damage to the cable disrupted Internet connectivity to a number of connected countries, with reported problems posted to social media over the next several days. These posts indicated that the ACE submarine cable was cut near Noukachott, Mauritania, but did not provide any specific information about what severed the cable.

The Sierra Leone Cable Limited (SALCAB) says the data connection to #SierraLeone is partly down due to the ACE Submarine cable cut in Nouakchott, Mauritania. #SierraLeoneDecides

— Leanne de Bassompierre (@leannedb01) April 1, 2018

Of the 22 countries listed as having landing points for the ACE Submarine Cable, 10 had significant disruptions evident in Oracle’s Continue reading

On Wednesday, March 21, a massive power failure impacted large parts of northern Brazil, leaving tens of millions of people without electricity. Beginning at about 3:40pm local time (18:40 UTC), the outage was reportedly due to the failure of a transmission line near the Belo Monte hydroelectric station.

As occurred in a major power outage in Brazil in 2009, this power failure had a measureable impact on the country’s Internet. This is illustrated below through graphs from Oracle Dyn’s Internet Intelligence team based on BGP and traceroute data, as well as graphs from Akamai’s mPulse service, based on end user Web traffic.

The graphic below depicts the counts of available networks (lower graph) and unstable networks (upper graph) for Brazil in the latter half of March 21. The number of unstable networks spikes around 18:40 UTC as routers of ISPs in Brazil began re-routing traffic away from disabled connections, while the lower graph shows that the corresponding drop in available networks (i.e. routed prefixes) was minor when compared to the total number routes that define the Internet of Brazil.

In addition to aggregating BGP routing information from around the globe, the Internet Intelligence team also performs millions of Continue reading

With 2017 drawing to a close, year-end lookbacks litter media and the blogosphere like so many leaves on the ground. (Or piles of snow, depending on where you are.) Many tend to focus on pop culture, product/movie/music releases, or professional sports. However, given the focus of Oracle Dyn’s Internet Intelligence team on monitoring and measuring the Internet, we’re going to take a look back at significant Internet “events” of the past year, and how they have impacted connectivity for Internet users around the world.

Hurricanes Harvey, Irma, and Maria Cause Internet Disruptions

In late August, and through September, an active Atlantic hurricane season spawned a number of destructive storms that wreaked havoc across the Caribbean, as well as Florida and Texas in the United States. On the Caribbean islands that were hardest hit by the storms, the resulting physical damage was immense, severely impacting last-mile Internet infrastructure across the whole country. This was also the case in Florida and Texas, though on a much more localized basis. On September 25, we looked at the impacts of these hurricanes on Internet connectivity in the affected areas, noting that while some “core” Internet components remained available during these storms thanks to Continue reading

In January 2011, what was arguably the first significant disconnection of an entire country from the Internet took place when routes to Egyptian networks disappeared from the Internet’s global routing table, leaving no valid paths by which the rest of the world could exchange Internet traffic with Egypt’s service providers. It was followed in short order by nationwide disruptions in Bahrain, Libya, and Syria. These outages took place during what became known as the Arab Spring, highlighting the role that the Internet had come to play in political protest, and heralding the wider use of national Internet shutdowns as a means of control.

“How hard is it to disconnect a country from the Internet, really?”

After these events, and another significant Internet outage in Syria, this question led a blog post published in November 2012 by former Dyn Chief Scientist Jim Cowie that examined the risk of Internet disconnection for countries around the world, based on the number of Internet connections at their international border. “You can think of this, to [a] first approximation,” Cowie wrote, “as the number of phone calls (or legal writs, or infrastructure attacks) that would have to be performed in order to Continue reading