Author Archives: David Belson
Author Archives: David Belson
Cloudflare operates in more than 270 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the second quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause or common geography.
This quarter, we saw the usual complement of damage to both terrestrial and submarine fiber-optic cables, including one that impacted multiple countries across thousands of miles, and another more localized outage that was due to an errant rodent.
On April 25, Comcast subscribers in nearly 20 southwestern Florida cities experienced an outage, reportedly due to a fiber cut. The traffic impact of this cut is clearly visible in the graph below, with Cloudflare traffic Continue reading
On May 19, 2021, a Microsoft blog post announced that “The future of Internet Explorer on Windows 10 is in Microsoft Edge” and that “the Internet Explorer 11 desktop application will be retired and go out of support on June 15, 2022, for certain versions of Windows 10.” According to an associated FAQ page, those “certain versions” include Windows 10 client SKUs and Windows 10 IoT. According to data from Statcounter, Windows 10 currently accounts for over 70% of desktop Windows market share on a global basis, so this “retirement” impacts a significant number of Windows systems around the world.
As the retirement date for Internet Explorer 11 has recently passed, we wanted to explore several related usage trends:
It is once again exam time in Syria, Sudan, and Algeria, and with it, we find these countries disrupting Internet connectivity in an effort to prevent cheating on these exams. As they have done over the past several years, Syria and Sudan are implementing multi-hour nationwide Internet shutdowns. Algeria has also taken a similar approach in the past, but this year appears to be implementing more targeted website/application blocking.
Syria has been implementing Internet shutdowns across the country since 2011, but exam-related shutdowns have only been in place since 2016. In 2021, exams took place between May 31 and June 22, with multi-hour shutdowns observed on each of the exam days.
This year, the first shutdown was observed on May 30, with subsequent shutdowns (to date) seen on June 2, 6, and 12. In the Cloudflare Radar graph below, traffic for Syria drops to zero while the shutdowns are active. According to Internet Society Pulse, several additional shutdowns are expected through June 21. Each takes place between 02000530 UTC (0500–0830 local time). According to a published report, the current exam cycle covers more than 500,000 students for basic and general secondary education certificates.
Consistent with Continue reading
Just after 1200 UTC on Tuesday, June 7, the Africa-Asia-Europe-1 (AAE-1) and SEA-ME-WE-5 (SMW-5) submarine cables suffered cable cuts. The damage reportedly occurred in Egypt, and impacted Internet connectivity for millions of Internet users across multiple countries in the Middle East and Africa, as well as thousands of miles away in Asia. In addition, Google Cloud Platform and OVHcloud reported connectivity issues due to these cable cuts.
Data from Cloudflare Radar showed significant drops in traffic across the impacted countries as the cable damage occurred, recovering approximately four hours later as the cables were repaired.
It appears that Saudi Arabia may have also been affected by the cable cut(s), but the impact was much less significant, and traffic recovered almost immediately.
In the graphs above, we show that Ethiopia was one of the impacted countries. However, as it is landlocked, there are obviously no submarine cable landing points within the country. The Afterfibre map from the Network Startup Resource Center (NSRC) shows that that fiber in Ethiopia connects to fiber in Somalia, which experienced an impact. In addition, Ethio Telecom also routes traffic through network providers in Kenya and Djibouti. Djibouti Telecom, one of these providers, in turn Continue reading
Cloudflare operates in more than 250 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the first quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause.
Internet outages caused by “earth movers” are more frequently caused by errant backhoes. However, two Internet disruptions in the first quarter were caused by more significant earth movement — a volcanic eruption and an earthquake.
The first impacted connectivity on the island nation of Tonga, when the Hunga Tonga–Hunga Ha'apai volcanic eruption damaged the submarine cable connecting Tonga to Fiji, resulting in a 38 day Internet outage. After the January 14 eruption, only minimal Internet traffic (via satellite Continue reading
On the morning of March 8, a post to Hacker News stated that “All .fj domains have gone offline”, listing several hostnames in domains within the Fiji top level domain (known as a ccTLD) that had become unreachable. Commenters in the associated discussion thread had mixed results in being able to reach .fj hostnames—some were successful, while others saw failures. The fijivillage news site also highlighted the problem, noting that the issue also impacted Vodafone’s M-PAiSA app/service, preventing users from completing financial transactions.
The impact of this issue can be seen in traffic to Cloudflare customer zones in the .com.fj second-level domain. The graph below shows that HTTP traffic to these zones dropped by approximately 40% almost immediately starting around midnight UTC on March 8. Traffic volumes continued to decline throughout the rest of the morning.
Looking at Cloudflare’s 188.8.131.52 resolver data for queries for .com.fj hostnames, we can also see that error volume associated with those queries climbs significantly starting just after midnight as well. This means that our resolvers encountered issues with the answers from .fj servers.
This observation suggests that the problem was strictly DNS related, rather than connectivity related—Cloudflare Radar Continue reading
Internet outages are more common than most people think, and may be caused by misconfigurations, power outages, extreme weather, or infrastructure damage. Note that such outages are distinct from state-imposed shutdowns that also happen all too frequently, generally used to deal with situations of unrest, elections or even exams.
On the morning of January 4, 2022, citizens of The Gambia woke up to a country-wide Internet outage. Gamtel (the main state-owned telecommunications company of the West Africa country), announced that it happened due to "technical issues on the backup links" — we elaborate more on this below.
Cloudflare Radar shows that the outage had a significant impact on Internet traffic in the country and started after 01:00 UTC (which is the same local time), lasting until ~09:45 — a disruption of over 8 hours.
Looking at BGP (Border Gateway Protocol) updates from Gambian ASNs around the time of the outage, we see a clear spike at 01:10 UTC. These update messages are BGP signaling that the Gambian ASNs are no longer routable.
It is important to know that BGP is a mechanism to exchange routing information between autonomous systems (networks) on the Internet. The routers that make the Continue reading
Six months ago, when the World Health Organization declared COVID-19 a pandemic, it accelerated the shift out of offices and schools and onto the Internet. Back then, we asked if the Internet was resilient enough to withstand Coronavirus. After several months of observations, we confirmed that it is, thanks to the strength, resilience and success of the open architecture that underpins it. Since then, concerns about the Internet’s ability to handle the increase in lockdown-driven traffic seem to have abated, resulting in fewer articles and blog posts on the topic.
As we head into the final months of 2020, some businesses have reopened in a limited capacity, allowing employees to return to their brick-and-mortar workplaces. Many students are also returning to school, whether in person or online. Yet, the lack of affordable and available Internet access remains a significant issue. Earlier this year, we heard stories about students sitting outside schools and libraries in search of reliable WiFi in order to attend classes. As the new school year starts in North America, we heard about students using WiFi signals from a local fast food chain restaurant to complete their homework. And with students now needing to Continue reading
Although the COVID-19 pandemic is, unfortunately, far from over, the Internet continues to be resilient, supporting the additional demands that we have placed on it, including the rapid growth in online learning, work videoconferences, e-commerce, streaming video entertainment, and more.
Because the Internet exists as a network of networks, this resilience is largely due to the planning, actions, and cooperation of all of the interconnected participants. These participants include, but are certainly not limited to, network providers, Internet Exchange Points (IXPs), and Content Delivery Networks (CDNs).
On the network side, major providers such as Comcast have invested billions of dollars over the last several years in expanding fiber infrastructure and growing network capacity. In contrast, community network provider NYC Mesh is leveraging DIY customer installations to grow its own infrastructure during the pandemic. While there is still much work to be done, these efforts by both large and small network operators as well as similar operators all over the world, are making reliable broadband connectivity more widely available. Infrastructure expansion is vital as the current pandemic has shown that access to reliable Internet is now more important than ever before.
Fast and Reliable Content Delivery
CDN providers are also doing their Continue reading
If your household is anything like mine, your Internet connection has experienced a significant increase in usage over these last several months. We’re streaming more and more media each day, and we’re on seemingly endless hours of videoconferences for work and for school. While all of that streaming media consumes downstream capacity, those videoconferences can generate a significant amount of upstream traffic. I’m fortunate enough to have fiber-based broadband connectivity that can easily handle this traffic, but I know others aren’t as lucky. They’re stuck with copper-based connections or satellite links that struggle to deliver streaming media or video calls with any sort of viewable quality.
Across this spectrum of “last mile” Internet connections, I looked at the impact from both a provider and user perspective. What kind of traffic growth have last mile network providers experienced? What steps have these providers taken to ensure they have sufficient capacity? And most importantly for end users, how has increased traffic impacted last mile connection speeds?
The network connections from customer- and subscriber-facing Internet service providers are often referred to as last mile networks. These are Internet services delivered over a notional distance – the “last mile” – to subscriber premises, such Continue reading
Earlier this year, as COVID-19 began to dominate our lives, the world turned to the Internet. This sudden shift to distance learning, working from home, and families sheltering in place drove up online streaming demand, placing additional load on Internet application platforms like Zoom, Netflix, and educational tools such as Kahoot. There was also a dramatic traffic increase across supporting network providers.
Faced with the specter of millions of daily Zoom calls and endless hours of Netflix binging, many wondered if the Internet could handle the strain of such rapid traffic growth and increased latency. Would it cause a catastrophic failure of the Internet? Our answer then: not likely.
But were we right? As the world is now more than a month into mandatory lockdowns and stay-at-home orders, with anticipated growth in application platform usage, media consumption, and overall Internet traffic, we can now state:
“No – increased Internet usage will not cause a catastrophic Internet failure.”
As expected, the Internet has remained resilient. There is no single “Internet” to catastrophically fail, thanks to its foundational “network of networks” architecture.
This architecture means that many interconnected participants all have a role in keeping the Internet resilient:
It’s being called the world’s biggest work-from-home experiment. With concerns growing over the spread of the Coronavirus, schools and businesses around the world are implementing contingency plans that encourage distance learning and work from home. Usage of e-learning, messaging, and videoconferencing tools is also growing rapidly, placing additional load on these Internet-based applications and platforms and generating additional traffic. And with more people staying at home, online media consumption is poised to increase as well.
Many are wondering if the Internet can handle the strain of rapid traffic growth and increased latency. Will it cause a catastrophic failure of the Internet? The answer: not likely.
Core Internet infrastructure providers should be able to easily absorb the increase in traffic and demand, especially if the growth is gradual over a period of days, weeks, or months. Cloud infrastructure providers should also have sufficient additional compute, storage, and bandwidth capacity to enable their customers, including the e-learning, messaging, and videoconferencing tool providers, to scale their systems as necessary. In order to keep traffic local, content delivery infrastructure from companies including Akamai, Cloudflare, Google, Netflix, and Apple is deployed in many last-mile networks. Internet Exchange Points (IXPs) can also help keep traffic local, Continue reading
In March 2019, in a move described in one news report as a “government-imposed Internet shutdown,” the president of Sri Lanka temporarily blocked Facebook, WhatsApp, Instagram, Viber, and other services. In this case, limited access to a class of applications was inaccurately painted as a full-scale Internet shutdown. Unfortunately, this isn’t unusual. Media coverage and general discussion of Internet disruptions often misclassify what happened. The confusion is likely unintentional. Many journalists, as well as the general public, are not well-versed in the various ways Internet access and access to content can be disrupted.
When people can’t get to their favorite social media site, chat app, or video platform, there can be many causes. Maybe there’s a local Internet outage, or access to these sites has been blocked because of a government decree, or a nationwide Internet shutdown has been ordered by the government. Internet disruptions can take several forms, but end users experience the same problem across all of them – an inability to use the Internet to communicate and access content.
If, in the end, the end user experience is the same, why is it important to delineate between the various types of Internet disruptions? Proper delineation can help Continue reading
Global Internet freedom declined for the ninth consecutive year in 2019, largely as a result of social media increasingly being used by governments around the world as a conduit for mass surveillance and electoral manipulation. The Freedom on the Net 2019 report, the latest edition of the annual country-by-country assessment of Internet freedom, was released on November 5 by Freedom House, and highlights the shift in social media from a level playing field for civic discussion to an instrument of political distortion and societal control.
The Freedom on the Net 2019 report analyzed Internet freedom in 65 countries worldwide, covering 87% of global Internet users. Surveyed countries are designated as ‘Free’, ‘Partly Free’, or ‘Not Free’ based on an examination of, and scoring against, three categories: obstacles to access, limits on content, and violations of user rights.
Of the 65 countries assessed, 33 of them saw Internet freedom decline over the last year, with the biggest drops observed in Sudan and Kazakhstan. The longtime presidents of both countries were ousted, leading to widespread blocking of social media platforms, disruptions of Internet connectivity, and the increased use of electronic surveillance to undermine free expression.
The report called digital platforms Continue reading
This post is presented in conjunction with The Internet Society.
February was a surprisingly quiet month for major Internet disruptions. In contrast to previous months, we observed few full outages or multi-day disruptions in the Oracle Internet Intelligence Map during the month. As always, there were a number of brief and unattributed disruptions observed over the course of the month, but the issues highlighted below were related to fiber cuts (and repairs) and likely problems with satellite connectivity. And while not yet a visible disruption, reportssurfaced in February that Russian authorities and major Internet providers are planning to disconnect the country from the global Internet as part of a planned experiment.
Kicking off the month, Burkina Faso experienced brief partial disruptions to its Internet connectivity on February 1 & 2, as shown in the Country Statistics graphs below. The disruptions are also evident in the Traffic Shifts graphs below for AS25543 (Onatel), which is the country’s National Office of Telecommunications, holding a monopoly on fixed-line telecommunications there. Facebook posts from Onatel (February 1, 2) indicated that road work between the towns of Sabou and Boromo had resulted in a fiber cut, and subsequent posts made Continue reading
This post is presented in conjunction with The Internet Society.
During the second half of 2018, the causes of significant Internet disruptions observed through the Oracle Internet Intelligence Map could be clustered into a few overarching areas: government-directed, cable problems, power outages, distributed denial of service (DDoS) attacks, and general technical issues. Little changed heading into 2019, with two new government-directed Internet disruptions observed in Africa, alongside disruptions caused by fiber cuts and other network issues that impacted a number of countries around the world.
Initially covered in last month’s overview, the Internet disruption in the Democratic Republic of Congo continued into January, lasting through the third week of the month. Government authorities reportedly cut off Internet access in the country in December to prevent “rumor mongering” in the run-up to presidential elections.
An attempted military coup in Gabon led to a day-long Internet disruption in the country. The disruption started just after 07:00 UTC on January 7, as seen in the figure below, which shows clear declines in the Traceroute Completion Ratio and BGP Routes metrics, as well as a disruption to the usual diurnal pattern seen in the DNS Query Rate metric. Although the coup Continue reading
Closing out 2018, in December the Oracle Internet Intelligence team observed Internet disruptions in countries around the world due to power outages, government direction, technical faults, and possible issues relating to satellite connectivity. While these causes have become relatively common, it is interesting to note that other common reasons for Internet disruptions, including severe weather (such as typhoons and hurricanes), concerns over cheating on exams, and denial-of-service attacks did not appear to drive significant Internet disruptions observed in Oracle’s Internet Intelligence Map during the month. And while we tend to focus on Internet disruptions, it is also important to highlight that after several rounds of testing, nationwide mobile Internet access was finally activated across Cuba.
In three tranches (based on the first two digits of a subscriber’s mobile phone number) over December 6, 7, and 8, ETECSA, Cuba’s national telecommunications company, enabled nationwide mobile Internet access. The rollout was reportedly stable, in contrast to the congestion experienced during the trials conducted several months prior. The figure below shows the gradual adoption of this newly available connectivity through changes in the DNS Query Rate. As seen in the graph, the query rate was comparatively low in the days ahead of Continue reading
In November, we saw fewer significant Internet disruptions in the Oracle Internet Intelligence Map as compared to prior months. As usual, there were hundreds of brief issues with limited impact and generally unknown causes, but the most notable issues last month were due to reported DDoS attacks, problems with terrestrial and submarine cables, and general network issues.
On November 4 and 5, several Cambodian ISPs were targeted by DDoS attacks described as the “biggest attacks in Cambodian history.” Published reports indicated that ISPs including EZECOM, SINET, Telcotech, and Digi were targeted by DDoS attacks totaling nearly 150 Gbps, causing subscriber downtime lasting as much as half a day. Disruption from the attacks was visible in the Country Statistics view for Cambodia in the Internet Intelligence Map, as shown in the figure below. However, because Internet connectivity remained generally available (albeit impaired) across the country, the impact appears nominal in the graphs.
However, when viewed at a network level, the impact of the attacks appears to be more significant. SINET, one of the ISPs targeted by the DDoS attacks, posted a Tweet on November 5 letting users know that they were under attack, and followed up Continue reading
The level of significant Internet disruptions observed through the Oracle Internet Intelligence Map was lower in October, though the underlying reasons for these disruptions remained generally consistent compared to prior months. For enterprises, the importance of redundant Internet connectivity and regularly exercised failover plans is clear. Unfortunately, for state-imposed Internet outages, such planning and best practices may need to include failsafes for operations while periodically offline.
On October 10, Ethiopian Prime Minister Abiy Ahmed met with several hundred soldiers who had marched on his office to demand increased pay. The Ethiopian Broadcasting Corporation (formerly known as ETV) did not cover the soldiers marching but noted that Internet connectivity within the country had been shut off for several hours to prevent “fake news” from circulating on social media. This aligned with residents’ reports of a three-hour Internet outage. The figure below shows that the disruption began around 12:00 GMT, significantly impacting both traceroutes to, and DNS query traffic from, Ethiopia for several hours.
The impact of the Internet shutdown is also clearly evident in the figure below, which shows traceroutes into Ethio Telecom, the state-owned telecommunications service provider. Similar to the country-level graph shown above, the number of Continue reading
Over the course of a given month, hundreds of Internet-impacting “events” are visible within the Oracle Internet Intelligence Map. Many are extremely short-lived, lasting only minutes, while others last for hours or days; some have a minor impact on a single metric, while others significantly disrupt all three metrics. In addition, for some events, the root cause is publicly available/known, while for other events, digging into the underlying data helps us make an educated guess about what happened. Ultimately, this creates challenges in separating the signal from the noise, triaging and prioritizing that month’s events for review in this blog post.
Having said that, in September we observed Internet disruptions due to exams, power outages, extreme weather, and submarine cable issues, as well as a number of others with unknown causes. Additionally, a third test of nationwide mobile Internet connectivity took place in Cuba.
As noted in our August post, ETECSA (the Cuban state telecommunications company) carried out two tests of nationwide mobile Internet connectivity, which were evident as spikes in the DNS query rates from Cuba. In a Facebook post, they noted, “On August 14th was a first test that measured levels of traffic congestion and Continue reading