David Belson, Author at NetworkingNexus.net

David Belson

Author Archives: David Belson

Exam time means Internet disruptions in Syria, Sudan and Algeria

It is once again exam time in Syria, Sudan, and Algeria, and with it, we find these countries disrupting Internet connectivity in an effort to prevent cheating on these exams. As they have done over the past several years, Syria and Sudan are implementing multi-hour nationwide Internet shutdowns. Algeria has also taken a similar approach in the past, but this year appears to be implementing more targeted website/application blocking.

Syria

Syria has been implementing Internet shutdowns across the country since 2011, but exam-related shutdowns have only been in place since 2016. In 2021, exams took place between May 31 and June 22, with multi-hour shutdowns observed on each of the exam days.

This year, the first shutdown was observed on May 30, with subsequent shutdowns (to date) seen on June 2, 6, and 12. In the Cloudflare Radar graph below, traffic for Syria drops to zero while the shutdowns are active. According to Internet Society Pulse, several additional shutdowns are expected through June 21. Each takes place between 02000530 UTC (0500–0830 local time). According to a published report, the current exam cycle covers more than 500,000 students for basic and general secondary education certificates.

Consistent with Continue reading

AAE-1 & SMW5 cable cuts impact millions of users across multiple countries

Just after 1200 UTC on Tuesday, June 7, the Africa-Asia-Europe-1 (AAE-1) and SEA-ME-WE-5 (SMW-5) submarine cables suffered cable cuts. The damage reportedly occurred in Egypt, and impacted Internet connectivity for millions of Internet users across multiple countries in the Middle East and Africa, as well as thousands of miles away in Asia. In addition, Google Cloud Platform and OVHcloud reported connectivity issues due to these cable cuts.

The impact

Data from Cloudflare Radar showed significant drops in traffic across the impacted countries as the cable damage occurred, recovering approximately four hours later as the cables were repaired.

It appears that Saudi Arabia may have also been affected by the cable cut(s), but the impact was much less significant, and traffic recovered almost immediately.

In the graphs above, we show that Ethiopia was one of the impacted countries. However, as it is landlocked, there are obviously no submarine cable landing points within the country. The Afterfibre map from the Network Startup Resource Center (NSRC) shows that that fiber in Ethiopia connects to fiber in Somalia, which experienced an impact. In addition, Ethio Telecom also routes traffic through network providers in Kenya and Djibouti. Djibouti Telecom, one of these providers, in turn Continue reading

Internet disruptions overview for Q1 2022

Cloudflare operates in more than 250 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional government-directed shutdown. In this post, we review selected Internet disruptions observed by Cloudflare during the first quarter of 2022, supported by traffic graphs from Cloudflare Radar and other internal Cloudflare tools, and grouped by associated cause.

Plate tectonics

Internet outages caused by “earth movers” are more frequently caused by errant backhoes. However, two Internet disruptions in the first quarter were caused by more significant earth movement — a volcanic eruption and an earthquake.

The first impacted connectivity on the island nation of Tonga, when the Hunga Tonga–Hunga Ha'apai volcanic eruption damaged the submarine cable connecting Tonga to Fiji, resulting in a 38 day Internet outage. After the January 14 eruption, only minimal Internet traffic (via satellite Continue reading

DNSSEC issues take Fiji domains offline

On the morning of March 8, a post to Hacker News stated that “All .fj domains have gone offline”, listing several hostnames in domains within the Fiji top level domain (known as a ccTLD) that had become unreachable. Commenters in the associated discussion thread had mixed results in being able to reach .fj hostnames—some were successful, while others saw failures. The fijivillage news site also highlighted the problem, noting that the issue also impacted Vodafone’s M-PAiSA app/service, preventing users from completing financial transactions.

The impact of this issue can be seen in traffic to Cloudflare customer zones in the .com.fj second-level domain. The graph below shows that HTTP traffic to these zones dropped by approximately 40% almost immediately starting around midnight UTC on March 8. Traffic volumes continued to decline throughout the rest of the morning.

Looking at Cloudflare’s 1.1.1.1 resolver data for queries for .com.fj hostnames, we can also see that error volume associated with those queries climbs significantly starting just after midnight as well. This means that our resolvers encountered issues with the answers from .fj servers.

This observation suggests that the problem was strictly DNS related, rather than connectivity related—Cloudflare Radar Continue reading

How The Gambia lost access to the Internet for more than 8 hours

Internet outages are more common than most people think, and may be caused by misconfigurations, power outages, extreme weather, or infrastructure damage. Note that such outages are distinct from state-imposed shutdowns that also happen all too frequently, generally used to deal with situations of unrest, elections or even exams.

On the morning of January 4, 2022, citizens of The Gambia woke up to a country-wide Internet outage. Gamtel (the main state-owned telecommunications company of the West Africa country), announced that it happened due to "technical issues on the backup links" — we elaborate more on this below.

Cloudflare Radar shows that the outage had a significant impact on Internet traffic in the country and started after 01:00 UTC (which is the same local time), lasting until ~09:45 — a disruption of over 8 hours.

Looking at BGP (Border Gateway Protocol) updates from Gambian ASNs around the time of the outage, we see a clear spike at 01:10 UTC. These update messages are BGP signaling that the Gambian ASNs are no longer routable.

It is important to know that BGP is a mechanism to exchange routing information between autonomous systems (networks) on the Internet. The routers that make the Continue reading

What Does Normal Look Like Now? COVID’s Impact on the Internet

Six months ago, when the World Health Organization declared COVID-19 a pandemic, it accelerated the shift out of offices and schools and onto the Internet. Back then, we asked if the Internet was resilient enough to withstand Coronavirus. After several months of observations, we confirmed that it is, thanks to the strength, resilience and success of the open architecture that underpins it. Since then, concerns about the Internet’s ability to handle the increase in lockdown-driven traffic seem to have abated, resulting in fewer articles and blog posts on the topic.

Getting Back to Normal

As we head into the final months of 2020, some businesses have reopened in a limited capacity, allowing employees to return to their brick-and-mortar workplaces. Many students are also returning to school, whether in person or online. Yet, the lack of affordable and available Internet access remains a significant issue. Earlier this year, we heard stories about students sitting outside schools and libraries in search of reliable WiFi in order to attend classes. As the new school year starts in North America, we heard about students using WiFi signals from a local fast food chain restaurant to complete their homework. And with students now needing to Continue reading

Holding Steady: How CDNs, IXPs, and Network Providers Help Keep us Online

Although the COVID-19 pandemic is, unfortunately, far from over, the Internet continues to be resilient, supporting the additional demands that we have placed on it, including the rapid growth in online learning, work videoconferences, e-commerce, streaming video entertainment, and more.

Because the Internet exists as a network of networks, this resilience is largely due to the planning, actions, and cooperation of all of the interconnected participants. These participants include, but are certainly not limited to, network providers, Internet Exchange Points (IXPs), and Content Delivery Networks (CDNs).

Expanding Infrastructure

On the network side, major providers such as Comcast have invested billions of dollars over the last several years in expanding fiber infrastructure and growing network capacity. In contrast, community network provider NYC Mesh is leveraging DIY customer installations to grow its own infrastructure during the pandemic. While there is still much work to be done, these efforts by both large and small network operators as well as similar operators all over the world, are making reliable broadband connectivity more widely available. Infrastructure expansion is vital as the current pandemic has shown that access to reliable Internet is now more important than ever before.

Fast and Reliable Content Delivery

CDN providers are also doing their Continue reading

How Has COVID-19 Impacted Last Mile Networks?

If your household is anything like mine, your Internet connection has experienced a significant increase in usage over these last several months. We’re streaming more and more media each day, and we’re on seemingly endless hours of videoconferences for work and for school. While all of that streaming media consumes downstream capacity, those videoconferences can generate a significant amount of upstream traffic. I’m fortunate enough to have fiber-based broadband connectivity that can easily handle this traffic, but I know others aren’t as lucky. They’re stuck with copper-based connections or satellite links that struggle to deliver streaming media or video calls with any sort of viewable quality.

Across this spectrum of “last mile” Internet connections, I looked at the impact from both a provider and user perspective. What kind of traffic growth have last mile network providers experienced? What steps have these providers taken to ensure they have sufficient capacity? And most importantly for end users, how has increased traffic impacted last mile connection speeds?

The network connections from customer- and subscriber-facing Internet service providers are often referred to as last mile networks. These are Internet services delivered over a notional distance – the “last mile” – to subscriber premises, such Continue reading

The Internet Is Resilient Enough to Withstand Coronavirus – But There’s a Catch

Earlier this year, as COVID-19 began to dominate our lives, the world turned to the Internet. This sudden shift to distance learning, working from home, and families sheltering in place drove up online streaming demand, placing additional load on Internet application platforms like Zoom, Netflix, and educational tools such as Kahoot. There was also a dramatic traffic increase across supporting network providers.

Faced with the specter of millions of daily Zoom calls and endless hours of Netflix binging, many wondered if the Internet could handle the strain of such rapid traffic growth and increased latency. Would it cause a catastrophic failure of the Internet? Our answer then: not likely.

But were we right? As the world is now more than a month into mandatory lockdowns and stay-at-home orders, with anticipated growth in application platform usage, media consumption, and overall Internet traffic, we can now state:

“No – increased Internet usage will not cause a catastrophic Internet failure.”

As expected, the Internet has remained resilient. There is no single “Internet” to catastrophically fail, thanks to its foundational “network of networks” architecture.

This architecture means that many interconnected participants all have a role in keeping the Internet resilient:

Subscriber (“last Continue reading

Is the Internet Resilient Enough to Withstand Coronavirus?

It’s being called the world’s biggest work-from-home experiment. With concerns growing over the spread of the Coronavirus, schools and businesses around the world are implementing contingency plans that encourage distance learning and work from home. Usage of e-learning, messaging, and videoconferencing tools is also growing rapidly, placing additional load on these Internet-based applications and platforms and generating additional traffic. And with more people staying at home, online media consumption is poised to increase as well.

Many are wondering if the Internet can handle the strain of rapid traffic growth and increased latency. Will it cause a catastrophic failure of the Internet? The answer: not likely.

Core Internet infrastructure providers should be able to easily absorb the increase in traffic and demand, especially if the growth is gradual over a period of days, weeks, or months. Cloud infrastructure providers should also have sufficient additional compute, storage, and bandwidth capacity to enable their customers, including the e-learning, messaging, and videoconferencing tool providers, to scale their systems as necessary. In order to keep traffic local, content delivery infrastructure from companies including Akamai, Cloudflare, Google, Netflix, and Apple is deployed in many last-mile networks. Internet Exchange Points (IXPs) can also help keep traffic local, Continue reading

From Content Blocking to National Shutdowns: Understanding Internet Disruptions

In March 2019, in a move described in one news report as a “government-imposed Internet shutdown,” the president of Sri Lanka temporarily blocked Facebook, WhatsApp, Instagram, Viber, and other services. In this case, limited access to a class of applications was inaccurately painted as a full-scale Internet shutdown. Unfortunately, this isn’t unusual. Media coverage and general discussion of Internet disruptions often misclassify what happened. The confusion is likely unintentional. Many journalists, as well as the general public, are not well-versed in the various ways Internet access and access to content can be disrupted.

When people can’t get to their favorite social media site, chat app, or video platform, there can be many causes. Maybe there’s a local Internet outage, or access to these sites has been blocked because of a government decree, or a nationwide Internet shutdown has been ordered by the government. Internet disruptions can take several forms, but end users experience the same problem across all of them – an inability to use the Internet to communicate and access content.

If, in the end, the end user experience is the same, why is it important to delineate between the various types of Internet disruptions? Proper delineation can help Continue reading

Social Media Crisis Drives Ongoing Decline In Global Internet Freedom

Global Internet freedom declined for the ninth consecutive year in 2019, largely as a result of social media increasingly being used by governments around the world as a conduit for mass surveillance and electoral manipulation. The Freedom on the Net 2019 report, the latest edition of the annual country-by-country assessment of Internet freedom, was released on November 5 by Freedom House, and highlights the shift in social media from a level playing field for civic discussion to an instrument of political distortion and societal control.

The Freedom on the Net 2019 report analyzed Internet freedom in 65 countries worldwide, covering 87% of global Internet users. Surveyed countries are designated as ‘Free’, ‘Partly Free’, or ‘Not Free’ based on an examination of, and scoring against, three categories: obstacles to access, limits on content, and violations of user rights.

Of the 65 countries assessed, 33 of them saw Internet freedom decline over the last year, with the biggest drops observed in Sudan and Kazakhstan. The longtime presidents of both countries were ousted, leading to widespread blocking of social media platforms, disruptions of Internet connectivity, and the increased use of electronic surveillance to undermine free expression.

The report called digital platforms Continue reading

Last Month in Internet Intelligence: February 2019

This post is presented in conjunction with The Internet Society.

February was a surprisingly quiet month for major Internet disruptions. In contrast to previous months, we observed few full outages or multi-day disruptions in the Oracle Internet Intelligence Map during the month. As always, there were a number of brief and unattributed disruptions observed over the course of the month, but the issues highlighted below were related to fiber cuts (and repairs) and likely problems with satellite connectivity. And while not yet a visible disruption, reportssurfaced in February that Russian authorities and major Internet providers are planning to disconnect the country from the global Internet as part of a planned experiment.

Fiber

Kicking off the month, Burkina Faso experienced brief partial disruptions to its Internet connectivity on February 1 & 2, as shown in the Country Statistics graphs below. The disruptions are also evident in the Traffic Shifts graphs below for AS25543 (Onatel), which is the country’s National Office of Telecommunications, holding a monopoly on fixed-line telecommunications there. Facebook posts from Onatel (February 1, 2) indicated that road work between the towns of Sabou and Boromo had resulted in a fiber cut, and subsequent posts made Continue reading

Last Month in Internet Intelligence: January 2019

This post is presented in conjunction with The Internet Society.

During the second half of 2018, the causes of significant Internet disruptions observed through the Oracle Internet Intelligence Map could be clustered into a few overarching areas: government-directed, cable problems, power outages, distributed denial of service (DDoS) attacks, and general technical issues. Little changed heading into 2019, with two new government-directed Internet disruptions observed in Africa, alongside disruptions caused by fiber cuts and other network issues that impacted a number of countries around the world.

Government Directed

Initially covered in last month’s overview, the Internet disruption in the Democratic Republic of Congo continued into January, lasting through the third week of the month. Government authorities reportedly cut off Internet access in the country in December to prevent “rumor mongering” in the run-up to presidential elections.

An attempted military coup in Gabon led to a day-long Internet disruption in the country. The disruption started just after 07:00 UTC on January 7, as seen in the figure below, which shows clear declines in the Traceroute Completion Ratio and BGP Routes metrics, as well as a disruption to the usual diurnal pattern seen in the DNS Query Rate metric. Although the coup Continue reading

Last Month in Internet Intelligence: December 2018

Closing out 2018, in December the Oracle Internet Intelligence team observed Internet disruptions in countries around the world due to power outages, government direction, technical faults, and possible issues relating to satellite connectivity. While these causes have become relatively common, it is interesting to note that other common reasons for Internet disruptions, including severe weather (such as typhoons and hurricanes), concerns over cheating on exams, and denial-of-service attacks did not appear to drive significant Internet disruptions observed in Oracle’s Internet Intelligence Map during the month. And while we tend to focus on Internet disruptions, it is also important to highlight that after several rounds of testing, nationwide mobile Internet access was finally activated across Cuba.

Cuba

In three tranches (based on the first two digits of a subscriber’s mobile phone number) over December 6, 7, and 8, ETECSA, Cuba’s national telecommunications company, enabled nationwide mobile Internet access. The rollout was reportedly stable, in contrast to the congestion experienced during the trials conducted several months prior. The figure below shows the gradual adoption of this newly available connectivity through changes in the DNS Query Rate. As seen in the graph, the query rate was comparatively low in the days ahead of Continue reading

Last Month in Internet Intelligence: November 2018

In November, we saw fewer significant Internet disruptions in the Oracle Internet Intelligence Map as compared to prior months. As usual, there were hundreds of brief issues with limited impact and generally unknown causes, but the most notable issues last month were due to reported DDoS attacks, problems with terrestrial and submarine cables, and general network issues.

DDoS Attacks

On November 4 and 5, several Cambodian ISPs were targeted by DDoS attacks described as the “biggest attacks in Cambodian history.” Published reports indicated that ISPs including EZECOM, SINET, Telcotech, and Digi were targeted by DDoS attacks totaling nearly 150 Gbps, causing subscriber downtime lasting as much as half a day. Disruption from the attacks was visible in the Country Statistics view for Cambodia in the Internet Intelligence Map, as shown in the figure below. However, because Internet connectivity remained generally available (albeit impaired) across the country, the impact appears nominal in the graphs.

However, when viewed at a network level, the impact of the attacks appears to be more significant. SINET, one of the ISPs targeted by the DDoS attacks, posted a Tweet on November 5 letting users know that they were under attack, and followed up Continue reading

Last Month in Internet Intelligence: October 2018

The level of significant Internet disruptions observed through the Oracle Internet Intelligence Map was lower in October, though the underlying reasons for these disruptions remained generally consistent compared to prior months. For enterprises, the importance of redundant Internet connectivity and regularly exercised failover plans is clear. Unfortunately, for state-imposed Internet outages, such planning and best practices may need to include failsafes for operations while periodically offline.

Directed disconnection

On October 10, Ethiopian Prime Minister Abiy Ahmed met with several hundred soldiers who had marched on his office to demand increased pay. The Ethiopian Broadcasting Corporation (formerly known as ETV) did not cover the soldiers marching but noted that Internet connectivity within the country had been shut off for several hours to prevent “fake news” from circulating on social media. This aligned with residents’ reports of a three-hour Internet outage. The figure below shows that the disruption began around 12:00 GMT, significantly impacting both traceroutes to, and DNS query traffic from, Ethiopia for several hours.

The impact of the Internet shutdown is also clearly evident in the figure below, which shows traceroutes into Ethio Telecom, the state-owned telecommunications service provider. Similar to the country-level graph shown above, the number of Continue reading

Last Month in Internet Intelligence: September 2018

Over the course of a given month, hundreds of Internet-impacting “events” are visible within the Oracle Internet Intelligence Map. Many are extremely short-lived, lasting only minutes, while others last for hours or days; some have a minor impact on a single metric, while others significantly disrupt all three metrics. In addition, for some events, the root cause is publicly available/known, while for other events, digging into the underlying data helps us make an educated guess about what happened. Ultimately, this creates challenges in separating the signal from the noise, triaging and prioritizing that month’s events for review in this blog post.

Having said that, in September we observed Internet disruptions due to exams, power outages, extreme weather, and submarine cable issues, as well as a number of others with unknown causes. Additionally, a third test of nationwide mobile Internet connectivity took place in Cuba.

Cuba

As noted in our August post, ETECSA (the Cuban state telecommunications company) carried out two tests of nationwide mobile Internet connectivity, which were evident as spikes in the DNS query rates from Cuba. In a Facebook post, they noted, “On August 14th was a first test that measured levels of traffic congestion and Continue reading

Last Month In Internet Intelligence: August 2018

During August 2018, the Oracle Internet Intelligence Map surfaced Internet disruptions around the world due to familiar causes including nationwide exams, elections, maintenance, and power outages. A more targeted disruption due to a DDoS attack was also evident, as were a number of issues that may have been related to submarine cable connectivity. In addition, in a bit of good news, the Internet Intelligence Map also provided evidence of two nationwide trials of mobile Internet services in Cuba.

Cuba

On August 15, the Oracle Internet Intelligence Twitter account highlighted that a surge in DNS queries observed the prior day was related to a nationwide test of mobile Internet service, marking the first time that Internet services were available nationwide in Cuba’s history. The figure below shows two marked peaks in DNS query rates from resolvers located in Cuba during the second half of the day (GMT) on the 14^th. Paul Calvano, a Web performance architect at Akamai, also observed a roughly 25% increase in their HTTP traffic to Cuba during the trial period.

This testing was reported by ETECSA (the Cuban state telecommunications company) in a Facebook post in which they noted:

The Telecommunications company of Cuba S.A. Continue reading

Last Month in Internet Intelligence: July 2018

In June, we launched the Internet Intelligence microsite, including the new Internet Intelligence Map. In July, we published the inaugural “Last Month in Internet Intelligence” overview, covering Internet disruptions observed during the prior month. The first summary included insights into exam-related outages and problems caused by fiber cuts. In this month’s summary, covering July, we saw power outages and fiber cuts, as well as exam-related and government-directed shutdowns, disrupt Internet connectivity. In addition, we observed Internet disruptions in several countries where we were unable to ascertain a definitive cause.

Power Outages

It is no surprise that power outages can wreak havoc on Internet connectivity – not every data center or router is connected to backup power, and last mile access often becomes impossible as well.

At approximately 20:00 GMT on July 2, the Internet Intelligence Map Country Statistics view showed a decline in the traceroute completion ratio and DNS query rate for Azerbaijan, related to a widespread blackout. These metrics gradually recovered over the next day. Published reports (Reuters, Washington Post) noted that the blackout was due to an explosion at a hydropower station, following an overload of the electrical system due to increased use Continue reading

« Previous 1 2 3 Next »