The Threat of Telecom Sabotage
Earlier this week, an article in New York Times captured the world’s imagination with the prospect of secret Russian submarines possessing the ability to sabotage undersea communication cables (with perhaps Marko Ramius at the helm, pictured above). While it is a bit of a Hollywood scenario, it is still an interesting one to consider, although, as we’ll see, perhaps an unrealistic one, despite the temptation to exaggerate the risk.
Submarine cable cuts occur with regularity and the cable repair industry has considerable experience dealing with these incidents. However, the vast majority of these failures are the result of accidents occurring in relatively shallow water, and not due to a deliberate actor intending to maximize downtime. There is enormous capacity and resiliency among the cables crossing the Atlantic (the subject of the New York Times article), so to even make a dent, a saboteur would need to take out numerous cables in short order.
A mass telecom sabotage event involving the severing of many submarine cables (perhaps at multiple hard-to-reach deep-water locations to complicate repairs) would be profoundly disruptive to international communications — Internet or otherwise. For countries like the U.S. with extensive local hosting, the impact Continue reading
Global Impacts of Recent Leaks
Recent routing leaks remind us why monitoring Internet routing and performance is important and requires effective tools. Routing leaks are the ‘benign cousin’ of the malicious BGP route hijack. They happen accidentally, but the result is the same: traffic to affected prefixes is redirected, lost, or intercepted. And if they happen to you, your online business and brand suffers.
In this blog, we look at examples of a full-table peer leak, an origination leak, and a small peer leak and what happens to traffic when these incidents occur. As we will see, some events can go on for years, undetected and hence, unremediated, but extremely impactful never the less. As you read this blog, keep the following questions in mind. Would you know if the events described here were happening to you? Would you know how to identify the culprit if you did?
iTel/Peer1 routing leak
Starting on 10 October at 10:54 UTC, iTel (AS16696) leaked a full routing table (555,010 routes) to Peer 1 (AS13768). Normally, iTel exports 49 routes to Peer 1; however, over the course of several minutes, it leaked 436,776 routes from Hurricane Electric (AS6939) and 229,537 Continue reading
Iran: Latest Nation to Host Critical Global Internet Infrastructure
As crippling economic sanctions are poised to be lifted by the United States, Iran is starting to emerge from its isolation as a regional and, in a very limited sense, global Internet player. Iran continues to methodically build out its Internet infrastructure, working on its domestic connectivity (including IPv6), providing service to neighboring countries (such as Iraq and Afghanistan), stockpiling limited IPv4 address space, and providing a strategic terrestrial alternative to vulnerable submarine cables.
Recently, Iran began hosting a root DNS server, thereby potentially providing this critical service to the rest of the world. In this blog, we’ll explore some of these latest developments and their challenges. In November, European Internet registrar RIPE will hold its regional operator meeting (MENOG) in Tehran, where attendees from around the world will learn firsthand about recent developments in the fast-growing Iranian Internet.
K-root Debuts in Iran
As most readers of this blog will know, when you access any resource on the Internet by name (e.g., www.cnn.com), your computer must first convert this name into an IP address (e.g., 23.235.46.73), which it then uses to gain access to Continue reading
Global Collateral Damage of TMnet leak
The Washington Post recently published a great piece about the development and current weaknesses of the Border Gateway Protocol (BGP, which is used to route all Internet traffic). This morning Telekom Malaysia (a.k.a TMnet) helped to illustrate the points made in the article by leaking almost half of the global routing table via Level 3 at 08:44 UTC.
Some of the most affected companies were those peering with Telekom Malaysia. The following graphics illustrate the impact to routes from Amazon and Cloudflare.
 |
 |
Google’s extensive peering likely insulated it from some of the effects of having its routes leaked. However, it didn’t escape the incident completely unscathed. Here is an example of a normal traceroute to Google’s data center in Council Bluffs, Iowa from Prague, which goes via Frankfurt and London before crossing the Atlantic Ocean.
trace from Prague to Google, Council Bluffs, IA at 02:45 Jun 11, 2015
1 *
2 212.162.8.253 ge-6-14.car2.Prague1.Level3.net 16.583
3 4.69.154.135 ae-3-80.edge3.Frankfurt1.Level3.net 22.934
4 4.68.70.186 Level 3 (Frankfurt, DE) 23.101
5 209.85.241.110 Google (Frankfurt, DE) 23.796
6 209.85.250.143 Google (Frankfurt, DE) 24.086
7 72.14.235.17 Google (London, GB) 32.709
8 209.85.247.145 Google (New York City) 103.091
9 216.239.46.217 Google (Council Bluffs) 133.098
10 209.85.250.4 Google (Council Bluffs) 133.245
11 216.239.43.217 Google (Council Bluffs) 133. Continue reading
Earthquake rocks Internet in Nepal
Saturday’s earthquake in Nepal, which claimed the lives of at least 4,000 victims and injured many more, took a toll on the country’s Internet connectivity, which was already one of the least developed in the region. A recent evaluation of Internet infrastructure in South Asia commissioned by the United Nations Economic and Social Commission for Asia and the Pacific (ESCAP) classified Nepal’s international connectivity as ‘weak’ and its fixed and mobile infrastructure as ‘limited’.
While the loss of Internet connectivity pales in comparison to the loss of life, the ability to communicate both domestically and internationally will be crucial in coming days for the coordination of relief efforts already underway. Innovative services from Facebook and Google to facilitate communicating the status of those affected by the massive earthquake would be largely useless if Nepal had been knocked entirely offline. In fact, Nepal’s international links generally survived the earthquake, however last mile connectivity is another matter.
As we reported on Saturday, we began seeing severe Internet outages and instabilities immediately following the earthquake at 6:11 UTC. On the left is a timeline of outages through today and on the right is the volume of DNS queries Continue reading
IPv4 Address Market Takes Off
As the available supply of IPv4 addresses dwindles, the market for these virtual commodities is heating up. In recent months, the pace of the address transfers has greatly accelerated as evidenced by RIPE’s table of IPv4 transfers, as well as the increasing number of IPv4 brokers facilitating the exchange of IPv4 address space. However, the transfer of IPv4 address space isn’t always problem-free and, in this blog, we’ll review this new trend and some of the issues that can arise.
Buying and selling IPv4
In 2011, when Microsoft paid $7.5 million for 666,624 IPv4 addresses as part of the Nortel bankruptcy, observers wondered whether this development would usher in the era of the commercial sale of IPv4 address space. As statistics from European registrar RIPE show, the market may have had a slow start, but we’re in that new era now.
RIPE’s table of transfers of provider independent IPv4 address clearly shows a rapidly increasing rate of transfers of IPv4 address blocks and unique IPv4 addresses. The following two graphs illustrate the uptick in recent months of address space movement. February 2015 saw that most organizational transfers (373), while November 2014 saw the Continue reading
UK traffic diverted through Ukraine
On the heels of the BGP leak yesterday that briefly impaired Google services around the world, comes another routing incident that impacted some other important Internet services.
Beginning on Saturday, Ukrainian telecom provider, Vega, began announcing 14 British Telecom (BT) routes, resulting in the redirection of Internet traffic through Ukraine for a handful of British Telecom customers. Early yesterday morning, Vega announced another 167 BT prefixes for 1.5 hours resulting in the rerouting of additional traffic destined for some of BT’s customers, including the UK’s Atomic Weapons Establishment, the “organization responsible for the design, manufacture and support of warheads for the United Kingdom’s nuclear deterrent.”
Background
In early 2013, Ukrainian provider Vega (AS12883) became a reseller of BT services, but prior to Saturday had never announced any BT routes. Then, in the middle of a weekend night in Europe (02:37 UTC on Saturday, March 7th), Vega began announcing 14 prefixes typically announced by AS2856 of BT. These prefixes are listed below.
109.234.168.0/21 Thales Transport and Security Ltd (Barnet, GB)
109.234.169.0/24 Thales Transport and Security Ltd (Ealing, GB)
144.87.142.0/24 Royal Mail Group Limited (Sheffield, GB)
144.87.143.0/24 Royal Mail Group Limited (Chesterfield, GB)
147.182.214.0/24 Black & Veatch (Manchester, GB)
193.113.245.0/24 BT - 21CN (GB)
193.221.55.0/24 Svenska Cellulosa Aktiebolaget SCA (GB)
193. Continue reading
Routing Leak briefly takes down Google
This morning, users of Google around the world were unable to access many of the company’s services due to a routing leak in India. Beginning at 08:58 UTC Indian broadband provider Hathway (AS17488) incorrectly announced over 300 Google prefixes to its Indian transit provider Bharti Airtel (AS9498).
Bharti in turn announced these routes to the rest of the world, and a number of ISPs accepted these routes including US carriers Cogent (AS174), Level 3 (AS3549) as well as overseas incumbent carriers Orange (France Telecom, AS5511), Singapore Telecom (Singtel, AS7473) and Pakistan Telecom (PTCL, AS17557). Like many providers around the world, Hathway peers with Google so that their customers have more direct connectivity with Google services. But when that private relationship enters the public Internet the result can be accidental global traffic redirection.
Last fall, I wrote two blog posts here and here about the issues surrounding routing leaks such this one. Routing leaks happen regularly and can have the effect of misdirecting global traffic. Last month, I gave a talk in the NANOG 63 Peering Forum entitled “Hidden Risks of Peering” that went over some examples of routing leaks like this one.
Below is a graph showing the Continue reading
A Baker’s Dozen, 2014 Edition
As is our annual tradition, this blog provides a year-end review of how the Internet providers at the top of our IP Transit Intelligence global rankings (formerly, Renesys’ Market Intelligence) fared over the previous year. The structure and performance of the Internet remains a huge blind spot for most enterprises, even those critically dependent on it for business operations. Whether it’s the next 3 billion people coming online, poor performance due to suboptimal routing, or security breaches of a trust-based Internet infrastructure, Dyn provides critical insight into the structure and performance of the Internet, both real-time and historical, via its Internet Intelligence products. More importantly, our services help our customers make the changes necessary to optimize Internet availability, reliability, and reach. This blog reviews a single very small slice of our data related to the sizes of the top global players as it pertains to the markets and customers they serve.
Back in 2008, we chose to look at the 13 providers that spent at least some time in the Top Ten that year, hence the name “Baker’s Dozen“. We repeated that exercise in 2009, 2010, 2011, Continue reading
The Vast World of Fraudulent Routing
As network security engineers have attempted to categorize blocks of IP addresses associated with spam or malware for subsequent filtering at their firewalls, the bad guys have had to evolve to continue to target their victims. Since routing on the global Internet is based entirely on trust, it’s relatively easy to commandeer IP address space that belongs to someone else. In other words, if the bad guys’ IP space is blocked, well then they can just steal someone else’s and continue on as before.
In an attempt to cover their tracks, these criminals will sometimes originate routes using autonomous system numbers (ASNs) that they don’t own either. In one of the cases described below, perpetrators hijacked the victim’s ASN to originate IP address space that could have plausibly been originated by the victim. However, in this case, the traffic was misdirected to the bad guy and an unsophisticated routing analysis would have probably shown nothing amiss.
The weakness of all spoofing techniques is that, at some point, the routes cross over from the fabricated to the legitimate Internet — and, when they do, they appear quite anomalous when compared against historical data and derived business Continue reading
Internet for the Next 3 Billion
Last month, I traveled to Doha, Qatar to participate in the ITU’s Telecom World conference. While there I got to understand how a satellite provider brings Internet access to South Sudan using medium-earth orbit satellites and, amazingly, achieves terrestrial latencies to a region where reliable terrestrial connections simply don’t exist! The mission of this company is to help close the digital divide by extending Internet access to the estimated three billion people on the planet who are currently not served. Our measurements show the that performance improvement over traditional satellite can be dramatic.
ITU Telecom World
First, let me say a few words about the conference itself and then I’ll review this intriguing new satellite service. In Doha, I was on a panel entitled Affordable International Backhaul and chaired by Abu Saaed Kahn of LIRNEAsia, a telecommunications policy institute primarily focused on the Asia-Pacific region.
On-going BGP Hijack Targets Palestinian ISP
It’s a new year, but some things never change. In the past few days we have observed a spate of incidents of routing misbehavior including two man-in-the-middle routing hijacks conducted in the past couple of days by A2B Internet out of the Netherlands.
Beginning at 00:33:44 UTC on Thursday, 8 January, we began observing a routing hijack of IP address space normally announced by Mada Telecom (AS51047), a Palestinian ISP with presence in both Gaza and the West Bank. Beginning at that time, A2B Internet B.V. (AS51088) began announcing 46.244.81.0/24, which is a more-specific route of 46.244.80.0/23, normally announced by Mada.
Traceroutes directed to this address space are presently being re-directed to A2B Internet’s network in the Netherlands before continuing on to Palestine. For example:
trace from Cyberjava, Malaysia to Mada Telecom, PS on Jan 09, 2015
1 *
2 x.x.x.x (Cyberjaya, Malaysia) 3.442
3 113.23.163.57 (Extreme Broadband, Malaysia) 0.696
4 113.23.190.109 (Extreme Broadband, Malaysia) 1.222
5 218.189.12.101 global.hgc.com.hk 35.854
6 218.189.8.102 global.hgc.com.hk 36.742
7 118.143.224.243 (Hutchison, Singapore) 41.628
8 218.189.8.142 (Hutchison, Amsterdam) 190.787
9 195.219.150.6 (Tata, Amsterdam, NL) 213.494
10 46.244.0.4 (A2B Internet, NL) 200.990
11 141.136.97.5 (GTT, Amsterdam) 268.366
12 4.68.70.97 xe-5-0-1.edge3.Amsterdam.Level3.net 300.909
13 4. Continue readingSomeone Disconnects North Korea – Who?
North Korea went off the Internet Monday, 22 December 2014, at 16:15 UTC (01:15 UTC Tuesday in Pyongyang) after more than 24 hours of sustained weekend instability. Dyn continually measures the connectivity and performance of more than 510,000 individual networks worldwide, identifying impairments to Internet commerce. It’s a rare event these days when an entire country leaves the Internet (as Egypt did, or Syria). Even so, when North Korea’s four networks went dark, we were not entirely surprised, based on the fragility of their national connectivity to the global Internet.
Who caused this, and how? A long pattern of up-and-down connectivity, followed by a total outage, seems consistent with a fragile network under external attack. But it’s also consistent with more common causes, such as power problems. Point causes such as breaks in fiberoptic cables, or deliberate upstream provider disconnections, seem less likely because they don’t generate prolonged instability before a total failure. We can only guess. The data themselves don’t speak to motivations, or distinguish human factors from physical infrastructure problems.
As the sun rises in Pyongyang, the national Internet disconnection continues. An outage of this duration is not without precedent for North Korea. As we’ve written before, Continue reading
What’s Next for Cuba?
Nearly two years ago, we broke the story about the activation of the first submarine cable connecting Cuba to the global Internet – a cable that, prior to its activation in January 2013, mysteriously lay dormant on the ocean floor for nearly two years. When the Cuban government issued a confirmation in the days following our report, it contained the following statement:
|
|
In other words, Cubans should not expect greater access to the Internet just because the ALBA-1 submarine cable was now in operation. Yesterday’s historic agreement to begin normalizing relations between Cuba and the United States contains a pledge by the Cuban government to “greatly expand its citizens’ access to the Internet.” What exactly this pledge entails will determine how the Internet evolves in Cuba in the near term. Decision makers in Cuba should look at another country that recently opened up its telecom sector and is presently experiencing an explosion in Internet growth: Myanmar.
Cuban Isolation
The isolation of Cuba is plainly evident when looking at a map of the submarine cables in the Continue reading
Use Protection if Peering Promiscuously
Last week, I wrote a blog post discussing the dangers of BGP routing leaks between peers, illustrating the problem using examples of recent snafus between China Telecom and Russia’s Vimpelcom. This follow-up blog post provides three additional examples of misbehaving peers and further demonstrates the impact unmonitored routes can have on Internet performance and security. Without monitoring, you are essentially trusting everyone on the Internet to route your traffic appropriately.
In the first two cases, an ISP globally announced routes from one of its peers, effectively inserting itself into the path of the peer’s international communications (i.e., becoming a transit provider rather than remaining a peer) for days on end. The third example looks back at the China Telecom routing leak of April 2010 to see how a US academic backbone network prioritized bogus routes from one of its peers, China Telecom, to (briefly) redirect traffic from many US universities through China.
Recap: How this works
To recap the explanation from the previous blog (and to reuse the neat animations our graphics folks made), we first note that ISPs form settlement-free direct connections (peering) in order to save on the cost of sending Continue reading
Chinese Routing Errors Redirect Russian Traffic
In recent weeks, Russian President Vladimir Putin announced a plan to enact measures to protect the Internet of Russia. In a speech to the Russian National Security Council he said, “we need to greatly improve the security of domestic communications networks and information resources.” Perhaps he should add Internet routing security to his list because, on a number of occasions in the past year, Russian Internet traffic (including domestic traffic) was re-routed out of the country due to routing errors by China Telecom. When international partners carry a country’s domestic traffic out of the country, only to ultimately return it, there are inevitable security and performance implications.
Last year, Russian mobile provider Vimpelcom and China Telecom signed a network sharing agreement and established a BGP peering relationship. However, as can often happen with these relationships, one party can leak the routes received from the other and effectively insert itself into the path of the other party’s Internet communications. This happened over a dozen times in the past year between these two providers. This is a general phenomenon that occurs with some regularity but isn’t often discussed in BGP security literature. In this blog post, we’ll explore the issue Continue reading
Renesys Team Launches Dyn Research
Welcome to the new Dyn Research Blog! We’re certainly glad you’re here, and we hope you like the snazzy new look.
Since the Renesys team joined Dyn in May, the number one question we’ve received is “will you keep publishing the blog?” The answer is yes, absolutely, and we hope to bring you some diverse perspectives on Internet performance from other members of the Dyn technical team as well. Please do let us know what you think of the new Dyn Research Blog, and feel free to suggest topics you’d like us to cover.
A moment for reflection
Looking back over the eight years that we’ve been publishing our observations about Internet structure and operations, I’m struck mostly by how you, our audience, have evolved and grown. In the early days, news about Internet infrastructure appealed to a pretty narrow group of readers within the network operations community. We never had to buy beer at conferences like NANOG, but the rest of the world was more or less content to ignore the dirty details of IPv6, peering and depeering, Net Neutrality, and the evolution of the IP wholesale transit industry.
Why Far-Flung Parts of the Internet Broke Today
VolumeDrive is a Pennsylvania-based hosting company that uses Cogent and (since late May of this year) Atrato for Internet transit. A routing leak this morning by VolumeDrive was passed on to the global Internet by Atrato causing disruptions to traffic in places as far-flung from the USA as Pakistan and Bulgaria.
Background
The way Internet transit is supposed to work in BGP is that a provider announces the global routing table to its customers (i.e., a large number of routes). Then, in turn, the customers announce local routes to their respective providers (generally a small number of routes). Each customer selects the routes it prefers from the options it receives. When a transit customer accidentally announces the global routing table to back one of its providers, things get messy. This is what happened earlier today and it had far-reaching consequences.
At 06:49 UTC this morning (18-September), VolumeDrive (AS46664) began announcing to Atrato (AS5580) nearly all the BGP routes it learned from Cogent (AS174). The resulting AS paths were of the following format:
… 5580 46664 174 …
Normally, VolumeDrive announces 39 prefixes (networks) to Atrato: 27 it originates itself and 12 it transits for two of its downstream Continue reading
Sprint, Windstream: Latest ISPs to hijack foreign networks
Last year my colleague Jim Cowie broke a story about routing hijacks that resulted in Internet traffic being redirected through Iceland and Belarus. Unfortunately, little has changed since then and the phenomenon of BGP route hijacking continues unabated and on an almost daily basis.
In the past three days, the situation has gone from bad to downright strange as we have observed a flurry of this activity. Now, for the first time, we’re seeing major US carriers, Sprint and Windstream, involved in hijacking, along with the return of an operation out of Poland targeting Brazilian networks. We see router misconfigurations regularly in BGP data – could these incidents also be explained by simple command-line typos?
Route hijacking continues
In May my colleague, Earl Zmijewski gave a presentation about routing hijacks at the LINX 85 meeting, describing a comprehensive system that can be used to identify suspicious hijacks on a global basis and without any prior knowledge about the networks involved. While we now detect suspicious routing events on an almost daily basis, in the last couple of days we have witnessed a flurry of hijacks that really make you scratch your head.
To mention a few recent events, last Continue reading
Internet Touches Half Million Routes: Outages Possible Next Week
There was minor consternation in Internet engineering circles today, as the number of IPv4 networks worldwide briefly touched another magic “power of 2″ size limit. As it turns out, 512K (524,288 to be exact, or 2-to-the-19th power) is the maximum number of routes supported by the default TCAM configuration on certain aging hardware platforms.
The problem is real, and we still haven’t seen the full effects, because most of the Internet hasn’t yet experienced the conditions that could cause problems for underprovisioned equipment. Everyone on the Internet has a slightly different idea of how big the global routing table is, thanks to slightly different local business rules about peering and aggregation (the merging of very similar routes to close-by parts of the Internet address space). Everyone has a slightly different perspective, but the consensus estimate is indeed just under 512K, and marching higher with time.
The real test, when large providers commonly believe that the Internet contains 512K routes, and pass that along to all their customers as a consensus representation of Internet structure, will start later this week, and will be felt nearly everywhere by the end of next week.
Enterprises that rely on the Internet for delivery of Continue reading