2020: A Record Year for MANRS

Over the past year, COVID-19 underlined the importance of a secure and resilient Internet to ensure we stay connected online. For MANRS, this meant even more incentive to work with network operators, Internet exchange points (IXPs), and content delivery network (CDN) and cloud providers to ensure data went where it was supposed to go via secure paths.

It was, therefore, really encouraging to see a record number of participants joining MANRS last year: the number almost doubled from 317 participants at the beginning of the year to 588 participants as of 31 December 2020. MANRS participants now manage 651 autonomous systems from over 60 countries across all continents.

MANRS contributed to the decline in reported routing incidents from more than 5,000 in 2017 to below 4,000 in 2020, making the entire Internet more secure for everyone. While we cannot claim full credit, we can attribute the fewer routing incidents to the increasing number of network operators implementing best routing practices.

The year also saw us launching a new program for CDN and cloud providers in collaboration with eight founding participants: Akamai, Amazon Web Services, Azion, Cloudflare, Facebook, Google, Continue reading

MANRS Welcomes 500th Network Operator

Today, we are glad to share a milestone for the Mutually Agreed Norms for Routing Security (MANRS) initiative: the number of participants in the network operator program has reached 500.

By joining the community-driven initiative, these network operators, big and small, from around the world have taken specific, concrete actions to improve the resilience and security of the Internet’s inherently insecure routing infrastructure.

Systemic security issues about how traffic is routed on the Internet make it a relatively easy target for criminals. MANRS helps reduce the most common routing threats and increase efficiency and transparency among Internet service providers (ISPs) on peering relationships.

The growth of the network operator program – the oldest among three today – has been accelerating in recent years. Launched in 2014 with a group of nine operators, the number of participants in the program took four years to reach 100 in 2018 and has risen sharply in the last two years, with 156 joining in 2019 and 244 so far in 2020.

The 500 network operators manage 651 autonomous systems in total, as some of them manage multiple networks.

Meanwhile, the Internet Exchange Point (IXP) program, which we launched in 2018, now has 60 Continue reading

Why Is So Much Internet Traffic Leaving Pakistan?

This article appeared first on the APNIC website.

At the recent SANOG meeting held in my homeland, Pakistan, I wanted to provide the local community with some insights into the importance of Internet exchanges (IXs), specifically the need to host content locally.

Knowing that data is king among network operators, I set up a virtual machine as soon as I arrived to collect information on several key metrics, including latency and the hosting location of .pk domains. Needless to say, the results were surprising.

How long does it take to connect to public Domain Name System (DNS) services?

First, I tested for latency, specifically the time it takes to PING three of the most popular public DNS services: Cloudflare DNS (1.1.1.1), Google Public DNS (8.8.8.8), and Quad9 (9.9.9.9). PING is not the best way to test DNS but this is for reachability purpose only.

Before leaving my home in Sydney, Australia, I did the same to offer a comparison. As you can see from the results in Figure 1, all were below 1ms.

The results for Pakistan were Continue reading

Let’s Improve Routing Security at APRICOT 2020

Internet builders in Asia-Pacific get together around this time every year at APRICOT to learn from each other and other leaders from around the world. Routing security will be a key theme, and we will be sharing in multiple sessions why the MANRS initiative is important to the global routing system.

Also called the Asia-Pacific Regional Internet Conference on Operational Technologies, the conference is the largest meeting of the technical community in the region. It draws many of the world’s best Internet engineers, operators, researchers, service providers, and policy enthusiasts from over 50 countries to learn, share, and network.

Held annually, the ten-day meeting consists of workshops, tutorials, and conference sessions, birds-of-a-feather (BoFs) sessions, and peering forums all with the goal of spreading the knowledge needed to run and expand the Internet.

Technical training workshops will run from Feb 12 to 16, and the conference itself from 17 to 21 in Melbourne, Australia.

Our team at the Mutually Agreed Norms for Routing Security (MANRS) initiative will speak at various sessions throughout the conference, including the Resource Public Key Infrastructure (RPKI) Deployathon on 17 February that I will facilitate. I will also be chairing the inaugural APNIC Routing Security/RPKI SIG on 20 February.

RPKI Continue reading

Route Leak Causes Major Google Outage

Google recently faced a major outage in many parts of the world thanks to a BGP leak. This incident that was caused by a Nigerian ISP – Mainone – occurred on 12 November 2018 between 21.10 and 22.35 UTC, and was identified in tweets from the BGP monitoring service BGPMon, as well as the network monitoring provider Thousand Eyes.

Google also announced the problem through their status page:

We’ve received a report of an issue with Google Cloud Networking as of Monday, 2018-11-12 14:16 US/Pacific. We have reports of Google Cloud IP addresses being erroneously advertised by internet service providers other than Google. We will provide more information by Monday, 2018-11-12 15:00 US/Pacific.

In order to understand this issue, MainOne Inc (AS37282) is peering at IXPN (Internet Exchange Point of Nigeria) in Lagos where Google (AS151169) and China Telecom (AS4809) are also members.

Google (AS15169) advertise their prefixes (more than 500) through the IXPN Route Server, where PCH (Packet Clearing House) collects a daily snapshot of BGP announcements of IXPN. Unfortunately, 212 prefixes (aggregates of those 500+ announcements) from Google were leaked, which was recorded by BGPMon and RIPEstat.

Looking at the RIPE stats it is evident Continue reading

Routing Security boosted by major network operators

There have been some important developments towards improving routing security over the past few weeks, with announcements at NLNOG and AusNOG, as well as from Cloudflare about commitments to validate IP prefixes and reduce route leaks and hijacks. This supports the work we’ve being doing with the MANRS initiative to raise awareness of this issue, and to persuade network operators to take collaborative responsibility for this critical aspect of the Internet.

Cloudflare to deploy RPKI

Cloudflare has been a long-time advocate of routing security, and during their recent Crypto Week, they announced that they’ll be deploying RPKI on their networks. Resource Public Key Infrastructure (RPKI) allows IP address prefixes and AS numbers to be cryptographically verified (using Route Origin Authorization), and therefore provides some assertion that the holders of these have the right to announce them. The use of RPKI is included as one of the four MANRS actions “Global Validation – facilitating validation of routing information on global scale” which includes the creation of ROAs and the maintenance of accurate data in Internet Routing Registries (IRRs).

Cloudflare also announced GoRTR, which is an open-source implementation of the RPKI to Router (RTR) protocol (see RFC 6810). This is Continue reading

Even Better MANRS During August

We already discussed the MANRS activities during SANOG 32 where we organised a Network Security Workshop and signed an MoU with the ISP Association of Bangladesh (ISPAB), but the Internet Society was also involved with three other events during the month of August. This included the Symposium on Internet Routing Security and RPKI, VNIX-NOG 2018 and the inaugural INNOG 1.

Symposium on Internet Routing Security and RPKI

ZDNS along with CNCERT organised a symposium on 17th August at Crowne Plaza Beijing to discuss routing security issues and how RPKI can help address this problem. There were many prominent participants representing local, regional and international entities including Baidu, Tencent, Alibaba, Huawei, ZTE, the Chinese Academy of Sciences, APNIC, ICANN, along with the Internet Society.

Dr Stephen Kent (BBN) was the keynote speaker, having played an important role in the SIDR (Secure Internet Domain Routing) Working Group at the IETF (Internet Engineering Task Force) and also co-authored many RFCs (Request for Comments) on RPKI. He discussed the ideas behind RPKI and Route Origin Authorization/Validation.

George Michaelson (APNIC) who along with his colleague Geoff Huston co-authored RFC 6483 – Validation of Route Origination Using the Resource Certificate Public Key Infrastructure (PKI) and Route Origin Authorizations Continue reading

SANOG 32 – Another Success Story for MANRS

The SANOG 32 meeting was held on 2-10 August 2018 in Dhaka, Bangladesh, which marked fifteen amazing years of collaboration between network operators in the South Asia region. The Internet Society is proud to support the SANOG fellowship programme that provides opportunities for network engineers from countries in the region to attend, as well as organising the Network Security workshop during the event.

SANOG 32 also saw another MANRS milestone reached when the ISP Association of Bangladesh (ISPAB) signed a Memorandum of Understanding (MoU) with the Internet Society. ISPAB is a membership-based, not-for-profit organization that provides a forum for Bangladeshi ISPs to discuss technology, policy, regulatory and commercial issues and find collective solutions.

In accordance with the MoU, both ISPAB and ISOC will work together to promote and support MANRS, to encourage network operators in Bangladesh to join the initiative. There are currently only two MANRS participants in the country, so being able to increase engagement with the networking community is a welcome development.

Dr Philip Smith (NSRC and Chair of SANOG Programme Committee) also provided a MANRS update during the conference session.

The Network Security workshop attracted 40 participants and was lead by MANRS founding member Matsuzaki Yoshinobu (IIJ) and Continue reading

A Deeper Dive Into Public DNS Resolver Quad9

There are plenty of public DNS resolvers. The best known was Google Public DNS i.e. 8.8.8.8 and 8.8.4.4 for IPv4 and 2001:4860:4860::8888 and 2001:4860:4860::8844 for IPv6. But there are a few other options available now, each with different policies and technical features.

Two new Public DNS resolvers were recently launched. Quad9 (launched Nov 2017) and 1dot1dot1dot1 (launched Apr 2018). We have already covered 1.1.1.1 in detail in a recent blog. So let’s talk about Quad9 (9.9.9.9).

The Global Cyber Alliance (GCA), an organization founded by a partnership of law enforcement (New York County District Attorney and City of London Police) and research (Center for Internet Security – CIS) organizations focused on combating systemic cyber risk in real, measurable ways, partnered with IBM and Packet Clearing House (PCH) to launch a Global Public Recursive DNS Resolver Service. Quad9 protects users from accessing known malicious websites, leveraging threat intelligence from multiple industry leaders; it currently blocks up to two million threats per day.

A handy little infographic on the Quad9 website helps show how it works. Essentially, you set up Quad 9 as your Continue reading

Working Together with APNIC on Routing Security and MANRS in Asia Pacific

The Internet Society and APNIC signed a Memorandum of Understanding (MoU) to cooperate in supporting the MANRS initiative in the Asia Pacific Region. Paul Wilson (APNIC) and Rajnesh Singh (ISOC) signed the MoU in Brisbane, Australia on 13 June 2018.

It’s an exciting moment for everyone who believes that Internet routing security issues can be resolved through collaboration, providing limitless opportunities for good. The MoU formalises the existing long-term relationship between the two organizations to have a global, open, stable and secure Internet.

The MoU focuses on capacity building to undertake initiatives and activities to promote awareness of MANRS in the Asia-Pacific region, to cooperate and render mutual assistance, and to encourage the attendance of APNIC members to meetings, seminars, workshops and/or conferences on routing security.

Both organizations have agreed to exchange research information and training materials (whether printed, audio or visual) related to routing security in general. APNIC has a proven record of delivering hands-on and online quality training and providing analytical research data.

We look forward to welcoming more MANRS members from the Asia Pacific region, and working together with APNIC to improve routing security around the world.

The post Working Together with APNIC on Routing Security and Continue reading

APNIC Labs/CloudFlare DNS 1.1.1.1 Outage: Hijack or Mistake?

At 29-05-2018 08:09:45 UTC, BGPMon (A very well known BGP monitoring system to detect prefix hijacks, route leaks and instability) detected a possible BGP hijack of 1.1.1.0/24 prefix. Cloudflare Inc has been announcing this prefix from AS 13335 since 1st April 2018 after signing an initial 5-year research agreement with APNIC Research and Development (Labs) to offer DNS services.

Shanghai Anchang Network Security Technology Co., Ltd. (AS58879) started announcing 1.1.1.0/24 at 08:09:45 UTC, which is normally announced by Cloudflare (AS13335). The possible hijack lasted only for less than 2min. The last announcement of 1.1.1.0/24 was made at 08:10:27 UTC. The BGPlay screenshot of 1.1.1.0/24 is given below:

Anchang Network (AS58879) peers with China Telecom (AS4809), PCCW Global (AS3491), Cogent Communications (AS174), NTT America, Inc. (AS2914), LG DACOM Corporation (AS3786), KINX (AS9286) and Hurricane Electric LLC (AS6939). Unfortunately, Hurricane Electric (AS6939) allowed the announcement of 1.1.1.0/24 originating from Anchang Network (AS58879). Apparently, all other peers blocked this announcement. NTT (AS2914) and Cogent (AS174) are also MANRS Participants and actively filter prefixes.

Dan Goodin (Security Editor at Ars Technica, who extensively covers malware, computer espionage, botnets, and hardware hacking) reached Continue reading

What Happened? The Amazon Route 53 BGP Hijack to Take Over Ethereum Cryptocurrency Wallets

Yesterday, we published a blog post sharing the news and some initial details about Amazon’s DNS route hijack event to steal Ethereum cryptocurrency from myetherwallet.com. In this post, we’ll explore more details about the incident from the BGP hijack’s perspective.

As noted by Dyn, CloudFlare, and various other entities who monitor Internet routing and health, Amazon’s Route 53 (the DNS service offered by AWS) prefixes were hijacked. A BGP update taken from Isolario suggests that on 24 April, its BGP feeders were correctly receiving 205.251.192.0/23, 205.251.194.0/23, 205.251.196.0/23, 205.251.198.0/23, originated from Amazon (AS16509), until 11:04:00 (UTC). But, at 11:05:41 (UTC), Isolario recorded the first more specific /24 malicious announcements via BGP feeder and the announcements originated from eNET (AS10297) to its peer 1&1 Internet SE (AS8560). Click to enlarge image.

RIPE Stats collected the first more specific malicious advertisement at 11:05:42 (UTC) originating from eNET (AS10297), but this time through peer Hurricane Electric (AS6939).

Exactly at the same time, 11:05:42 (UTC), the Isolario BGP feeder received another update originating from eNET (AS10297) and it was also coming via Hurricane Electric (AS6939). Click to enlarge image.

Hurricane Electric has a worldwide Continue reading

Routing Security BoF – APRICOT 2018

On Sunday, 25 February, the first day of APRICOT 2018, a “Routing Security BoF” (birds of a feather: An informal discussion group) was organized to address the ever-growing routing related incidents happening on daily basis. We have discussed routing security in general within the Asia Pacific region but there was a need to have a platform for open and candid discussion among the network operator community to find a possible way forward, where operators can share their approach in securing their own infrastructure and keeping the internet routing table clean as well.

A quick introduction was provided by the moderator (Aftab Siddiqui) on why it is important to have this BoF. Here are the introductory slides:

The first technical community presenter was Yoshinobu Matsuzaki (Maz) from Internet Initiative Japan (IIJ), the first ISP in Japan started in 1992. IIJ is one of the few ISPs in the region implementing prefix filtering, source address validation for their end customers, and making sure that all their routing information is reflecting the current status in the peeringdb for AS2497. IIJ was the first Asia Pacific ISP to join MANRS (Mutually Agreed Norms for Routing Security), a global initiative, supported by the Continue reading

Busy Week for MANRS, Routing Security, and More at APRICOT 2018

APRICOT 2018 is underway in in Kathmandu, Nepal, and as usual the Internet Society is an active participant in many areas of Asia Pacific’s largest international Internet conference. The workshops are taking place this week, with the conference happening next week. Here are some of the conference activities where we’ll be.

Routing Security BoF

On Sunday, 25 February, from 18:00 to 19:00 (UTC +05:45), Aftab Siddiqui and Andrei Robachevsky will moderate a Birds of a Feather (BoF) session on routing security. From the abstract, the session will provide a space where “…operators can share their approach in securing their own infrastructure and keeping the internet routing table clean as well. Also, this will provide a platform to review and highlight various BCOP documents to address routing security.” The Mutually Agreed Norms for Routing Security (MANRS) initiative is a key piece of the routing security puzzle.

Tech Girls Social

On Monday, 26 February, from 13:00 to 14:00 (UTC +05:45), Salam Yamout will be speaking at the Tech Girls Social. This session provides a space for APRICOT participants to talk and network in an open, friendly environment. The event is open to ANYONE who is interested and is not restricted to Continue reading

PNG-IX Network Security Workshop

The Internet Society (Aftab Siddiqui) and APNIC (Tashi Phuntsho) jointly conducted a Network Security Workshop in Port Moresby, Papua New Guinea (PNG) on 3-5 October 2017. This was arranged for current and potential members of the first neutral Internet Exchange Point (IX) in the country called PNG-IX, at the request of NICTA – the National Information and Communications Technology Authority – a government agency responsible for the regulation and licensing of Information Communication Technology (ICT) in Papua New Guinea. NICTA is also a key partner in establishing the Internet Exchange in PNG.

This first half of Day 1 (3 October) was dedicated to the PNG-IX awareness., such the role of an IX, how it works, why an IX has been established in PNG and why everyone should peer in order to achieve both short- and long-term benefits to the local Internet ecosystem. NICTA CEO Charles Punaha, NICTA Director Kila Gulo Vui, and APNIC Development Director Che-Hoo Cheng shared their views  

There were more than 40 participants in the Network Security workshop, with diverse backgrounds ranging from enterprise environments, state universities, financial institutions, telcos and ISPS. The training alumni completed lab work and learned about important security topics such as Continue reading

BGPSec – A reality now

The Secure Inter Domain Routing (SIDR) initiative held its first BoF at IETF 64 back in November 2005, and was established as a Working Group in April 2006. Following the Youtube Hijack incident in 2008, the need to secure BGP became increasingly important and SIDR WG charter explains it well:

• Is an Autonomous System (AS) authorized to originate an IP prefix
• Is the AS-Path represented in the route the same as the path through which the NLRI traveled.

This last vulnerability was the basis for defining an AS Path validation specification which has become known as BGPsec.

BGPsec attempts to assure a BGP peer that the content of a BGP update it has received, correctly represents the inter-AS propagation path of the update from the point of origination to the receiver of the route.

So far, 39 RFCs have originated from the SIDR WG, with three drafts currently under discussion. Seven RFCs were published last month (September 2017) providing a big boost to the securing routing work:

• RFC 8205 (was draft-ietf-sidr-bgpsec-protocol) – BGPsec Protocol Specification
• RFC 8206 Continue reading

VNIX-NOG 2017 held in Ho Chi Minh City

VNNIC, the National Internet Registry (NIR) of Vietnam, organized 2 major events in Ho Chi Minh City. VNNIC IP Member meeting on 24th August at VNNIC HCMC Office and second VNIX-NOG event on 25th August at Saigon Prince Hotel and was attended by our Deploy360 colleague Aftab Siddiqui. NetNam, a local ISP, provided Internet connectivity for the NOG event and did a great job by providing a dual stack IPv4 and IPv6 network.

The first event was for VNNIC IP members where reports were presented and NIR policies discussed, but the NOG event the following day was attended by around 70 people from various organizations including all the major telcos and ISPs in the country, the Internet Society, APNIC, Google and BBIX.

The opening speech was provided by Nguyen Hong Thang – Deputy Director of VNNIC, and a welcome note was added by Vu The Binh – General Secretary of Vietnam Internet Association (VIA) . Presentations from the event are available here, but the emphasis was on local participants to share their experience and local community engagement.

Nguyen Tran Hieu (VNIX) shared some network statistics of the Hanoi and Ho Chi Minh City nodes, which unfortunately showed that most members Continue reading