Andrei Robachevsky, Author at NetworkingNexus.net

Andrei Robachevsky

Author Archives: Andrei Robachevsky

New Features in MANRS Observatory: More Informative, Intuitive, and Easy to Use

In August 2019, the Internet Society supported the Mutually Agreed Norms for Routing Security (MANRS) initiative by creating a platform to visualize its members’ routing security data from around the globe. The MANRS Observatory’s interactive dashboard allows networks to check their progress in improving their routing security.

Last week, we updated some key features of the MANRS Observatory guided by member feedback. Below we share a summary of those changes.

Please note, detailed statistics and reports for specific networks are only available to MANRS participants. Your organization can become an MANRS member for free, and join a global group of people committed to making the Internet safer for us all. Find out how.

MANRS Observatory 3.0.1: Latest updates

Shorter reporting cycle
Improved favorite functionality
Access to RIPEstat widget
Change to how we round numbers

1. Shorter reporting cycle

Previously the MANRS Observatory provided status report updates up to 31 days after members’ had added their latest figures. While this wasn’t a real problem when looking at general trends, it was an issue for network operators who use the platform to check their network conformance. It was also an issue for the MANRS team, as we Continue reading

MANRS Welcomes Three New CDN and Cloud Participants

The MANRS Content Delivery Network (CDN) and Cloud Program continues to grow in numbers and in strength with three new participants.

Hostmein, Verisign, and Vultr have deepened their commitment to strengthening the security and resilience of the Internet’s global routing system. Participants of this program, which launched in March 2020, implement important practices for mitigating common routing security threats.

Joining means committing to taking five mandatory, and one optional, security-strengthening actions. These include preventing propagation of incorrect routing information and traffic with illegitimate source IP addresses, and facilitating global operational communication and coordination. Read the full list of actions.

“MANRS is more an idea than a framework, and it is a tremendous idea,” said Hostmein CTO Alexander Stamatis. “It raises awareness, it raises new checks to be implemented in the industry, and it keeps us more in line with the primary mission: keeping the network clean, keeping it safe.

“[MANRS] is better because it was built by engineers for engineers. We discovered issues no other initiatives could detect.”

“MANRS is the best implementation that we have done to date. We have found it to be more effective than other specialised IT certifications. And it is better because it Continue reading

New Category of CDNs and Cloud Providers Join MANRS to Improve Routing Security

Today, we’re proud to announce the new MANRS Content Delivery Network (CDN) and Cloud Programme. This new program broadens support for the primary objective of MANRS – to implement crucial fixes needed to eliminate the most common threats to the Internet’s routing system.

The founding participants are: Akamai, Amazon Web Services, Azion, Cloudflare, Facebook, Google, Microsoft, and Netflix.

Now, let’s back up and explain how we got here.

What Is MANRS?

Mutually Agreed Norms for Routing Security (MANRS) is a global initiative, supported by the Internet Society, that requires collaboration among participants and shared responsibility for the global Internet routing system. It’s a community of security-minded organizations committed to making routing infrastructure more robust and secure.

Originally designed by and for network operators, the initiative has already been extended once to address the unique needs and concerns of Internet Exchange Points. These two facets of MANRS complement each other – the first secures customer-provider interconnections, while the second creates a safe public peering environment.

How Do CDNs and Cloud Providers Help?

CDNs are a geographically distributed group of servers that work together to provide fast delivery of Internet content across the globe, and today the majority of web traffic Continue reading

You Asked and We Listened: New Features in the MANRS Observatory

Collaboration and shared responsibility are two pillars of the Mutually Agreed Norms for Routing Security (MANRS) initiative, which we support so that there is a baseline of routing security for network operators around the world.

The same values apply to running the MANRS Observatory, an online tool we launched in August that lets users track the state of Internet routing security and network operators their “MANRS-readiness.” Aggregating data from trusted sources, it relies on the community with a shared goal to protect the core of the Internet.

Since we rolled out the tool, many of you have shared that you would like to see updates to make it more informative, intuitive, and easy to use. We take your comments seriously, and we are delighted to introduce some of the new features to you.

We’ve made several improvements to the user interface, including:

Improved Search. The search network now displays the name of a network as you type an ASN. This feature is only available to MANRS participants; public access does not provide data for individual networks.
Report Sharing. Individual network reports that provide detailed information about potential incidents and cases of non-conformance can now be easily shared with Continue reading

MANRS Observatory: Monitoring the State of Internet Routing Security

Routing security is vital to the future and stability of the Internet, but it’s under constant threat. Which is why we’ve launched a free online tool so that network operators can see how they’re doing, and what they can improve, while anyone can see the health of the Internet at a glance. The MANRS Observatory measures networks’ adherence to MANRS – their “MANRS readiness” – a key indicator of the state of routing security and resiliency of the Internet.

Here’s what the MANRS Observatory is in a nutshell:

Performance Barometer: MANRS participants can easily monitor how well they adhere to the requirements of this initiative and make any necessary adjustments to their security controls.
Business Development: Participants can see how they and their peers are performing. They can leverage the MANRS Observatory to determine whether potential partners’ security practices are up to par.
Government: Policymakers can better understand the state of routing security and resilience and help improve it by calling for MANRS best practices.
Social Responsibility: MANRS implementation is simple, voluntary, and non-disruptive. The Observatory can help participants ensure they and their peers are keeping their networks secure, which helps improve routing security of the Internet as a whole.

Internet Resilience Discussions at IETF 104

Let’s look at what’s happening in the Internet Engineering Task Force (IETF) and the upcoming IETF 104 meeting in the area of Internet infrastructure resilience. As usual, my focus here is primarily on the routing and forwarding planes, and specifically routing security and unwanted traffic of Distributed Denial of Service Attacks (DDoS) attacks. There’s interesting and important work underway at the IETF that can help addressing problems in both areas.

This time there are a lot of new ideas, especially of an operational nature, that people bring to the IETF in the form of Internet Drafts that aim to improve the security and resilience of the Internet infrastructure. So I’d like to introduce some of them to you, but keep in mind that an Internet Draft (I-D) does not necessarily indicate IETF endorsement. It also does not constitute a standard and may even not result in any work at the IETF.

So let’s look at what’s happening in BGP land.

Can BGP Communities be harmful?

In the recent paper “BGP Communities: Even more Worms in the Routing Can“, the authors demonstrated that Border Gateway Protocol (BGP) communities can be exploited by remote parties to influence routing in Continue reading

Routing Security – Getting Better, But No Reason to Rest!

Editor’s note: This is an abridged version of a post that was first published on MANRS.org. Read the full version.

In January last year I looked back at 2017 trying to figure out how routing security looked like globally and on a country level. I used BGPStream.com – a great public service providing information about suspicious events in the routing system.

The metrics I used for this analysis were number of incidents and networks involved, either by causing such incidents, or being affected by them.

An ‘incident’ is a suspicious change in the state of the routing system that can be attributed to an outage or a routing attack, like a route leak or hijack (either intentional or due to a configuration mistake). BGPStream is an operational tool that tries to minimize false positives, so the number of incidents may be on the low side.

Of course, there are a few caveats with this analysis – since any route view is incomplete and the intents of the changes are unknown, there are false positives. Some of the incidents went under the radar. Finally, the country attribution is based on geo-mapping and sometimes gets it wrong.

However, even if Continue reading

Rough Guide to IETF 102: Internet Infrastructure Resilience

As usual, in this post I’ll focus on important work the IETF is doing that helps improve the security and resilience of the Internet infrastructure.

At IETF 102 there are a lot of new ideas being brought to the community in the form of Internet Drafts aimed at improving the security and resilience of the Internet infrastructure, and I’d like to introduce some of them to you. But keep in mind – an Internet Draft does not indicate IETF endorsement, is not a standard, and may not result in any further work at the IETF.

So, let us look at what is happening in the domain of BGP, the routing protocol that connects the Internet.

Route leaks

There has been slow progress in the work on mitigating route leaks in the IDR Working Group (WG). One of the reasons for the slowness was that the group was considering two proposals addressing the route leak problem and both are IDR WG documents: “Methods for Detection and Mitigation of BGP Route Leaks”, and “Route Leak Prevention using Roles in Update and Open Messages”. Plus, there is a third submission “Route Leak Detection and Filtering using Roles Continue reading

New RFC 8360 – RPKI Validation Reconsidered – Offers Alternative Validation Procedures to Improve Routing Security

RFC 8360, Resource Public Key Infrastructure (RPKI) Validation Reconsidered, is now published in the RFC libraries.

What is RPKI?

Resource Public Key Infrastructure (RPKI) aims to improve the security of the Internet routing system, specifically the Border Gateway Protocol (BGP), by establishing a hierarchy of trust for BGP routes. Today, most organizations simply trust that routing updates they get are sent by authorized senders. This is how bad actors and misconfigurations can cause massive routing issues. With RPKI, the receiving organization can verify that the sending organization is authorized to send the routing update.

RPKI works by issuing X.509-based resource certificates to holders of IP addresses and AS numbers to prove assignment of these resources. These certificates are issued to Local Internet Registries (LIRs) by one of the five Regional Internet Registries (RIRs) who allocate and assign these resources in their service regions.

What Does This RFC Do?

In the IETF, participants have been discussing issues that may arise when resources move across registries. The problem happens when a subordinate certificate “over-claims” resources compared to its parent. According to the standard validation procedure specified in RFC 6487, the whole branch beneath would be invalidated. The closer to Continue reading

Rough Guide to IETF 101: Internet Infrastructure Resilience

In this post of the Internet Society Rough Guide to IETF 101, I’ll focus on important work the IETF is doing that helps improve security and resilience of the Internet infrastructure.

BGP

What happens if an IXP operator begins maintenance work on the switches without ensuring that BGP sessions between the peers have been shut down? A network disruption and outage. A draft now in the RFC editor queue, “Mitigating Negative Impact of Maintenance through BGP Session Culling”, provides guidance to IXP operators on how to avoid such situations by forcefully tearing down the BGP sessions (session culling) affected by the maintenance before the maintenance activities commence. This approach allows BGP speakers to pre-emptively converge onto alternative paths while the lower layer network’s forwarding plane remains fully operational.

Another draft also in the RFC editor queue, “Graceful BGP session shutdown”, addresses issues related to planned maintenance. The procedures described in this document can be applied to reduce or avoid packet loss for outbound and inbound traffic flows initially forwarded along the peering link to be shut down. These procedures trigger, in both Autonomous Systems (AS), rerouting to alternate paths if they exist within the Continue reading

Routing Security is a Serious Problem – and MANRS Can Help. A Report from APRICOT 2018.

Last week, at APRICOT 2018 in Kathmandu, Nepal, there were a lot of talks and discussions focused on routing security and the Mutually Agreed Norms for Routing Security (MANRS).

First, there was a Routing Security BoF, attended by about 150 people, where we talked about what it takes to implement routing security practices, how CDNs and other players can help, and why it is so difficult to make progress in this area. The BoF included an interactive poll at the end, and it showed some interesting results:

Participants almost unanimously see lack of routing security as a serious problem.
Slow progress in this area is largely seen as due to a lack of incentives
Participants see community initiatives (like MANRS) as the main driving forces for improvement, followed by CDNs and cloud providers. They doubt that governments or end-customers can effectively drive change.

My colleague Aftab Siddiqui is writing a separate blog post just about that BoF, so watch the blog in the next day or two.

Later, in the security track of the main APRICOT programme, Andrei Robachevsky, ISOC’s Technology Programme Manager, presented statistics on routing incidents and suggested a way forward based on the MANRS approach. In his Continue reading

14,000 Incidents: a 2017 Routing Security Year in Review

How was the state of the Internet’s routing system in 2017? Let’s take a look back using data from BGPStream. Some highlights:

13,935 total incidents (either outages or attacks like route leaks and hijacks)
Over 10% of all Autonomous Systems on the Internet were affected
3,106 Autonomous Systems were a victim of at least one routing incident
1,546 networks caused at least one incident

An ‘incident’ is a suspicious change in the state of the routing system that can be attributed to an outage or a routing attack, like a route leak or hijack (either intentional or due to a configuration mistake).[i] Let’s look at just a few examples of incidents picked up by the media.

March 2017. SECW Telecom in Brazil hijacked prefixes of Cloudflare, Google, and BancoBrazil causing some outage for these services in the region.

April 2017. Large chunks of network traffic belonging to MasterCard, Visa, and more than two dozen other financial services companies were briefly routed through a Russian telecom. For several minutes, Rostelecom was originating 50 prefixes for numerous other Autonomous Systems, hijacking their traffic.

August 2017. Google accidentally leaked BGP prefixes it learned from peering relationships, essentially becoming a transit provider instead Continue reading

Another BGP Routing Incident Highlights an Internet Without Checkpoints

Yesterday, there were two BGP routing incidents in which several high-profile sites (Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and Riot Games) were rerouted to a previously unused Russian AS. The incidents only lasted about three minutes each, but demonstrated once again the lack of routing controls like those called for in MANRS that could have prevented this from happening.

As reported in BGPmon’s blog post on 12 December 12,

“…our systems detected a suspicious event where many prefixes for high profile destinations were being announced by an unused Russian Autonomous System.

Starting at 04:43 (UTC) 80 prefixes normally announced by organizations such Google, Apple, Facebook, Microsoft, Twitch, NTT Communications and Riot Games were now detected in the global BGP routing tables with an Origin AS of 39523 (DV-LINK-AS), out of Russia.”

Either a configuration mistake or a malicious attack, it propagated quickly through the Internet without visible obstacles. This was one of almost 5000 route leaks and hijacks in 11 months of 2017. For comparison, network outages during the same period caused almost 8000 incidents (source: https://bgpstream.com/):

In practice, the efficacy of corrective actions strongly depends on the reliability and completeness of information related to Continue reading

Rough Guide to IETF 100: Internet Infrastructure Resilience

As we approach IETF 100 in Singapore next week, this post in the Rough Guide to IETF 100 has much progress to report in the world of Internet Infrastructure Resilience. After several years of hard work, the last major deliverable of the Secure Inter-Domain Routing (SIDR) WG is done – RFC 8205, the BGPSec Protocol Specification, was published in September 2017 as standard. BGPsec is an extension to the Border Gateway Protocol (BGP) that provides security for the path of autonomous systems (ASes) through which a BGP update message propagates.

There are seven RFCs in the suite of BGPsec specifications:

RFC 8205 (was draft-ietf-sidr-bgpsec-protocol) – BGPsec Protocol Specification
RFC 8206 (was draft-ietf-sidr-as-migration) – BGPsec Considerations for Autonomous System (AS) Migration
RFC 8207 (was draft-ietf-sidr-bgpsec-ops) – BGPsec Operational Considerations
RFC 8208 (was draft-ietf-sidr-bgpsec-algs) – BGPsec Algorithms, Key Formats, and Signature Formats
RFC 8209 (was draft-ietf-sidr-bgpsec-pki-profiles) – A Profile for BGPsec Router Certificates, Certificate Revocation Lists, and Certification Requests
RFC 8210 (was draft-ietf-sidr-rpki-rtr-rfc6810-bis) – The Resource Public Key Infrastructure (RPKI) to Router Protocol, Version 1
RFC 8211 (was draft-ietf-sidr-adverse-actions) – Adverse Actions by a Certification Authority (CA) or Repository Manager in the Resource Public Key Infrastructure (RPKI)

You can read more Continue reading

Can IoT platforms from Apple, Google and Samsung make home automation systems more secure?

In August 2017, a new botnet called WireX appeared and began causing damage by launching significant DDoS attacks. The botnet counted tens of thousands of nodes, most of which appeared to be hacked Android mobile devices.

There are a few important aspects of this story.

First, tracking the botnet down and mitigating its activities was part of a wide collaborative effort by several tech companies. Researchers from Akamai, Cloudflare, Flashpoint, Google, Oracle Dyn, RiskIQ, Team Cymru, and other organizations cooperated to combat this botnet. This is a great example of Collaborative Security in practice.

Second, while researchers shared the data, analysed the signatures, and were able to track a set of malware apps, Google played an important role in cleaning them up from the Play Store and infected devices.

Its Verify Apps is a cloud-based service that proactively checks every application prior to install to determine if the application is potentially harmful, and subsequently rechecks devices regularly to help ensure they’re safe. Verify Apps checks more than 6 billion instances of installed applications and scans around 400 million devices per day.

In the case of WireX, the apps had previously passed the checks. But thanks to the researcher’s findings, Google Continue reading

New BlueBorne Vulnerability to Bluetooth Devices – What happened and what to do about it

Billions of Bluetooth-enabled devices may be exposed to a new remote attack called “BlueBorne”, even without user interaction or pairing. Affected systems include Windows, iOS (older than iOS 10), the Linux kernel, and Android. What should you do about it?

Bluetooth is ubiquitous, commonly connecting accessories like headsets and keyboards, but is also used throughout the brave new Internet of Things (IoT) world. An attacker exploiting these BlueBorne vulnerabilities can mount a man-in-the-middle attack, or even take control of a device without the user even noticing it.

The vulnerabilities were discovered by a security company called Armis earlier this year. Researchers reached out to the companies responsible for vulnerable implementations that lead to the coordinated disclosure (and patches) on September 12. (You can read more about our views on responsible disclosure and collaborative security in Olaf Kolkman’s blog post here.)

This case once again highlights how crucial it is that software update mechanisms are available to fix vulnerabilities, update configuration settings, and add new functionality to devices. There are challenges, both technological and economic, in having update capabilities ubiquitously deployed, as discussed in the recently published Report from the Internet of Things Software Update (IoTSU) Workshop 2016.

Vulnerabilities Continue reading

New Study: Understanding MANRS’ Potential for Enterprises and Service Providers

Mutually Agreed Norms for Routing Security, or MANRS, was founded with the ambitious goal of improving the security and reliability of the global Internet routing system, based on collaboration among participants and shared responsibility for Internet infrastructure. These are undoubtedly essential pillars supporting the Internet’s tremendous growth and success, but we must better articulate the incentives of contributing to global security and resilience to grow MANRS participation and reach our goals.

To do so, we engaged 451 Research to understand the attitudes and perceptions of Internet service providers and the broader enterprise community around MANRS and how it relates to their organizations. The results of the study are documented in the report: https://www.routingmanifesto.org/resources/research/.

The study results demonstrate considerable unrealized potential for MANRS, showing that enterprises are interested in security and their interest should be a strong incentive for more service providers to participate. Market education could be particularly effective in overcoming the operational inertia that many providers face.

The key points from the study are:

While MANRS itself is not well known by enterprises, its attributes are highly valued.
Enterprises have high expectations for MANRS efforts.
Enterprise perceptions of MANRS can translate into increased revenue for service Continue reading

New Study: Understanding MANRS’ Potential for Enterprises and Service Providers

[Note: This post was originally published on the MANRS Blog at https://www.routingmanifesto.org/2017/09/new-study-understanding-manrs-potential-for-enterprises-and-service-providers/.]

The key points from the study are:

While MANRS itself is not well known by enterprises, its attributes are highly valued.
Enterprises have high Continue reading

Google leaked prefixes – and knocked Japan off the Internet

Last Friday, 25 August, a routing incident caused large-scale internet disruption. It hit Japanese users the hardest, slowing or blocking access to websites and online services for dozens of Japanese companies.

What happened is that Google accidentally leaked BGP prefixes it learned from peering relationships, essentially becoming a transit provider instead of simply exchanging traffic between two networks and their customers. This also exposed some internal traffic engineering that caused many of these prefixes to get de-aggregated and therefore raised their probability of getting accepted elsewhere.

The incident technically lasted less than ten minutes, but spread quickly around the Internet and caused some damage. Connectivity was restored, but persistently slow connection speeds affected industries like finance, transportation, and online gaming for several hours. Google apologized for the trouble, saying it was caused by an errant network setting that was corrected within eight minutes of its discovery.

This incident showed, again, how fragile the global routing system still is against configuration mistakes, to say nothing about malicious attacks.

What it also showed is a lack of defense – the incident propagated seemingly without any attempt from other networks to stop it.

The Internet Society works to address security in many ways, Continue reading