WISP Design – Building Highly Available VPLS for Public Subnets

What is VPLS?

Virtual Private LAN Service or VPLS is a Layer 2 overlay or tunnel that allows for the encapsulation of ethernet frames (with or without VLAN tags) over an MPLS network.

https://tools.ietf.org/html/rfc4762

VPLS is often found in Telco networks that rely on PPPoE to create centralized BRAS deployments by bringing all of the end users to a common point via L2.

MikroTik VPLS example (https://wiki.mikrotik.com/wiki/Transparently_Bridge_two_Networks_using_MPLS)

Background

The idea for this post came out of a working session (at the bar of course) at WISPAmerica 2018 in Birmingham, Alabama.

There was a discussion about how to create redundancy for VPLS tunnels on multiple routers. I started working on this in EVE-NG as we were talking about it.

The goal is creating highly available endpoints for VPLS when using them to deploy a public subnet that can be delivered to any tower in the WISP. The same idea works for wireline networks as well.

Use Case

As IPv4 becomes harder to get, ISPs like WISPs, without large blocks of public space find it difficult to deploy them in smaller subnets. The idea behind breaking up a /23 or /24 for example, is that every Continue reading

WISP Design – Building Highly Available VPLS for Public Subnets

What is VPLS?

Virtual Private LAN Service or VPLS is a Layer 2 overlay or tunnel that allows for the encapsulation of ethernet frames (with or without VLAN tags) over an MPLS network.

https://tools.ietf.org/html/rfc4762

VPLS is often found in Telco networks that rely on PPPoE to create centralized BRAS deployments by bringing all of the end users to a common point via L2.

MikroTik VPLS example (https://wiki.mikrotik.com/wiki/Transparently_Bridge_two_Networks_using_MPLS)

Background

The idea for this post came out of a working session (at the bar of course) at WISPAmerica 2018 in Birmingham, Alabama.

There was a discussion about how to create redundancy for VPLS tunnels on multiple routers. I started working on this in EVE-NG as we were talking about it.

The goal is creating highly available endpoints for VPLS when using them to deploy a public subnet that can be delivered to any tower in the WISP. The same idea works for wireline networks as well.

Use Case

As IPv4 becomes harder to get, ISPs like WISPs, without large blocks of public space find it difficult to deploy them in smaller subnets. The idea behind breaking up a /23 or /24 for example, is that every Continue reading

Cisco to MikroTik – command translation – OSPF

In the world of network engineering, learning a new syntax for a NOS can be overwhelming if you need a specific set of config in a short timeframe. The command structure for RouterOS can be a bit challenging if you are used to Cisco CLI commands.

If you’ve been in networking for a while, there’s a good chance you started with Cisco gear and so it is helpful to draw comparisons between the commands, especially if you are trying to build a network with a MikroTik and Cisco router.

This is the second post in a series that creates a Rosetta stone essentially between IOS and RouterOS. We plan to tackle  other command comparisons like MPLS, VLANs and basic operations to make it easier for network engineers trained in Cisco IOS to successfully implement Mikrotik / RouterOS devices.

Click here for the first article in this series – “Cisco to MikroTik BGP command translation”

While many commands have almost the exact same information, others are as close as possible. Since there isn’t always an exact match, sometimes you may have to run two or three commands to get the information needed.

Using  EVE-NG for testing

In the last article, Continue reading

Cisco to MikroTik – command translation – OSPF

In the world of network engineering, learning a new syntax for a NOS can be overwhelming if you need a specific set of config in a short timeframe. The command structure for RouterOS can be a bit challenging if you are used to Cisco CLI commands.

If you’ve been in networking for a while, there’s a good chance you started with Cisco gear and so it is helpful to draw comparisons between the commands, especially if you are trying to build a network with a MikroTik and Cisco router.

This is the second post in a series that creates a Rosetta stone essentially between IOS and RouterOS. We plan to tackle  other command comparisons like MPLS, VLANs and basic operations to make it easier for network engineers trained in Cisco IOS to successfully implement Mikrotik / RouterOS devices.

Click here for the first article in this series – “Cisco to MikroTik BGP command translation”

While many commands have almost the exact same information, others are as close as possible. Since there isn’t always an exact match, sometimes you may have to run two or three commands to get the information needed.

Using  EVE-NG for testing

In the last article, Continue reading

Preview: Networking Field Day Exclusive with Aruba (HPE) – The 8400 core switch

Back to Silicon Valley!

As a network type, it’s hard not to be excited when heading to a Networking Field Day event. I joined then NFD club by attending NFD14 and have been hooked ever since.

Not only is it an honor and a privilege to be invited to an NFD event, the personal relationships that are forged in the larger TFD community are some of the most valuable I’ve ever had in my career.

This go around we’ll be visiting Aruba (A Hewlett Packard Enterprise Company) in Santa Clara to deep dive on the newest addition to the Aruba product line – the 8400 core switch.

A new face in campus town – the Aruba 8400

It’s been a while since anything exciting happened in the world of campus networking. It’s a steady segment for most vendors but nothing disruptive has really happened in the last few years.

And that’s not incredibly surprising. For better or worse, as long as campus networks aren’t broken in most enterprises, they are often neglected in favor of the data center and cloudy pursuits.

Aruba is touting the 8400 to increase automation and visibility in the campus core – both are areas Continue reading

Preview: Networking Field Day Exclusive with Aruba (HPE) – The 8400 core switch

Back to Silicon Valley!

As a network type, it’s hard not to be excited when heading to a Networking Field Day event. I joined then NFD club by attending NFD14 and have been hooked ever since.

Not only is it an honor and a privilege to be invited to an NFD event, the personal relationships that are forged in the larger TFD community are some of the most valuable I’ve ever had in my career.

This go around we’ll be visiting Aruba (A Hewlett Packard Enterprise Company) in Santa Clara to deep dive on the newest addition to the Aruba product line – the 8400 core switch.

A new face in campus town – the Aruba 8400

It’s been a while since anything exciting happened in the world of campus networking. It’s a steady segment for most vendors but nothing disruptive has really happened in the last few years.

And that’s not incredibly surprising. For better or worse, as long as campus networks aren’t broken in most enterprises, they are often neglected in favor of the data center and cloudy pursuits.

Aruba is touting the 8400 to increase automation and visibility in the campus core – both are areas Continue reading

WISP/FISP Design – Building your future MPLS network with whitebox switching.

The role of whitebox in a WISP/FISP MPLS core

Whitebox, if you aren’t familiar with it, is the idea of separating the network operating system and switching hardware into commodity elements that can be purchased separately. There was a good overview on whitebox in this StubArea51.net article a while back if you’re looking for some background.

Lately, in my work for IP ArchiTechs, I’ve had a number of clients interested in deploying IP Infusion with either Dell, Agema or Edge Core switches to build an MPLS core architecture in lieu of an L2 ring deployment via ERPs. Add to that a production deployment of Cumulus Linux and Edge Core that I’ve been working on building out and it’s been a great year for whitebox.

There are a number of articles written that extoll the virtues of whitebox for web scale companies, large service providers and big enterprises. However, not much has been written on how whitebox can help smaller Tier 2 and 3 ISPs – especially Wireless ISPs (WISPs) and Fiber ISPs (FISPs).

And the line between those types of ISPs gets more blurry by the day as WISPs are heavily getting into fiber and FISPs are Continue reading

WISP/FISP Design – Building your future MPLS network with whitebox switching.

The role of whitebox in a WISP/FISP MPLS core

Whitebox, if you aren’t familiar with it, is the idea of separating the network operating system and switching hardware into commodity elements that can be purchased separately. There was a good overview on whitebox in this StubArea51.net article a while back if you’re looking for some background.

Lately, in my work for IP ArchiTechs, I’ve had a number of clients interested in deploying IP Infusion with either Dell, Agema or Edge Core switches to build an MPLS core architecture in lieu of an L2 ring deployment via ERPs. Add to that a production deployment of Cumulus Linux and Edge Core that I’ve been working on building out and it’s been a great year for whitebox.

There are a number of articles written that extoll the virtues of whitebox for web scale companies, large service providers and big enterprises. However, not much has been written on how whitebox can help smaller Tier 2 and 3 ISPs – especially Wireless ISPs (WISPs) and Fiber ISPs (FISPs).

And the line between those types of ISPs gets more blurry by the day as WISPs are heavily getting into fiber and FISPs are Continue reading

WISP Design – Using eBGP and OSPF transit fabric for traffic engineering

One of the latest designs we have been working on at IP ArchiTechs is using eBGP and an OSPF transit fabric to provide traffic engineering and load balancing. If you missed this presentation at the 2017 MikroTik User Meeting in Denver, CO, here are the slides:

WISP-Design-Using-eBGP-and-OSPF-TF-traffic-engineering-MUM-2017_KevinMyers-4-by-3

WISP Design – Using eBGP and OSPF transit fabric for traffic engineering

One of the latest designs we have been working on at IP ArchiTechs is using eBGP and an OSPF transit fabric to provide traffic engineering and load balancing. If you missed this presentation at the 2017 MikroTik User Meeting in Denver, CO, here are the slides:

WISP-Design-Using-eBGP-and-OSPF-TF-traffic-engineering-MUM-2017_KevinMyers-4-by-3

Whitebox networking – coming soon to an edge near you?

What is whitebox networking and why is it important?

A brief history of the origins of whitebox

ONIE – The beginning of independent network Continue reading

Whitebox networking – coming soon to an edge near you?

What is whitebox networking and why is it important?

A brief history of the origins of whitebox

ONIE – The beginning of independent network Continue reading

A network geek pilgrimage – Networking Field Day 14

What is Networking Field Day?

Networking Field Day 14 or #NFD14 is almost upon us! I am heading to sunny San Jose, California to drink from the fire hose of data – the heavens will part and rain down golden non-fragmented packets of information and insight.

If you’re not familiar with Networking Field Day, which is part of Tech Field Day hosted by Gestalt IT, you can go here to get a full overview.

The Delegates

Networking Field Day is rare opportunity for individuals (delegates) that are engaged in the practice of network engineering/architecture to come together and interact not only with the vendors who are presenting but also fellow delegates.

While some of the delegates have attended previous Tech Field Day events, others, like myself are first timers and will be taking in the vast array of technical content as fast as our buffers permit.

The delegates comprise a group of like-minded and yet diverse networkers that are heavily invested in the community of network engineers and IT.

It’s truly a privilege to be be invited to NFD14 and I count myself fortunate to be in the company of some incredibly sharp practitioners of IP networking – a Continue reading

A network geek pilgrimage – Networking Field Day 14

What is Networking Field Day?

Networking Field Day 14 or #NFD14 is almost upon us! I am heading to sunny San Jose, California to drink from the fire hose of data – the heavens will part and rain down golden non-fragmented packets of information and insight.

If you’re not familiar with Networking Field Day, which is part of Tech Field Day hosted by Gestalt IT, you can go here to get a full overview.

The Delegates

Networking Field Day is rare opportunity for individuals (delegates) that are engaged in the practice of network engineering/architecture to come together and interact not only with the vendors who are presenting but also fellow delegates.

While some of the delegates have attended previous Tech Field Day events, others, like myself are first timers and will be taking in the vast array of technical content as fast as our buffers permit.

The delegates comprise a group of like-minded and yet diverse networkers that are heavily invested in the community of network engineers and IT.

It’s truly a privilege to be be invited to NFD14 and I count myself fortunate to be in the company of some incredibly sharp practitioners of IP networking – a Continue reading

MikroTik RouterOS new feature – Loop Protect

‘Loop Protect’ – New feature in 6.37rc24

Long-time MikroTik users have been after better loop prevention mechanisms for quite a while now. Rapid STP within bridges was the only feature available up until Fall of 2016 and now MikroTik has released Rapid Spanning Tree in hardware for switched ports as well as a new Loop Protect feature that seems to serve the same function as Cisco’s Loop Guard but not utilize spanning tree to detect the loop. MikroTik’s version compares the source MAC of the loop protect frame with the MAC of the interface it is received on and if they match, it will disable the port until the timer expires and check again for the existence of a loop.

This feature was introduced in 6.37rc24 on August 31st, 2016.

http://wiki.mikrotik.com/wiki/Manual:Loop_Protect

Use cases for ‘Loop Protect’

Loop protect seems to be designed more as an edge port protocol since it physically disables the port upon detection of a loop, whereas STP will leave the port physically active but logically block traffic on that path.  Some potential use cases for enabling this feature could include:

• Edge port on a MikroTik device facing the end subscriber equipment Continue reading

MikroTik RouterOS new feature – Loop Protect

‘Loop Protect’ – New feature in 6.37rc24

Long-time MikroTik users have been after better loop prevention mechanisms for quite a while now. Rapid STP within bridges was the only feature available up until Fall of 2016 and now MikroTik has released Rapid Spanning Tree in hardware for switched ports as well as a new Loop Protect feature that seems to serve the same function as Cisco’s Loop Guard but not utilize spanning tree to detect the loop. MikroTik’s version compares the source MAC of the loop protect frame with the MAC of the interface it is received on and if they match, it will disable the port until the timer expires and check again for the existence of a loop.

This feature was introduced in 6.37rc24 on August 31st, 2016.

http://wiki.mikrotik.com/wiki/Manual:Loop_Protect

Use cases for ‘Loop Protect’

Loop protect seems to be designed more as an edge port protocol since it physically disables the port upon detection of a loop, whereas STP will leave the port physically active but logically block traffic on that path.  Some potential use cases for enabling this feature could include:

• Edge port on a MikroTik device facing the end subscriber equipment Continue reading

WISP Design: Using OSPF to build a transit fabric over unequal links

Defining the problem – unused capacity

One of the single greatest challenges if you have ever owned, operated or designed a WISP (Wireless Internet Service Provider) is using all of the available bandwidth across multiple PtP links in the network. It is very common for two towers to have multiple RF PtP (Point-to-Point) links between them and run at different speeds. It is not unusual to have a primary link that runs at near-gigabit speeds and a backup link that may range anywhere from 50 Mbps to a few hundred Mbps.

This provides a pretty clean HA routing architecture, but it leaves capacity in the network unused until there is a failure. One of the headaches WISP designers always face is how to manage and engineer traffic for sub-rate ethernet links – essentially links that can’t deliver as much throughput as the physical link to the router or switch. In the fiber world, this is pretty straightforward as two links between any two points can be the exact same speed and either be channeled together with LACP or rely on ECMP with OSPF or BGP.

However, in the WISP world, this becomes problematic, as the links are unequal and Continue reading

WISP Design: Using OSPF to build a transit fabric over unequal links

Defining the problem – unused capacity

One of the single greatest challenges if you have ever owned, operated or designed a WISP (Wireless Internet Service Provider) is using all of the available bandwidth across multiple PtP links in the network. It is very common for two towers to have multiple RF PtP (Point-to-Point) links between them and run at different speeds. It is not unusual to have a primary link that runs at near-gigabit speeds and a backup link that may range anywhere from 50 Mbps to a few hundred Mbps.

This provides a pretty clean HA routing architecture, but it leaves capacity in the network unused until there is a failure. One of the headaches WISP designers always face is how to manage and engineer traffic for sub-rate ethernet links – essentially links that can’t deliver as much throughput as the physical link to the router or switch. In the fiber world, this is pretty straightforward as two links between any two points can be the exact same speed and either be channeled together with LACP or rely on ECMP with OSPF or BGP.

However, in the WISP world, this becomes problematic, as the links are unequal and Continue reading

The importance of situational awareness for network engineers

In another life, not too long ago, I spent a number of years in civilian and military law enforcement. When going through just about any kind of tactical training, one of the recurring themes they hammer into you is “situational awareness or SA.”

Wikipedia defines SA as:

Situational awareness or situation awareness (SA) is the perception of environmental elements with respect to time or space, the comprehension of their meaning, and the projection of their status after some variable has changed, such as time, or some other variable, such as a predetermined event. It is also a field of study concerned with understanding of the environment critical to decision-makers in complex, dynamic areas from aviation, air traffic control, ship navigation, power plant operations, military command and control, and emergency services such as fire fighting and policing; to more ordinary but nevertheless complex tasks such as driving an automobile or riding a bicycle.

Defining the need for SA in network engineering

It’s interesting to notice that critical infrastructure such as power plants and air traffic control are listed as disciplines that train in SA, however, I’ve never seen it taught in Continue reading

The importance of situational awareness for network engineers

In another life, not too long ago, I spent a number of years in civilian and military law enforcement. When going through just about any kind of tactical training, one of the recurring themes they hammer into you is “situational awareness or SA.”

Wikipedia defines SA as:

Situational awareness or situation awareness (SA) is the perception of environmental elements with respect to time or space, the comprehension of their meaning, and the projection of their status after some variable has changed, such as time, or some other variable, such as a predetermined event. It is also a field of study concerned with understanding of the environment critical to decision-makers in complex, dynamic areas from aviation, air traffic control, ship navigation, power plant operations, military command and control, and emergency services such as fire fighting and policing; to more ordinary but nevertheless complex tasks such as driving an automobile or riding a bicycle.

Defining the need for SA in network engineering

It’s interesting to notice that critical infrastructure such as power plants and air traffic control are listed as disciplines that train in SA, however, I’ve never seen it taught in Continue reading