Archive

Category Archives for "StubArea51.com"

BGP communities part 3: Customer BGP Traffic Engineering communities

If you’ve ever been asked to prioritize one internet connection over another for any variety of reasons, cost, latency, SLA, etc… this is for you.

Often I hear the same tactics to solve this problem:

  • AS-PATH prepending
  • conditional advertisements
  • scripting
  • some other manual process

However, most carriers offer customer BGP TE communities that you can use to influence traffic within their AS, with one notable exception Hurricane Electric. If you’re not sure what a BGP community take a quick look at this post on them first.

Lets explore how to utilize these, where to find them, and how they might give more deterministic path selection than the options laid out above.

BGP Topology

Default behavior with no modification

First to get familiar with the topology and show reachability we’ll leave all settings as “defaults” with no modifications.

ISP-1-RTR-1#traceroute 203.0.113.1 source 192.0.2.1
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
  1 100.123.0.1 1 msec 1 msec 1 msec
  2 100.124.0.2 1 msec 0 msec 0 msec
  3 100.126.0.10 2 msec *  1  Continue reading

MikroTik – ROSv7 – VPLS over IPv6 MPLS with LDPv6


The great migration

IPv6 adoption has really picked up in the last 12 months and MikroTik RouterOSv7 development is no exception. Dual stack networks are still the most common and easiest to initially deploy for carriers.

However, single stack networks with IPv4 as a service overlay are definitely on the horizon for MikroTik users now that MPLS can operate purely on IPv6.

Single stack networks are easier and cheaper to operate in the long run and are a natural evolution of dual stack networks as we begin to turn IPv4 off for underlay infrastructure.

LDPv6

There are a few different ways to distribute labels in IPv6 MPLS. SR-MPLS (less common and usually with IS-IS) and SRv6 are the other options besides LDPv6.

While I generally am in favor of SR-MPLS/SRv6 long term due to the protocol simplification and traffic management capabilities, having an IPv6 MPLS stack is a great starting point for MikroTik.

LDPv6 is defined by RFC 7552 and is fairly recent as it finalized in 2015. It generally operates in much the same way as LDPv4.


Use Cases

The most common use case among MikroTik users is more efficient subnetting of IPv4 and directly replaces LDPv4 for this Continue reading

WEBINAR: ISP Design – Separation of Network Functions


Recently, we recorded a webinar to explain a design concept frequently used by iparchitechs.com to build and migrate WISP, FISP and Telco networks – separation of network functions. It centers around simplification of roles within an ISP network. It also explores the use of lower-cost commodity network equipment to maximize the service area for a given ISP footprint while meeting key requirements like scale, redundancy and capacity.


Topics that were covered include:

  • What are network functions?
  • Design examples for WISP/FISP and Telco
  • Equipment and budget considerations



Here is an example of solving design/operational issues with network function separation:




Link to the webinar and slides:

Webinar: Webinar Recording
Slides: Slide Deck


https://iparchitechs.com/contact

MikroTik – RouterOS v7 – BGP performance testing for full tables

MikroTik has come a long way since the first release of RouterOS v7 beta.

One of the long-awaited features is improved BGP performance and the ability to leverage multiple CPU cores.

Testing BGP performance is a long process of lab and prod evaluation, so we decided to run some quick and basic tests to get a baseline.

When the CCR2216-1G-12XS-2XQ was released and MikroTik entered the world of 100G, we ordered some right away to test and just got them in the lab a few days ago – the results are below.

Hope this is helpful and look for more BGP perf tests in the coming months!

The BGP testing lab

TLDR; 2.1 million routes learned and forwarding in 46 seconds and withdrawn in 44 seconds. This was tested under a 25 Gbps load on both routers with a cpu load of 12%.

Lab overview: The lab consists of (2) CCR2216 routers running ROSv7.2 stable connected to a ProxMox hypervisor that runs (4) Linux route generators and MikroTik CHRs (also on 7.2) acting as border routers. The specific connectivity is in the overview drawing below.

IPv6: We are currently developing a route generator that will inject IPv4 Continue reading

Networking Field Day: Service Provider – Arista VPLS

During the first networking field day: service provider one of the big topics was EVPN versus VPLS. Arista has put a lot of work into their EVPN deployment and this has give then a ton of success in the data center. However, a large portion of the provider space, especially last mile providers, rely on VPLS heavily. This naturally led to discussion about Arista VPLS support.

I’m pleased to see that there is now basic support in EOS as of EOS 4.27.2F and more on the roadmap. Hopefully, we’ll see the off ramp, RFC8560, from VPLS to EVPN which was a hot button topic throughout the week.

In the release notes for EOS 4.27.2F it calls our basic VPLS support. So I took a look. Reviewing the new 4.27.2F manual I found support for LDP PWs on RFC4447 which is virtual private wire support. This also appeared to be in EOS 4.26 but not earlier. I might have missed a different call out in the doc, if I did someone please point me in the right direction so I can update this post!

In the meantime lets review how this works:

mpls  Continue reading

App delivery for an improved pizza experience

It’s been a while since we started work on one of our newest projects.  We have been trying to solve a problem in app location.  It all came from the notion that Little Caesars know where my pizza is, so why can’t the network resolve where the app is?    We also thought it would be novel use of Anycast because the app can be anywhere. 

So, what problems specifically have we solved using this design?  Intent based gateways are a signaling mechanism allows the apps to be delivered along with the pizza.  As we can see app Buffalo Wings can reach both the intent based gateway and Fried Pickles using TI-LFA, which strips the fat bits before they reach the gateway.   Our unique caching solution using Tupperware, which are stacked in K8s, allows for the apps to be delivered in a bursty nexthop specific competitive manner.  This has proven to keep the apps warm within the physical layer.

In our example, the Delivery Center Interconnect,  we are doing an east to west Multi Pizza Layered Service that can drop the apps with full BTU into any of the regions.  The apps are Continue reading

WEBINAR: Deploying IPv6 for WISPs and FISPs.

A few weeks ago, we recorded a webinar on deploying IPv6 for WISPs and FISPs. As IPv6 adoption continues to climb, developing an IPv6 strategy for design, deployment and system integration is an important step before subscribers begin asking for IPv6.

Some of the topics that were covered include:

  • IPv6 basics – addressing, subnetting, types
  • IPv6 design and deployment
  • IPv6 systems and operations

Here is an example of getting started with IPv6 deployment at the border of the ASN


Link to the webinar and slide deck

Webinar: click here

Slides: click here




BGP communities for traffic steering – part 2: State Management across Data Centers

This post has been a while in the making and follows up on an article about BGP communities that can be found here. Then we followed it up with some more discussion about FW design and place, or lack there of, on this podcast which inspired me to finish up “part 2”.

Anyone who has ever had to run active/active data centers and has come across this problem of how do I manage state?

You can ignore it and prepare yourself for a late night at the worst time.

Take everyone’s word that systems will never have to talk to the a system in a different security zone in the remote DC

Utilize communities and BGP policy to manage state; which we’ll focus on here

One of the biggest reasons we see for stretching a virtual routing and forwarding (vrf) is to move DC to DC flows of the same security zone below FWs. This reduces the load on the firewall and makes for easier rule management. However, it does introduce a state problem.

We’ll be using the smallest EVPN-multisite deployment you’ve ever seen with Nexus 9000v and Fortinet FWs.

Inter vrf intra data center

The first flow we’ll look Continue reading

Interop IP infusion and MikroTik: part 2 – VPLS

VPLS is a pretty common technology in ISPs to either sell layer 2 services or backhaul traffic to a centralized aggregation point to conserve IPv4 space; check out more on that here.

How can I take advantage of segment routing but still deliver the same services? We already looked at how to setup the label switched paths utilizing IP Infusion as a segment routing mapping server in this interop post. Now let’s see how we can deliver a VPLS service over this with mikrotik as a provider edge router.

Delivering a service with a L2VPN

After setting up the IGP and label distribution between the PEs we will start building the L2VPN.

Why a BGP signaled VPLS session instead of LDP signaled VPLS?

In the segment-routing domain there is no LDP running. However, some vendors support static pseudowires or other methods to bring up a targeted LDP session for VPLS. I did some basic testing here couldn’t easily identify the right combination of knobs to make this work. Don’t worry; I’ll come back to it.

BGP signaled VPLS is a standards based technology that both vendors support.

First thing we need to do after having loopback reachability is to build Continue reading

Interop LDP and Segment Routing with IP infusion and MikroTik

Introduction

During networking field day service provider 1 there was a ton of talk about segment routing (SR) and ethernet virtual private networks (EVPN). One of the biggest questions was “how do we get there?” and while we won’t examine EVPN in this post (it’s coming in a future post don’t worry) we will look at how you can take advantage of SR while still having large portions of LDP in your network.

The team here at IP architechs works on a lot of MikroTik and whitebox gear so we’ll focus on a deployment using MikroTik and IP infusion.

MPLS and IGP setup

The first thing to accomplish is end to end reachability between the provider edge (PE) routers. MikroTik doesn’t support IS-IS so we will have to perform redistribution between the IS-IS segment and the OSPF segment as seen above.

MPLS only requires the /32s of the loopbacks for functionality so redistribution is limited to the /32 loopbacks of the PE routers.

ip prefix-list LDP-PE-LOOPBACKS
 seq 10 permit 100.127.2.0/24 eq 32
!
ip prefix-list SR-PE-LOOPBACKS
 seq 10 permit 100.127.0.0/24 eq 32
!
route-map REDIS-OSPF-TO-ISIS permit 10
 match ip address prefix-list LDP-PE-LOOPBACKS
!
route-map REDIS-ISIS-TO-OSPF permit  Continue reading

Network Modeling: Automating Mikrotik RouterOS CHR Containerlab images

Introduction

In a previous post, we talked about using Containerlab and ZeroTier to provide remote access to a lab instance. One of the things that was glossed over was creating the Containerlab images. As part of building the images, Containerlab ready Mikrotik RouterOS images are available on Docker Hub.

Containerlab images

Containerlab currently supports 2 different kinds of images: pre-built containers and VMs packaged as containers.

Presently, only a handful of NOS vendors provide pre-built containers (Nokia’s SR Linux, Juniper cRPD, Arista cEOS, Cumulus VX, and SONIC VS).

The other option utilizes vrnetlab and is essentially a QEMU VM packaged in a Docker container. The more traditional network operating systems are currently supported via this method. This includes operating systems like Mikrotik RouterOS, Juniper vMX, Nokia SROS, and many more.

Building Mikrotik RouterOS containers

Containerlab provides a nice and easy way to build Containerlab ready Docker images. The process is fairly simple.

Setting it up

Containerlab uses a custom fork of vrnetlab that sets things up so that the images built will work with Containerlab. That repository can be found on Github. The first step is to clone that repository.

The next step is to download the Continue reading

MikroTik CHR – Breaking the 100G barrier

Introduction

The world is strange today. Despite the Covid-19 crisis all over the world, most ISPs are fighting a battle to deliver more bandwidth on a daily basis.

  • Work from home
  • Online schools
  • increasing content consumption

All pushed ISPs to their bandwidth limits, leaving ISP’s no option but to look for upgrades, for everlasting bandwidth demands. There, they are having another set of problems, facing them in this completely new and strange world. Chip shortage, logistic and labor health issues caused higher prices and no stock availability. Here in IP ArchiTechs, we are spending lot of our time finding a good solution for our customers and to help them overcome these hard times. Whether that is our regular Team meeting or just a chat with our colleagues in almost any occasion someone mentions something about new solution to improve capacity and performance for our customers.

Starting with a thought, what is available as a platform today, and of course it’s ready to be shipped immediately after you checkout and pay one thing obviously was just in front of me. X86 server, dozens of them. They are left from the time when we were buying new hardware just because new generation Continue reading

Network Modeling: Segmented Lab access with Containerlab and ZeroTier

Introduction

When building out network labs, often multiple people will need access to the lab. The main way right now is to use something like EVE-NG or GNS3 to provide access.

There are 2 downsides to this method. The first is that your server is exposed to the internet and if your usernames/passwords aren’t strong enough, your server can become compromised. The second is that sometimes you may not want everyone to be able to add or edit to the lab topology.

The solution to this is using Containerlab and ZeroTier. This setup is great for things like testing new hires, training classes, or for providing lab access to others on a limited basis.

What is Containerlab?

Containerlab is a container orchestrating tool for managing container-based networking labs. It doesn’t just support Container based Network Operating Systems though. Through vrnetlab, there is support for a wide variety of commonly used NOSes: Mikrotik RouterOS, Nokia SROS, Juniper vMX and vQFX, and many more.

Configs are text based making it easy to add/update links between nodes. The lab does need to be destroyed and redeployed when adding/removing links. With some tools to generate configs, it’s easy to spin up a 500+ node Continue reading

Migrating from fabricpath to EVPN/VxLAN

Introduction

Do you have a 3 tier, switched, or vendor proprietary data center design?

Does it rely on spanning tree or proprietary solutions to eliminate spanning tree?

Not sure how to migrate to a new architecture without serious downtime?

If you answered yes to any of these questions then this post is for you. We’ll be looking at deploying an EVPN/VxLAN Data Center fabric and migrating a from a cisco fabricpath environment to the new design.

Although we will be focusing on a fabricpath migration many, if not all, of the principles apply to migrating a 3 tier architecture.

1. Building the new Data Center Fabric
2. Connecting the current fabricpath and new fabric
3. Migrating switched virtual interfaces
4. Migrating various types of physical devices

Building the new Data Center Fabric

The easiest part of designing and building the new fabric is the physical topology. This should be a symmetric topology to easily take advantage of equal cost multipath and add additional switches with ease. This is also known as a spine/leaf or clos topology. The basic idea is leafs connect to spines and spines connect to super spines. A leaf/spine should not connect to another switch of the Continue reading

ISP Design Guide: Separation of network functions – introduction and overview

PDF link is here


A reference guide for new & existing ISPs that need to understand network functions and separation.

“How do I add redundancy?”
“How do I scale?”
“How do I reduce downtime and operational costs?”

These are questions that I get asked practically every day as a consulting network architect that designs and builds ISPs.

In most cases the answer is the same whether the ISP uses fixed wireless broadband, copper or fiber to deliver the last mile – separation of network functions.

This illustrated guide is intended to define the topic and create visual context for each function using a network drawing. It’s the first in a new series on this subject.

A new series of content

This topic is deep and there is a lot to unpack so this will be the first segment in a series of blog posts and videos covering function separation.

Large ISPs typically already embrace the philosophy of separating network functions, so the focus of this series will be to help new or growing regional ISPs understand the design intent and the challenges/costs of running networks that don’t separate network functions.


http://iparchitechs.com/contact

Utilizing BGP Communities for traffic steering – part 1: Firewalls

Overview:

I typically spend more time in the enterprise data center than most of our team members and this comes with its own unique set of problems. One discussion that seems to never fail to come up is “where do I put the Firewalls (FWs)?”. That is typically followed by I have a disaster recovery or backup site with FWs there as well. This inevitably leads to a state management problem. Let’s look at how we can utilize BGP to address this problem:

  • what is a BGP standard community
  • BGP best path selection process
  • how to utilize them to steer traffic

This is something most service providers deal with on a daily basis but can be new to an enterprise.

BGP Standard communities

A BGP community is a route attribute that, essentially provides extra information for someone to take action or glean information from the route such as where it came from (location, type, organizational role).

By definition, a community is a 32 bit number that can be included with a route and when utilizing the new community format is displayed as (0-65535):(0-65535). It is recommend to utilize the new community format versus the old community format which is Continue reading

MikroTik – RouterOSv7 first look – feedback on routing filters

Overview

Routing filters have been a hot topic lately in the world of RouterOSv7. The first implementation of routing filters in ROSv7 was difficult to work with and documented in the two articles below:

MikroTik – RouterOSv7 first look – Dynamic routing with IPv6 and OSPFv3/BGP

MikroTik RouterOS – v7.0.3 stable (chateau) and status of general release


MikroTik then made some changes and opened up discussion to get feedback. I did a lot of work and testing using ROS 7.1beta7 which never made it to public release and was close to publishing the results when 7.1rc1 came out so this post will use that version.

https://forum.mikrotik.com/viewtopic.php?f=1&t=175201#p867750


RouterOSv7.1rc1 syntax example

Here is an example of the latest syntax in ROSv7.1rc1

CLI

### MikroTik RouterOS 7.1rc1 ###

/routing filter rule
add chain=dead.beef.101 rule="if (dst==200:dead:beef:101::/64) {accept}"
add chain=dead.beef.102 rule="if (dst==200:dead:beef:102::/64) {accept}"
add chain=dead.beef.agg rule="if (dst in 200:dead:beef::/48) {accept}"
add chain=bgp-out-v6 rule="if (chain dead.beef.101) {set bgp-local-pref 300; accept}"
add chain=bgp-out-v6 rule="if (chain dead.beef.102) {accept}"
add chain=bgp-out-v6 rule="if (chain dead.beef.agg && dst-len<128) {set bgp-local-pref 150; accept}"

Winbox


And the corresponding routes received Continue reading

WISP cheat sheet for MPLS/VPLS MTU

A guide for working with MTU and MPLS in a Wireless ISP

MPLS/VPLS MTU math can be complicated and is always a struggle to unravel.

To make it a little easier and put it into a WISP context, I designed this cheat sheet on 8.5 x 11 (to print for those that actually trust printers) and used common WISP equipment like MikroTik routers, Ubnt and Cambium radios with real world MTU values.

The MTU values are displayed in layers to make it easier to see where each value fits.

PDF is here

A starting point

These values are meant to be a starting point by representing the minimum values required for MPLS/VPLS with a single 802.1q VLAN tag.

In general, after going through hundreds of WISP migrations, I’ve found it to be easier to implement the minimum values required when working on a production WISP to identify the effective lowest MTU in the network.

Once the network equipment has been modified and has been running in a stable way on the minimum values, then higher values can be considered and implemented (now that the effective lowest MTU on the network is documented)

https://iparchitechs.com/contact

Let’s talk MPLS-VPLS, part 1 – use cases

Hey guys, today we’ll be taking a look at what are the main reasons that MPLS – and in particular VPLS – is a useful technology tool to get traffic from point A to point Z. There’s really 2 major use cases that I deal with regularly: IPv4 conservation and L2VPN.

Why am I talking about VPLS specifically? Well, mostly because many times I end up working with Mikrotik routers, which only support VPLS. What is VPLS? It stands for Virtual Private Line Service, and it’s a way to deliver layer 2 services over a layer 3 network. Said another way, it connects a single broadcast domain to multiple endpoints across a routed network. I’ll discuss why MPLS is better for you and your network than switching/bridging in part 2 of this series – for now just know that MPLS/VPLS will allow you to offer enhanced services without the risks of extending layer 2 (I’ll talk more about that below, and why that’s bad in part 2, also).


Use case 1: IPv4 Conservation

OK, so let’s visualize the problem with IP conservation on a small /24 allocation.

Subnetted /24 network

If you’re like most other service providers, you have a Continue reading

MikroTik RouterOS – v7.0.3 stable (chateau) and status of general release

If you don’t already use it, the MIkroTik v7 BETA forum (forum.mikrotik.com) is a fantastic source of information


When will stable be released?

This is the million dollar question. Technically, it already has been for one hardware platform…

!! Spoiler Alert – There is *already* a stable release of ROSv7 – v7.0.3!!

The Chateau 5G router originally shipped with a beta version of ROSv7 but was quietly moved to a stable version that’s developed specifically for that platform.

https://forum.mikrotik.com/viewtopic.php?t=175201#p865329

Because of the way MikroTik’s code repo works, this version can’t easily be added to the main download page and support provides the software:

ROSv7.0.3 Stable Download (!!! Chateau Only – will brick other hardware !!!)

https://box.mikrotik.com/f/7e3cad5779804d0b878d/?dl=1

It’s worth repeating MikroTik’s warning about using this on any platform other than the Chateau

v7 launch date – MikroTikhttps://forum.mikrotik.com/viewtopic.php?f=1&t=175201#p865452


https://iparchitechs.com/contact



What’s holding up v7 from being released?

If you’ve been around MikroTik for a while, then you know that version 7 has been in the works for a long time to add new functionality and address limitations of the older Linux kernel in ROSv6.

MikroTik recently Continue reading

1 2 3 5