Category Archives for ""

IP Infusion: EVPN-MPLS first look on GA 6.0

IP Infusion just released OcNOS version 6.0 and the release notes, as well as press release, show a focus on EVPN with an MPLS data plane. Don’t forget EVPN and VxLAN aren’t mutually exclusive, EVPN runs on and was originally designed for a MPLS data plane. I recently discussed this on a podcast EVPN doesn’t need VxLAN if you want to know more on that topic.

Lets take a look at basic EVPN-VPWS and EVPN-VPLS deployment. Since we’re looking at an MPLS data plane we will utilize ISIS-SR for MPLS. We’re utilizing ISIS-SR as it is increasingly replacing LDP and RSVP-TE for label distribution.

IGP and Label Distribution

First let’s look at the IGP setup and label distribution as everything else will be built on top of this. run int lo
interface lo
 ip address
 ip address secondary
 ipv6 address ::1/128
 ipv6 address 2001:db8::1/128
 prefix-sid index 101
 ip router isis UNDERLAY
 ipv6 router isis UNDERLAY

We have to set an index to create the node-sid for this device. In this case we use 101. run segment-routing
  Continue reading

Situational Awareness for Network Migrations

At IP Architechs we perform a lot of network migrations and it is no secret network migrations/ maintenance windows can be one of the most nerve-racking things for engineers, managers, and business leaders for a variety of reasons.

For the engineers the uncertainty might be caused by fear of failure, not being able to predict the outcome due to complexity, rushed on preparation to meet a deadline, or a litany of other reasons.

For managers and business leaders it might be more along the lines of; what happens if this goes wrong, how will this effect my bottom line, are there going to be 1000s of trouble tickets come 8/9am when everyone hits the office, and so on.

The Preparation

We’re going to look at this at the perspective of the engineer throughout. The prep work is probably one of the most important pieces of success. This is where you do many things including but not limited to:

  • building and testing the configuration to be implemented
  • making a rollback plan — this might be something as simple as move a cable and shut an interface or a multistep/multi-device plan
  • know the situation surrounding the window

Lets explore understanding the situation Continue reading

BGP Communities part 4: Active/Active datacenter

If you read part 2 of this series and came out wondering this is great but:

How do I connect to the internet?

Does this breakdown once I need to have connections?

What else do I have to do to manage state?

We’ll set out to answer these questions and show how it works. There are some dependancies such as your provider supporting customer BGP TE communities as laid out in part 3.

This seems to be the elusive grail in enterprise networking that everyone wants but is unsure of where to start. Hopefully, a few of those questions have been answered throughout this series but be sure to understand what you’re getting into and that your team can support it before and after you leave.

The overall topology

We’ve got data center 1 (DC1) and data center 2 (DC2). They each have a connection to an internal router in ASN 60500. A lot of networks I come across have dedicated routers coming out of the DC to terminate internet connections and support full tables. These router usually only pass a default internally. I don’t have the full tables but instead copy the topology and pass a default into the Continue reading

BGP communities part 3: Customer BGP Traffic Engineering communities

If you’ve ever been asked to prioritize one internet connection over another for any variety of reasons, cost, latency, SLA, etc… this is for you.

Often I hear the same tactics to solve this problem:

  • AS-PATH prepending
  • conditional advertisements
  • scripting
  • some other manual process

However, most carriers offer customer BGP TE communities that you can use to influence traffic within their AS, with one notable exception Hurricane Electric. If you’re not sure what a BGP community take a quick look at this post on them first.

Lets explore how to utilize these, where to find them, and how they might give more deterministic path selection than the options laid out above.

BGP Topology

Default behavior with no modification

First to get familiar with the topology and show reachability we’ll leave all settings as “defaults” with no modifications.

ISP-1-RTR-1#traceroute source
Type escape sequence to abort.
Tracing the route to
VRF info: (vrf in name/id, vrf out name/id)
  1 1 msec 1 msec 1 msec
  2 1 msec 0 msec 0 msec
  3 2 msec *  1  Continue reading

MikroTik – ROSv7 – VPLS over IPv6 MPLS with LDPv6

The great migration

IPv6 adoption has really picked up in the last 12 months and MikroTik RouterOSv7 development is no exception. Dual stack networks are still the most common and easiest to initially deploy for carriers.

However, single stack networks with IPv4 as a service overlay are definitely on the horizon for MikroTik users now that MPLS can operate purely on IPv6.

Single stack networks are easier and cheaper to operate in the long run and are a natural evolution of dual stack networks as we begin to turn IPv4 off for underlay infrastructure.


There are a few different ways to distribute labels in IPv6 MPLS. SR-MPLS (less common and usually with IS-IS) and SRv6 are the other options besides LDPv6.

While I generally am in favor of SR-MPLS/SRv6 long term due to the protocol simplification and traffic management capabilities, having an IPv6 MPLS stack is a great starting point for MikroTik.

LDPv6 is defined by RFC 7552 and is fairly recent as it finalized in 2015. It generally operates in much the same way as LDPv4.

Use Cases

The most common use case among MikroTik users is more efficient subnetting of IPv4 and directly replaces LDPv4 for this Continue reading

WEBINAR: ISP Design – Separation of Network Functions

Recently, we recorded a webinar to explain a design concept frequently used by to build and migrate WISP, FISP and Telco networks – separation of network functions. It centers around simplification of roles within an ISP network. It also explores the use of lower-cost commodity network equipment to maximize the service area for a given ISP footprint while meeting key requirements like scale, redundancy and capacity.

Topics that were covered include:

  • What are network functions?
  • Design examples for WISP/FISP and Telco
  • Equipment and budget considerations

Here is an example of solving design/operational issues with network function separation:

Link to the webinar and slides:

Webinar: Webinar Recording
Slides: Slide Deck

MikroTik – RouterOS v7 – BGP performance testing for full tables

MikroTik has come a long way since the first release of RouterOS v7 beta.

One of the long-awaited features is improved BGP performance and the ability to leverage multiple CPU cores.

Testing BGP performance is a long process of lab and prod evaluation, so we decided to run some quick and basic tests to get a baseline.

When the CCR2216-1G-12XS-2XQ was released and MikroTik entered the world of 100G, we ordered some right away to test and just got them in the lab a few days ago – the results are below.

Hope this is helpful and look for more BGP perf tests in the coming months!

The BGP testing lab

TLDR; 2.1 million routes learned and forwarding in 46 seconds and withdrawn in 44 seconds. This was tested under a 25 Gbps load on both routers with a cpu load of 12%.

Lab overview: The lab consists of (2) CCR2216 routers running ROSv7.2 stable connected to a ProxMox hypervisor that runs (4) Linux route generators and MikroTik CHRs (also on 7.2) acting as border routers. The specific connectivity is in the overview drawing below.

IPv6: We are currently developing a route generator that will inject IPv4 Continue reading

Networking Field Day: Service Provider – Arista VPLS

During the first networking field day: service provider one of the big topics was EVPN versus VPLS. Arista has put a lot of work into their EVPN deployment and this has give then a ton of success in the data center. However, a large portion of the provider space, especially last mile providers, rely on VPLS heavily. This naturally led to discussion about Arista VPLS support.

I’m pleased to see that there is now basic support in EOS as of EOS 4.27.2F and more on the roadmap. Hopefully, we’ll see the off ramp, RFC8560, from VPLS to EVPN which was a hot button topic throughout the week.

In the release notes for EOS 4.27.2F it calls our basic VPLS support. So I took a look. Reviewing the new 4.27.2F manual I found support for LDP PWs on RFC4447 which is virtual private wire support. This also appeared to be in EOS 4.26 but not earlier. I might have missed a different call out in the doc, if I did someone please point me in the right direction so I can update this post!

In the meantime lets review how this works:

mpls  Continue reading

App delivery for an improved pizza experience

It’s been a while since we started work on one of our newest projects.  We have been trying to solve a problem in app location.  It all came from the notion that Little Caesars know where my pizza is, so why can’t the network resolve where the app is?    We also thought it would be novel use of Anycast because the app can be anywhere. 

So, what problems specifically have we solved using this design?  Intent based gateways are a signaling mechanism allows the apps to be delivered along with the pizza.  As we can see app Buffalo Wings can reach both the intent based gateway and Fried Pickles using TI-LFA, which strips the fat bits before they reach the gateway.   Our unique caching solution using Tupperware, which are stacked in K8s, allows for the apps to be delivered in a bursty nexthop specific competitive manner.  This has proven to keep the apps warm within the physical layer.

In our example, the Delivery Center Interconnect,  we are doing an east to west Multi Pizza Layered Service that can drop the apps with full BTU into any of the regions.  The apps are Continue reading

WEBINAR: Deploying IPv6 for WISPs and FISPs.

A few weeks ago, we recorded a webinar on deploying IPv6 for WISPs and FISPs. As IPv6 adoption continues to climb, developing an IPv6 strategy for design, deployment and system integration is an important step before subscribers begin asking for IPv6.

Some of the topics that were covered include:

  • IPv6 basics – addressing, subnetting, types
  • IPv6 design and deployment
  • IPv6 systems and operations

Here is an example of getting started with IPv6 deployment at the border of the ASN

Link to the webinar and slide deck

Webinar: click here

Slides: click here

BGP communities for traffic steering – part 2: State Management across Data Centers

This post has been a while in the making and follows up on an article about BGP communities that can be found here. Then we followed it up with some more discussion about FW design and place, or lack there of, on this podcast which inspired me to finish up “part 2”.

Anyone who has ever had to run active/active data centers and has come across this problem of how do I manage state?

You can ignore it and prepare yourself for a late night at the worst time.

Take everyone’s word that systems will never have to talk to the a system in a different security zone in the remote DC

Utilize communities and BGP policy to manage state; which we’ll focus on here

One of the biggest reasons we see for stretching a virtual routing and forwarding (vrf) is to move DC to DC flows of the same security zone below FWs. This reduces the load on the firewall and makes for easier rule management. However, it does introduce a state problem.

We’ll be using the smallest EVPN-multisite deployment you’ve ever seen with Nexus 9000v and Fortinet FWs.

Inter vrf intra data center

The first flow we’ll look Continue reading

Interop IP infusion and MikroTik: part 2 – VPLS

VPLS is a pretty common technology in ISPs to either sell layer 2 services or backhaul traffic to a centralized aggregation point to conserve IPv4 space; check out more on that here.

How can I take advantage of segment routing but still deliver the same services? We already looked at how to setup the label switched paths utilizing IP Infusion as a segment routing mapping server in this interop post. Now let’s see how we can deliver a VPLS service over this with mikrotik as a provider edge router.

Delivering a service with a L2VPN

After setting up the IGP and label distribution between the PEs we will start building the L2VPN.

Why a BGP signaled VPLS session instead of LDP signaled VPLS?

In the segment-routing domain there is no LDP running. However, some vendors support static pseudowires or other methods to bring up a targeted LDP session for VPLS. I did some basic testing here couldn’t easily identify the right combination of knobs to make this work. Don’t worry; I’ll come back to it.

BGP signaled VPLS is a standards based technology that both vendors support.

First thing we need to do after having loopback reachability is to build Continue reading

Interop LDP and Segment Routing with IP infusion and MikroTik


During networking field day service provider 1 there was a ton of talk about segment routing (SR) and ethernet virtual private networks (EVPN). One of the biggest questions was “how do we get there?” and while we won’t examine EVPN in this post (it’s coming in a future post don’t worry) we will look at how you can take advantage of SR while still having large portions of LDP in your network.

The team here at IP architechs works on a lot of MikroTik and whitebox gear so we’ll focus on a deployment using MikroTik and IP infusion.

MPLS and IGP setup

The first thing to accomplish is end to end reachability between the provider edge (PE) routers. MikroTik doesn’t support IS-IS so we will have to perform redistribution between the IS-IS segment and the OSPF segment as seen above.

MPLS only requires the /32s of the loopbacks for functionality so redistribution is limited to the /32 loopbacks of the PE routers.

ip prefix-list LDP-PE-LOOPBACKS
 seq 10 permit eq 32
ip prefix-list SR-PE-LOOPBACKS
 seq 10 permit eq 32
route-map REDIS-OSPF-TO-ISIS permit 10
 match ip address prefix-list LDP-PE-LOOPBACKS
route-map REDIS-ISIS-TO-OSPF permit  Continue reading

Network Modeling: Automating Mikrotik RouterOS CHR Containerlab images


In a previous post, we talked about using Containerlab and ZeroTier to provide remote access to a lab instance. One of the things that was glossed over was creating the Containerlab images. As part of building the images, Containerlab ready Mikrotik RouterOS images are available on Docker Hub.

Containerlab images

Containerlab currently supports 2 different kinds of images: pre-built containers and VMs packaged as containers.

Presently, only a handful of NOS vendors provide pre-built containers (Nokia’s SR Linux, Juniper cRPD, Arista cEOS, Cumulus VX, and SONIC VS).

The other option utilizes vrnetlab and is essentially a QEMU VM packaged in a Docker container. The more traditional network operating systems are currently supported via this method. This includes operating systems like Mikrotik RouterOS, Juniper vMX, Nokia SROS, and many more.

Building Mikrotik RouterOS containers

Containerlab provides a nice and easy way to build Containerlab ready Docker images. The process is fairly simple.

Setting it up

Containerlab uses a custom fork of vrnetlab that sets things up so that the images built will work with Containerlab. That repository can be found on Github. The first step is to clone that repository.

The next step is to download the Continue reading

MikroTik CHR – Breaking the 100G barrier


The world is strange today. Despite the Covid-19 crisis all over the world, most ISPs are fighting a battle to deliver more bandwidth on a daily basis.

  • Work from home
  • Online schools
  • increasing content consumption

All pushed ISPs to their bandwidth limits, leaving ISP’s no option but to look for upgrades, for everlasting bandwidth demands. There, they are having another set of problems, facing them in this completely new and strange world. Chip shortage, logistic and labor health issues caused higher prices and no stock availability. Here in IP ArchiTechs, we are spending lot of our time finding a good solution for our customers and to help them overcome these hard times. Whether that is our regular Team meeting or just a chat with our colleagues in almost any occasion someone mentions something about new solution to improve capacity and performance for our customers.

Starting with a thought, what is available as a platform today, and of course it’s ready to be shipped immediately after you checkout and pay one thing obviously was just in front of me. X86 server, dozens of them. They are left from the time when we were buying new hardware just because new generation Continue reading

Network Modeling: Segmented Lab access with Containerlab and ZeroTier


When building out network labs, often multiple people will need access to the lab. The main way right now is to use something like EVE-NG or GNS3 to provide access.

There are 2 downsides to this method. The first is that your server is exposed to the internet and if your usernames/passwords aren’t strong enough, your server can become compromised. The second is that sometimes you may not want everyone to be able to add or edit to the lab topology.

The solution to this is using Containerlab and ZeroTier. This setup is great for things like testing new hires, training classes, or for providing lab access to others on a limited basis.

What is Containerlab?

Containerlab is a container orchestrating tool for managing container-based networking labs. It doesn’t just support Container based Network Operating Systems though. Through vrnetlab, there is support for a wide variety of commonly used NOSes: Mikrotik RouterOS, Nokia SROS, Juniper vMX and vQFX, and many more.

Configs are text based making it easy to add/update links between nodes. The lab does need to be destroyed and redeployed when adding/removing links. With some tools to generate configs, it’s easy to spin up a 500+ node Continue reading

Migrating from fabricpath to EVPN/VxLAN


Do you have a 3 tier, switched, or vendor proprietary data center design?

Does it rely on spanning tree or proprietary solutions to eliminate spanning tree?

Not sure how to migrate to a new architecture without serious downtime?

If you answered yes to any of these questions then this post is for you. We’ll be looking at deploying an EVPN/VxLAN Data Center fabric and migrating a from a cisco fabricpath environment to the new design.

Although we will be focusing on a fabricpath migration many, if not all, of the principles apply to migrating a 3 tier architecture.

1. Building the new Data Center Fabric
2. Connecting the current fabricpath and new fabric
3. Migrating switched virtual interfaces
4. Migrating various types of physical devices

Building the new Data Center Fabric

The easiest part of designing and building the new fabric is the physical topology. This should be a symmetric topology to easily take advantage of equal cost multipath and add additional switches with ease. This is also known as a spine/leaf or clos topology. The basic idea is leafs connect to spines and spines connect to super spines. A leaf/spine should not connect to another switch of the Continue reading

ISP Design Guide: Separation of network functions – introduction and overview

PDF link is here

A reference guide for new & existing ISPs that need to understand network functions and separation.

“How do I add redundancy?”
“How do I scale?”
“How do I reduce downtime and operational costs?”

These are questions that I get asked practically every day as a consulting network architect that designs and builds ISPs.

In most cases the answer is the same whether the ISP uses fixed wireless broadband, copper or fiber to deliver the last mile – separation of network functions.

This illustrated guide is intended to define the topic and create visual context for each function using a network drawing. It’s the first in a new series on this subject.

A new series of content

This topic is deep and there is a lot to unpack so this will be the first segment in a series of blog posts and videos covering function separation.

Large ISPs typically already embrace the philosophy of separating network functions, so the focus of this series will be to help new or growing regional ISPs understand the design intent and the challenges/costs of running networks that don’t separate network functions.

Utilizing BGP Communities for traffic steering – part 1: Firewalls


I typically spend more time in the enterprise data center than most of our team members and this comes with its own unique set of problems. One discussion that seems to never fail to come up is “where do I put the Firewalls (FWs)?”. That is typically followed by I have a disaster recovery or backup site with FWs there as well. This inevitably leads to a state management problem. Let’s look at how we can utilize BGP to address this problem:

  • what is a BGP standard community
  • BGP best path selection process
  • how to utilize them to steer traffic

This is something most service providers deal with on a daily basis but can be new to an enterprise.

BGP Standard communities

A BGP community is a route attribute that, essentially provides extra information for someone to take action or glean information from the route such as where it came from (location, type, organizational role).

By definition, a community is a 32 bit number that can be included with a route and when utilizing the new community format is displayed as (0-65535):(0-65535). It is recommend to utilize the new community format versus the old community format which is Continue reading

MikroTik – RouterOSv7 first look – feedback on routing filters


Routing filters have been a hot topic lately in the world of RouterOSv7. The first implementation of routing filters in ROSv7 was difficult to work with and documented in the two articles below:

MikroTik – RouterOSv7 first look – Dynamic routing with IPv6 and OSPFv3/BGP

MikroTik RouterOS – v7.0.3 stable (chateau) and status of general release

MikroTik then made some changes and opened up discussion to get feedback. I did a lot of work and testing using ROS 7.1beta7 which never made it to public release and was close to publishing the results when 7.1rc1 came out so this post will use that version.

RouterOSv7.1rc1 syntax example

Here is an example of the latest syntax in ROSv7.1rc1


### MikroTik RouterOS 7.1rc1 ###

/routing filter rule
add chain=dead.beef.101 rule="if (dst==200:dead:beef:101::/64) {accept}"
add chain=dead.beef.102 rule="if (dst==200:dead:beef:102::/64) {accept}"
add chain=dead.beef.agg rule="if (dst in 200:dead:beef::/48) {accept}"
add chain=bgp-out-v6 rule="if (chain dead.beef.101) {set bgp-local-pref 300; accept}"
add chain=bgp-out-v6 rule="if (chain dead.beef.102) {accept}"
add chain=bgp-out-v6 rule="if (chain dead.beef.agg && dst-len<128) {set bgp-local-pref 150; accept}"


And the corresponding routes received Continue reading

1 2 3 5