MikroTik ROS 7.14beta8 released
MikroTik Routers and Wireless – Software
RouterOS continues to mature as we move through the versions in the teens.
When we transitioned between ROSv5 and ROSv6 in the early 2010s, it was right around this version numbering that we started to see production stability. By the time 6.2x versions came out, the general consensus was that v6 was ready for prime time. We are getting closer to that point in ROSv7 – depending on your use case.
Certainly, there are still issues to solve for advanced users like ISPs and Data Centers that need protocols like BGP, OSPF, IS-IS and MPLS, but simpler use cases seem to really be stabilizing with the last few months of releases.
Notable changes in this release:
*) bgp – allow to leak routes between local VRFs;
There are a few reasons this is a really important addition to ROSv7. First, it’s an issue that’s been on the roadmap for a very long time as noted in the Routing Protocol Overview section of MikroTik’s help docs. This is encouraging because it’s likely been one of the harder problems for the development team to solve given the length of time it sat open.
Secondly, it’s Continue reading
WISP/FISP Design: Switch Centric (SWC) Topology
Overview
This is an article i’ve wanted to write for a long time. In the last decade, the work that we have done at iparchitechs.com with WISPs/FISPs in network design using commodity equipment like MikroTik and FiberStore has yielded quite a few best practices and lessons learned.
While the idea of “router on a stick” isn’t new, when we first started working with WISPs/FISPs and MikroTik routers 10+ years ago, we immediately noticed a few common elements in the requests we’d get for consulting:
“I’m out of ports on my router…how do I add more?”
“I started with a single router, how do I make it redundant and keep NAT/peering working properly”?
“I have high CPU on my router and I don’t know how to add capacity and split the traffic”
“I can’t afford Cisco or Juniper but I need a network that’s highly available and resilient”
Coming from a telco background where a large chassis was used pretty much everywhere for redundancy and relying on links split across multiple line cards with LACP, that was one of my first inclinations to solve the Continue reading
IP Infusion: EVPN-MPLS first look on GA 6.0
IP Infusion just released OcNOS version 6.0 and the release notes, as well as press release, show a focus on EVPN with an MPLS data plane. Don’t forget EVPN and VxLAN aren’t mutually exclusive, EVPN runs on and was originally designed for a MPLS data plane. I recently discussed this on a podcast EVPN doesn’t need VxLAN if you want to know more on that topic.
Lets take a look at basic EVPN-VPWS and EVPN-VPLS deployment. Since we’re looking at an MPLS data plane we will utilize ISIS-SR for MPLS. We’re utilizing ISIS-SR as it is increasingly replacing LDP and RSVP-TE for label distribution.
IGP and Label Distribution
First let’s look at the IGP setup and label distribution as everything else will be built on top of this.
ipi-1.lab.jan1.us.ipa.net#show run int lo
interface lo
ip address 127.0.0.1/8
ip address 100.127.0.1/32 secondary
ipv6 address ::1/128
ipv6 address 2001:db8::1/128
prefix-sid index 101
ip router isis UNDERLAY
ipv6 router isis UNDERLAY
!We have to set an index to create the node-sid for this device. In this case we use 101.
ipi-1.lab.jan1.us.ipa.net#show run segment-routing
segment-routing
Continue readingSituational Awareness for Network Migrations
At IP Architechs we perform a lot of network migrations and it is no secret network migrations/ maintenance windows can be one of the most nerve-racking things for engineers, managers, and business leaders for a variety of reasons.
For the engineers the uncertainty might be caused by fear of failure, not being able to predict the outcome due to complexity, rushed on preparation to meet a deadline, or a litany of other reasons.
For managers and business leaders it might be more along the lines of; what happens if this goes wrong, how will this effect my bottom line, are there going to be 1000s of trouble tickets come 8/9am when everyone hits the office, and so on.
The Preparation
We’re going to look at this at the perspective of the engineer throughout. The prep work is probably one of the most important pieces of success. This is where you do many things including but not limited to:
- building and testing the configuration to be implemented
- making a rollback plan — this might be something as simple as move a cable and shut an interface or a multistep/multi-device plan
- know the situation surrounding the window
Lets explore understanding the situation Continue reading
BGP Communities part 4: Active/Active datacenter
If you read part 2 of this series and came out wondering this is great but:
How do I connect to the internet?
Does this breakdown once I need to have connections?
What else do I have to do to manage state?
We’ll set out to answer these questions and show how it works. There are some dependancies such as your provider supporting customer BGP TE communities as laid out in part 3.
This seems to be the elusive grail in enterprise networking that everyone wants but is unsure of where to start. Hopefully, a few of those questions have been answered throughout this series but be sure to understand what you’re getting into and that your team can support it before and after you leave.
The overall topology
We’ve got data center 1 (DC1) and data center 2 (DC2). They each have a connection to an internal router in ASN 60500. A lot of networks I come across have dedicated routers coming out of the DC to terminate internet connections and support full tables. These router usually only pass a default internally. I don’t have the full tables but instead copy the topology and pass a default into the Continue reading
BGP communities part 3: Customer BGP Traffic Engineering communities
If you’ve ever been asked to prioritize one internet connection over another for any variety of reasons, cost, latency, SLA, etc… this is for you.
Often I hear the same tactics to solve this problem:
- AS-PATH prepending
- conditional advertisements
- scripting
- some other manual process
However, most carriers offer customer BGP TE communities that you can use to influence traffic within their AS, with one notable exception Hurricane Electric. If you’re not sure what a BGP community take a quick look at this post on them first.
Lets explore how to utilize these, where to find them, and how they might give more deterministic path selection than the options laid out above.
BGP Topology
Default behavior with no modification
First to get familiar with the topology and show reachability we’ll leave all settings as “defaults” with no modifications.
ISP-1-RTR-1#traceroute 203.0.113.1 source 192.0.2.1
Type escape sequence to abort.
Tracing the route to 203.0.113.1
VRF info: (vrf in name/id, vrf out name/id)
1 100.123.0.1 1 msec 1 msec 1 msec
2 100.124.0.2 1 msec 0 msec 0 msec
3 100.126.0.10 2 msec * 1 Continue readingMikroTik – ROSv7 – VPLS over IPv6 MPLS with LDPv6
The great migration
IPv6 adoption has really picked up in the last 12 months and MikroTik RouterOSv7 development is no exception. Dual stack networks are still the most common and easiest to initially deploy for carriers.
However, single stack networks with IPv4 as a service overlay are definitely on the horizon for MikroTik users now that MPLS can operate purely on IPv6.
Single stack networks are easier and cheaper to operate in the long run and are a natural evolution of dual stack networks as we begin to turn IPv4 off for underlay infrastructure.
LDPv6
There are a few different ways to distribute labels in IPv6 MPLS. SR-MPLS (less common and usually with IS-IS) and SRv6 are the other options besides LDPv6.
While I generally am in favor of SR-MPLS/SRv6 long term due to the protocol simplification and traffic management capabilities, having an IPv6 MPLS stack is a great starting point for MikroTik.
LDPv6 is defined by RFC 7552 and is fairly recent as it finalized in 2015. It generally operates in much the same way as LDPv4.
Use Cases
The most common use case among MikroTik users is more efficient subnetting of IPv4 and directly replaces LDPv4 for this Continue reading
WEBINAR: ISP Design – Separation of Network Functions
Recently, we recorded a webinar to explain a design concept frequently used by iparchitechs.com to build and migrate WISP, FISP and Telco networks – separation of network functions. It centers around simplification of roles within an ISP network. It also explores the use of lower-cost commodity network equipment to maximize the service area for a given ISP footprint while meeting key requirements like scale, redundancy and capacity.
Topics that were covered include:
- What are network functions?
- Design examples for WISP/FISP and Telco
- Equipment and budget considerations
Here is an example of solving design/operational issues with network function separation:
Link to the webinar and slides:
Webinar: Webinar Recording
Slides: Slide Deck
MikroTik – RouterOS v7 – BGP performance testing for full tables
MikroTik has come a long way since the first release of RouterOS v7 beta.
One of the long-awaited features is improved BGP performance and the ability to leverage multiple CPU cores.
Testing BGP performance is a long process of lab and prod evaluation, so we decided to run some quick and basic tests to get a baseline.
When the CCR2216-1G-12XS-2XQ was released and MikroTik entered the world of 100G, we ordered some right away to test and just got them in the lab a few days ago – the results are below.
Hope this is helpful and look for more BGP perf tests in the coming months!
The BGP testing lab
TLDR; 2.1 million routes learned and forwarding in 46 seconds and withdrawn in 44 seconds. This was tested under a 25 Gbps load on both routers with a cpu load of 12%.
Lab overview: The lab consists of (2) CCR2216 routers running ROSv7.2 stable connected to a ProxMox hypervisor that runs (4) Linux route generators and MikroTik CHRs (also on 7.2) acting as border routers. The specific connectivity is in the overview drawing below.
IPv6: We are currently developing a route generator that will inject IPv4 Continue reading
Networking Field Day: Service Provider – Arista VPLS
During the first networking field day: service provider one of the big topics was EVPN versus VPLS. Arista has put a lot of work into their EVPN deployment and this has give then a ton of success in the data center. However, a large portion of the provider space, especially last mile providers, rely on VPLS heavily. This naturally led to discussion about Arista VPLS support.
I’m pleased to see that there is now basic support in EOS as of EOS 4.27.2F and more on the roadmap. Hopefully, we’ll see the off ramp, RFC8560, from VPLS to EVPN which was a hot button topic throughout the week.
In the release notes for EOS 4.27.2F it calls our basic VPLS support. So I took a look. Reviewing the new 4.27.2F manual I found support for LDP PWs on RFC4447 which is virtual private wire support. This also appeared to be in EOS 4.26 but not earlier. I might have missed a different call out in the doc, if I did someone please point me in the right direction so I can update this post!
In the meantime lets review how this works:
mpls Continue readingApp delivery for an improved pizza experience
It’s been a while since we started work on one of our newest projects. We have been trying to solve a problem in app location. It all came from the notion that Little Caesars know where my pizza is, so why can’t the network resolve where the app is? We also thought it would be novel use of Anycast because the app can be anywhere.
So, what problems specifically have we solved using this design? Intent based gateways are a signaling mechanism allows the apps to be delivered along with the pizza. As we can see app Buffalo Wings can reach both the intent based gateway and Fried Pickles using TI-LFA, which strips the fat bits before they reach the gateway. Our unique caching solution using Tupperware, which are stacked in K8s, allows for the apps to be delivered in a bursty nexthop specific competitive manner. This has proven to keep the apps warm within the physical layer.
In our example, the Delivery Center Interconnect, we are doing an east to west Multi Pizza Layered Service that can drop the apps with full BTU into any of the regions. The apps are Continue reading
WEBINAR: Deploying IPv6 for WISPs and FISPs.
A few weeks ago, we recorded a webinar on deploying IPv6 for WISPs and FISPs. As IPv6 adoption continues to climb, developing an IPv6 strategy for design, deployment and system integration is an important step before subscribers begin asking for IPv6.
Some of the topics that were covered include:
- IPv6 basics – addressing, subnetting, types
- IPv6 design and deployment
- IPv6 systems and operations
Here is an example of getting started with IPv6 deployment at the border of the ASN
Link to the webinar and slide deck
Webinar: click here
Slides: click here
BGP communities for traffic steering – part 2: State Management across Data Centers
This post has been a while in the making and follows up on an article about BGP communities that can be found here. Then we followed it up with some more discussion about FW design and place, or lack there of, on this podcast which inspired me to finish up “part 2”.
Anyone who has ever had to run active/active data centers and has come across this problem of how do I manage state?
You can ignore it and prepare yourself for a late night at the worst time.
Take everyone’s word that systems will never have to talk to the a system in a different security zone in the remote DC
Utilize communities and BGP policy to manage state; which we’ll focus on here
One of the biggest reasons we see for stretching a virtual routing and forwarding (vrf) is to move DC to DC flows of the same security zone below FWs. This reduces the load on the firewall and makes for easier rule management. However, it does introduce a state problem.
We’ll be using the smallest EVPN-multisite deployment you’ve ever seen with Nexus 9000v and Fortinet FWs.
Inter vrf intra data center
The first flow we’ll look Continue reading
Interop IP infusion and MikroTik: part 2 – VPLS
VPLS is a pretty common technology in ISPs to either sell layer 2 services or backhaul traffic to a centralized aggregation point to conserve IPv4 space; check out more on that here.
How can I take advantage of segment routing but still deliver the same services? We already looked at how to setup the label switched paths utilizing IP Infusion as a segment routing mapping server in this interop post. Now let’s see how we can deliver a VPLS service over this with mikrotik as a provider edge router.
Delivering a service with a L2VPN
After setting up the IGP and label distribution between the PEs we will start building the L2VPN.
Why a BGP signaled VPLS session instead of LDP signaled VPLS?
In the segment-routing domain there is no LDP running. However, some vendors support static pseudowires or other methods to bring up a targeted LDP session for VPLS. I did some basic testing here couldn’t easily identify the right combination of knobs to make this work. Don’t worry; I’ll come back to it.
BGP signaled VPLS is a standards based technology that both vendors support.
First thing we need to do after having loopback reachability is to build Continue reading
Interop LDP and Segment Routing with IP infusion and MikroTik
Introduction
During networking field day service provider 1 there was a ton of talk about segment routing (SR) and ethernet virtual private networks (EVPN). One of the biggest questions was “how do we get there?” and while we won’t examine EVPN in this post (it’s coming in a future post don’t worry) we will look at how you can take advantage of SR while still having large portions of LDP in your network.
The team here at IP architechs works on a lot of MikroTik and whitebox gear so we’ll focus on a deployment using MikroTik and IP infusion.
MPLS and IGP setup
The first thing to accomplish is end to end reachability between the provider edge (PE) routers. MikroTik doesn’t support IS-IS so we will have to perform redistribution between the IS-IS segment and the OSPF segment as seen above.
MPLS only requires the /32s of the loopbacks for functionality so redistribution is limited to the /32 loopbacks of the PE routers.
ip prefix-list LDP-PE-LOOPBACKS
seq 10 permit 100.127.2.0/24 eq 32
!
ip prefix-list SR-PE-LOOPBACKS
seq 10 permit 100.127.0.0/24 eq 32
!
route-map REDIS-OSPF-TO-ISIS permit 10
match ip address prefix-list LDP-PE-LOOPBACKS
!
route-map REDIS-ISIS-TO-OSPF permit Continue readingNetwork Modeling: Automating Mikrotik RouterOS CHR Containerlab images
Introduction
In a previous post, we talked about using Containerlab and ZeroTier to provide remote access to a lab instance. One of the things that was glossed over was creating the Containerlab images. As part of building the images, Containerlab ready Mikrotik RouterOS images are available on Docker Hub.
Containerlab images
Containerlab currently supports 2 different kinds of images: pre-built containers and VMs packaged as containers.
Presently, only a handful of NOS vendors provide pre-built containers (Nokia’s SR Linux, Juniper cRPD, Arista cEOS, Cumulus VX, and SONIC VS).
The other option utilizes vrnetlab and is essentially a QEMU VM packaged in a Docker container. The more traditional network operating systems are currently supported via this method. This includes operating systems like Mikrotik RouterOS, Juniper vMX, Nokia SROS, and many more.
Building Mikrotik RouterOS containers
Containerlab provides a nice and easy way to build Containerlab ready Docker images. The process is fairly simple.
Setting it up
Containerlab uses a custom fork of vrnetlab that sets things up so that the images built will work with Containerlab. That repository can be found on Github. The first step is to clone that repository.
The next step is to download the Continue reading
MikroTik CHR – Breaking the 100G barrier
Introduction
The world is strange today. Despite the Covid-19 crisis all over the world, most ISPs are fighting a battle to deliver more bandwidth on a daily basis.
- Work from home
- Online schools
- increasing content consumption
All pushed ISPs to their bandwidth limits, leaving ISP’s no option but to look for upgrades, for everlasting bandwidth demands. There, they are having another set of problems, facing them in this completely new and strange world. Chip shortage, logistic and labor health issues caused higher prices and no stock availability. Here in IP ArchiTechs, we are spending lot of our time finding a good solution for our customers and to help them overcome these hard times. Whether that is our regular Team meeting or just a chat with our colleagues in almost any occasion someone mentions something about new solution to improve capacity and performance for our customers.
Starting with a thought, what is available as a platform today, and of course it’s ready to be shipped immediately after you checkout and pay one thing obviously was just in front of me. X86 server, dozens of them. They are left from the time when we were buying new hardware just because new generation Continue reading
Network Modeling: Segmented Lab access with Containerlab and ZeroTier
Introduction
When building out network labs, often multiple people will need access to the lab. The main way right now is to use something like EVE-NG or GNS3 to provide access.
There are 2 downsides to this method. The first is that your server is exposed to the internet and if your usernames/passwords aren’t strong enough, your server can become compromised. The second is that sometimes you may not want everyone to be able to add or edit to the lab topology.
The solution to this is using Containerlab and ZeroTier. This setup is great for things like testing new hires, training classes, or for providing lab access to others on a limited basis.
What is Containerlab?
Containerlab is a container orchestrating tool for managing container-based networking labs. It doesn’t just support Container based Network Operating Systems though. Through vrnetlab, there is support for a wide variety of commonly used NOSes: Mikrotik RouterOS, Nokia SROS, Juniper vMX and vQFX, and many more.
Configs are text based making it easy to add/update links between nodes. The lab does need to be destroyed and redeployed when adding/removing links. With some tools to generate configs, it’s easy to spin up a 500+ node Continue reading
Migrating from fabricpath to EVPN/VxLAN
Introduction
Do you have a 3 tier, switched, or vendor proprietary data center design?
Does it rely on spanning tree or proprietary solutions to eliminate spanning tree?
Not sure how to migrate to a new architecture without serious downtime?
If you answered yes to any of these questions then this post is for you. We’ll be looking at deploying an EVPN/VxLAN Data Center fabric and migrating a from a cisco fabricpath environment to the new design.
Although we will be focusing on a fabricpath migration many, if not all, of the principles apply to migrating a 3 tier architecture.
1. Building the new Data Center Fabric
2. Connecting the current fabricpath and new fabric
3. Migrating switched virtual interfaces
4. Migrating various types of physical devices
Building the new Data Center Fabric
The easiest part of designing and building the new fabric is the physical topology. This should be a symmetric topology to easily take advantage of equal cost multipath and add additional switches with ease. This is also known as a spine/leaf or clos topology. The basic idea is leafs connect to spines and spines connect to super spines. A leaf/spine should not connect to another switch of the Continue reading
ISP Design Guide: Separation of network functions – introduction and overview
PDF link is here
A reference guide for new & existing ISPs that need to understand network functions and separation.
“How do I add redundancy?”
“How do I scale?”
“How do I reduce downtime and operational costs?”
These are questions that I get asked practically every day as a consulting network architect that designs and builds ISPs.
In most cases the answer is the same whether the ISP uses fixed wireless broadband, copper or fiber to deliver the last mile – separation of network functions.
This illustrated guide is intended to define the topic and create visual context for each function using a network drawing. It’s the first in a new series on this subject.
A new series of content
This topic is deep and there is a lot to unpack so this will be the first segment in a series of blog posts and videos covering function separation.
Large ISPs typically already embrace the philosophy of separating network functions, so the focus of this series will be to help new or growing regional ISPs understand the design intent and the challenges/costs of running networks that don’t separate network functions.
http://iparchitechs.com/contact