ISP Architechture – MPLS Overview, Design and Implementation for WISPs
Recently, I was fortunate enough to be invited by Brian Horn with WISPA.org to teach a session at WISP America 2016 in Lousiville, KY. We had the class on Tuesday, March 15th 2016 and the turnout and response were great. Many different people have asked for the presentation, so I decided to go ahead and post it here. Hope this helps some of you who are trying to get into MPLS and although it does have a bit of a WISP focus, almost all of the concepts in the presentation apply to wireline networks as well.
About the presentation
Scope: This session was 30 minutes long with a Q&A afterwards, so the material is really a deep dive on MPLS. The goal was to introduce WISP engineers and owners to MPLS and how it improves the network as well as revenue.
When should I put MPLS in my WISP or Service Provider network? The answer is ASAP! I was asked this question by a small WISP earlier in the week and he said i’m just too small to be thinking about MPLS. My response to him was simply – “Do you want to get MPLS in and working Continue reading
ISP Architechture – MPLS Overview, Design and Implementation for WISPs
Recently, I was fortunate enough to be invited by Brian Horn with WISPA.org to teach a session at WISP America 2016 in Lousiville, KY. We had the class on Tuesday, March 15th 2016 and the turnout and response were great. Many different people have asked for the presentation, so I decided to go ahead and post it here. Hope this helps some of you who are trying to get into MPLS and although it does have a bit of a WISP focus, almost all of the concepts in the presentation apply to wireline networks as well.
About the presentation
Scope: This session was 30 minutes long with a Q&A afterwards, so the material is really a deep dive on MPLS. The goal was to introduce WISP engineers and owners to MPLS and how it improves the network as well as revenue.
When should I put MPLS in my WISP or Service Provider network? The answer is ASAP! I was asked this question by a small WISP earlier in the week and he said i’m just too small to be thinking about MPLS. My response to him was simply – “Do you want to get MPLS in and working Continue reading
Put 500,000+ BGP routes in your lab network!!! Download this VM and become your own upstream BGP ISP for testing.
[adrotate banner=”5″]
Happy New Year and welcome to the VM you can punish your routers with 
Hello from stubarea51.net and Happy New Year! We are back from the holidays and recharged with lots of new stuff in the world of network engineering. If you ever thought it would be cool to put a full BGP table into a lab router, GNS3 or other virtualized router, you’re not alone.
A while back, I tackled this post and got everything up and running:
http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
First of all, thanks to evilrouters.net for figuring out the hard parts so we could build this into a VM. After basking for a while in the high geek factor of this project, it gave me an idea to build a VM that could be distributed among network engineers and IT professionals. The idea is to easily spin up one or more full BGP tables to test a particular network design or convergence speed, playing with BGP attributes, etc. After a few months of tweaking it and getting the VM ready for distribution, we finally are ready to put it out for everyone to use.
Network Diagram
Here is an overview of the topology we Continue reading
Put 500,000+ BGP routes in your lab network!!! Download this VM and become your own upstream BGP ISP for testing.
[adrotate banner=”5″]
Happy New Year and welcome to the VM you can punish your routers with 
Hello from stubarea51.net and Happy New Year! We are back from the holidays and recharged with lots of new stuff in the world of network engineering. If you ever thought it would be cool to put a full BGP table into a lab router, GNS3 or other virtualized router, you’re not alone.
A while back, I tackled this post and got everything up and running:
http://evilrouters.net/2009/08/21/getting-bgp-routes-into-dynamips-with-video/
First of all, thanks to evilrouters.net for figuring out the hard parts so we could build this into a VM. After basking for a while in the high geek factor of this project, it gave me an idea to build a VM that could be distributed among network engineers and IT professionals. The idea is to easily spin up one or more full BGP tables to test a particular network design or convergence speed, playing with BGP attributes, etc. After a few months of tweaking it and getting the VM ready for distribution, we finally are ready to put it out for everyone to use.
Network Diagram
Here is an overview of the topology we Continue reading
Cisco to MikroTik – command translation – BGP
In the world of network engineering, learning a new syntax can challenging especially if you need a lot of detail quickly. The command structure for RouterOS can be a bit challenging sometimes if you are used to Cisco CLI commands. Most of us that have been in networking for a while got our start with Cisco gear and so it is helpful to draw comparisons between the commands, especially if you are trying to build a network with a MikroTik and Cisco router.
This is the first post in a series I’ve wanted to do for a while that creates a Rosetta stone essentially between IOS and RouterOS. We plan to tackle a number of other command comparisons like OSPF, MPLS and VLANs to make it easier for network engineers trained in Cisco IOS to successfully implement MikroTik / RouterOS devices. While many commands have almost the exact same information, others are as close as possible. Since there isn’t always an exact match, sometimes you may have to run two or three commands to get the information needed.
We plan to tackle a number of other command comparisons like OSPF, MPLS and VLANs to make it easier for network Continue reading
Cisco to MikroTik – command translation – BGP
In the world of network engineering, learning a new syntax can challenging especially if you need a lot of detail quickly. The command structure for RouterOS can be a bit challenging sometimes if you are used to Cisco CLI commands. Most of us that have been in networking for a while got our start with Cisco gear and so it is helpful to draw comparisons between the commands, especially if you are trying to build a network with a MikroTik and Cisco router.
This is the first post in a series I’ve wanted to do for a while that creates a Rosetta stone essentially between IOS and RouterOS. We plan to tackle a number of other command comparisons like OSPF, MPLS and VLANs to make it easier for network engineers trained in Cisco IOS to successfully implement MikroTik / RouterOS devices. While many commands have almost the exact same information, others are as close as possible. Since there isn’t always an exact match, sometimes you may have to run two or three commands to get the information needed.
We plan to tackle a number of other command comparisons like OSPF, MPLS and VLANs to make it easier for network Continue reading
MikroTik – CCR1072-1G-8S+ – PPPoE testing preview – 30,000 connections and queues.
[adrotate banner=”5″]
Why we chose PPPoE as the next test
First of all, thanks to everyone for all the positive feedback, comments and questions about the CCR1072-1G-8S+ testing we have been posting in the last few months. Even MikroTik has taken an interest in this testing and we have gotten some great feedback from them as well.
We received more questions about the PPPoE capabilities of the CCR1072-1G-8S+ than any other type of request. Since we have already published the testing on BGP, throughput and EoIP, we have decided to tackle the PPPoE testing to understand where the limits of the CCR1072-1G-8S+ are. This is only a preview of the testing as we are working on different methods of testing and config, but this will at least give you a glimpse of what is possible.
30,000 PPPoE Connections !!!!
Overview of PPPoE connections and CPU load
PRTG Monitoring
We have started using PRTG in the StubArea51.net lab as it makes monitoring of resource load over time much easier when we are testing. Check it out as it is free up to 100 sensors and works very well with MikroTik
https://www.paessler.com/prtg/download
PRTG CPU Profile
PRTG PPPoE Continue reading
MikroTik – CCR1072-1G-8S+ – PPPoE testing preview – 30,000 connections and queues.
[adrotate banner=”5″]
Why we chose PPPoE as the next test
First of all, thanks to everyone for all the positive feedback, comments and questions about the CCR1072-1G-8S+ testing we have been posting in the last few months. Even MikroTik has taken an interest in this testing and we have gotten some great feedback from them as well.
We received more questions about the PPPoE capabilities of the CCR1072-1G-8S+ than any other type of request. Since we have already published the testing on BGP, throughput and EoIP, we have decided to tackle the PPPoE testing to understand where the limits of the CCR1072-1G-8S+ are. This is only a preview of the testing as we are working on different methods of testing and config, but this will at least give you a glimpse of what is possible.
30,000 PPPoE Connections !!!!
Overview of PPPoE connections and CPU load
PRTG Monitoring
We have started using PRTG in the StubArea51.net lab as it makes monitoring of resource load over time much easier when we are testing. Check it out as it is free up to 100 sensors and works very well with MikroTik
https://www.paessler.com/prtg/download
PRTG CPU Profile
PRTG PPPoE Continue reading
10 Gbps of Layer 2 throughput is possible using MikroTik’s EoIP tunnel.
[adrotate banner=”5″]
[metaslider id=282]
Getting to 10 Gbps using EoIP
The EoIP tunnel protocol is one of the more popular features we see deployed in MikroTik routers. It is useful anywhere a Layer 2 extension over a Layer 3 network is needed and can be done with very little effort / complexity. One of the questions that seems to come up on the forums frequently is how much traffic can an EoIP tunnel handle which is typically followed by questions about performance with IPSEC turned on. Answers given by MikroTik and others on forums.mikrotik.com typically fall into the 1 to 3 Gbps range with some hints that more is possible. We searched to see if anyone had done 10 Gbps over EoIP with or without IPSEC and came up empty handed. That prompted us to dive into the StubArea51 lab and set up a test network so we could get some hard data and definitive answers.
The EoIP protocol and recent enhancements
Ethernet over IP or EoIP is a protocol that started as an IETF draft somewhere around 2002 and MikroTik developed a proprietary implementation of it that has been in RouterOS for quite a while. Continue reading
MikroTik CCR1072-1G-8S+ Review – Part 3 – 80 Gbps Throughput testing
[adrotate banner=”5″]
[metaslider id=249]
The 80 Gbps barrier has finally been broken (and yes we are rounding up) !!!!
Well at least it has been reached by someone other than MikroTik. It’s taken us quite a while to get all the right pieces to push 80 Gbps of traffic through the CC1072 but with the latest round of servers that just got delivered to our lab, we were able to go beyond our previous high water mark of 54 Gbps all the way to just under 80 Gbps. There have been a number of questions about this particular router and what the performance will look like in the real world. While this is still a lab test, we are using non-MikroTik equipment and iperf which is considered an extremely accurate performance measuring tool for TCP and UDP.
Video of the CCR1072-1G-8S+ in action (Turn up your volume to hear the roar of the ESXi servers as they approach 80 Gbps)
How we did it – The Hardware
CCR1072-1G-8S+ – Obviously you can’t have a test of the CCR1072 without one to test on. Our CCR1072-1G-8S+ is a pre-production model so there are some minor differences between it and the Continue reading
MikroTik CCR1072-1G-8S+ Review – Part 3 – 80 Gbps Throughput testing
[adrotate banner=”5″]
[metaslider id=249]
The 80 Gbps barrier has finally been broken (and yes we are rounding up) !!!!
Well at least it has been reached by someone other than MikroTik. It’s taken us quite a while to get all the right pieces to push 80 Gbps of traffic through the CC1072 but with the latest round of servers that just got delivered to our lab, we were able to go beyond our previous high water mark of 54 Gbps all the way to just under 80 Gbps. There have been a number of questions about this particular router and what the performance will look like in the real world. While this is still a lab test, we are using non-MikroTik equipment and iperf which is considered an extremely accurate performance measuring tool for TCP and UDP.
Video of the CCR1072-1G-8S+ in action (Turn up your volume to hear the roar of the ESXi servers as they approach 80 Gbps)
How we did it – The Hardware
CCR1072-1G-8S+ – Obviously you can’t have a test of the CCR1072 without one to test on. Our CCR1072-1G-8S+ is a pre-production model so there are some minor differences between it and the Continue reading
MikroTik CCR1072-1G-8S+ Review – update on Part 3 – Throughput
[adrotate banner=”4″]
Breaking the 80 Gbps barrier!!!
The long wait for real-world 1072 performance numbers is almost over – the last two VMWARE server chassis we needed to push the full 80 Gpbs arrived in the StubArea51 lab today. Thanks to everyone who wrote in and commented on the first two reviews we did on the CCR-1072-1G-8S+. We initially began work on performance testing throughput for the CCR1072 in late July of this year, but had to order a lot of parts to get enough 10 Gig PCIe cards, SFP+ modules and fiber to be able to push 80 Gbps of traffic through this router.
Challenges
There have been a number of things that we have had to work through to get to 80 Gbps but we are very close. This will be detailed in the full review we plan to release next week but here are a few.
- VMWARE ESXi – LACP Hashing – Initially we built LACP channels between the ESXi hosts and the 1072 expecting to load the links by using multiple source and destination IPs but we ran into issues with traffic getting stuck on one side of the LACP channel and had to move to Continue reading
MikroTik CCR1072-1G-8S+ Review – update on Part 3 – Throughput
[adrotate banner=”4″]
Breaking the 80 Gbps barrier!!!
The long wait for real-world 1072 performance numbers is almost over – the last two VMWARE server chassis we needed to push the full 80 Gpbs arrived in the StubArea51 lab today. Thanks to everyone who wrote in and commented on the first two reviews we did on the CCR-1072-1G-8S+. We initially began work on performance testing throughput for the CCR1072 in late July of this year, but had to order a lot of parts to get enough 10 Gig PCIe cards, SFP+ modules and fiber to be able to push 80 Gbps of traffic through this router.
Challenges
There have been a number of things that we have had to work through to get to 80 Gbps but we are very close. This will be detailed in the full review we plan to release next week but here are a few.
- VMWARE ESXi – LACP Hashing – Initially we built LACP channels between the ESXi hosts and the 1072 expecting to load the links by using multiple source and destination IPs but we ran into issues with traffic getting stuck on one side of the LACP channel and had to move to Continue reading
MikroTik unveils new RouterOS development cycle
[adrotate banner=”4″]
As we all patiently await the release of RouterOS (RoS) v7 beta, MikroTik has announced a change in the way RoS development is organized. There will now be three different tracks of development:
Bugfix only – When a current build is released, only fixes to known bugs will be added to this branch of development
Current – Current release will contain bugfixes and new features
Release Candidate – The release candidate will remain the development build for the next “current” release.
Graphical Overview of the new development cycle
Image and notes below are from here
A small addendum: the Bugfix only will only contain verified fixes, and no new features. The Current release contains the same fixes but also new features and other improvements, sometimes also less critical fixes than in Bugfix. And finally the Release Candidate is more likely to a nightly build. We will not to intensive testing before publishing these, only quick check if upgrade can be done and if most features work fine.
Origin
The idea originally came out of this thread and after a flurry of positive commentary, it became a working practice shortly therafter.
We plan to make sub-version releases Continue reading
MikroTik unveils new RouterOS development cycle
[adrotate banner=”4″]
As we all patiently await the release of RouterOS (RoS) v7 beta, MikroTik has announced a change in the way RoS development is organized. There will now be three different tracks of development:
Bugfix only – When a current build is released, only fixes to known bugs will be added to this branch of development
Current – Current release will contain bugfixes and new features
Release Candidate – The release candidate will remain the development build for the next “current” release.
Graphical Overview of the new development cycle
Image and notes below are from here
A small addendum: the Bugfix only will only contain verified fixes, and no new features. The Current release contains the same fixes but also new features and other improvements, sometimes also less critical fixes than in Bugfix. And finally the Release Candidate is more likely to a nightly build. We will not to intensive testing before publishing these, only quick check if upgrade can be done and if most features work fine.
Origin
The idea originally came out of this thread and after a flurry of positive commentary, it became a working practice shortly therafter.
We plan to make sub-version releases Continue reading
MikroTik CCR1072-1G-8S+ Review (Part 2) – BGP Performance
[adrotate banner=”4″]
Here is Part 1 of the CCR1072-1G-8S+ review in case you missed it!
CCR1072-1G-8S+ Ultimate BGP Performance test
After many days of testing, Part 2 is finally here! Welcome to the stubarea51.net BGP gauntlet. We subjected the CCR1072 to different types of network torture stress testing. Continuing on from our initial review, we chose BGP as the first way to test the limits and capacity of the CCR1072-1G-8S+.
Here is an overview of our lab environment to test the new CCR
- CCR1072-1G-8S+
- CCR1009-8G-1S-1S+
- CRS-125-24G-1S+
- x86 VMs on ESXi 6.0 for upstream BGP peering
- (2) ESXi 6.0 Hosts with 20 Gb (4×10) connectivity
- Multimode 10 gig SFP+ using 50/125 OM3 fiber
All RouterOS devices were loaded with the latest stable code (6.30.1 at the time of testing)
Network Design of the StubArea51 LAB setup for BGP testing
For this series of testing, we took our two ESXi 6.0 hosts and built a number of VMs using RouterOS and Ubuntu to supply the 1.21 Gigawatts 3.6 Million routes we would need to beat up on the CCR1072 for a few days. If you’re not familiar with the RIPE Routing Information Service Continue reading
MikroTik CCR1072-1G-8S+ Review (Part 2) – BGP Performance *** Preview ***
Part 2 is on the way!
It has been a busy week and we have been working hard on CCR1072 testing. The full review for BGP Performance is still a few days away, but here is a sneak preview of some of the testing in the IP ArchiTechs Area 51 Lab.
We put just under 1.7 Million routes across 4 BGP peers into the CCR1072-1G-8S+ and it was fully converged in 2 Minutes and 42 seconds.
Here is a screen shot of the CCR1072-1G-8S+ after converging with 4 full feeds.
MikroTik and Ubiquity routers being Hijacked by Dyre Malware?
[adrotate banner=”4″]
Came across several interesting articles that claim there is a change in the way Dyre aka Upatre malware is spreading. Dyre seems to be getting a lot of press as it is used in browser hijacks to compromise online banking credentials and other sensitive private data. However, most recently – instead of infecting hosts, it appears to be compromising routers as well. Blogger krebsonsecurity.com writes:
Recently, researchers at the Fujitsu Security Operations Center in Warrington, UK began tracking Upatre being served from hundreds of compromised home routers — particularly routers powered by MikroTik and Ubiquiti’s AirOS.
As I first started researching this, I was wondering how they determined the router itself is compromised and not a host that sits on a NAT behind the router. Certainly different devices leave telltale signs visible in an IP packet capture that help point towards the true origin of a packet, so it’s possible that something was discovered in that way. It’s also possible the router isn’t being compromised via the Internet, but rather on the LAN side as it would be much easier for malware to scan the private subnet it sits on and attempt to use well known Continue reading
MikroTik and Ubiquity routers being Hijacked by Dyre Malware?
[adrotate banner=”4″]
Came across several interesting articles that claim there is a change in the way Dyre aka Upatre malware is spreading. Dyre seems to be getting a lot of press as it is used in browser hijacks to compromise online banking credentials and other sensitive private data. However, most recently – instead of infecting hosts, it appears to be compromising routers as well. Blogger krebsonsecurity.com writes:
Recently, researchers at the Fujitsu Security Operations Center in Warrington, UK began tracking Upatre being served from hundreds of compromised home routers — particularly routers powered by MikroTik and Ubiquiti’s AirOS.
As I first started researching this, I was wondering how they determined the router itself is compromised and not a host that sits on a NAT behind the router. Certainly different devices leave telltale signs visible in an IP packet capture that help point towards the true origin of a packet, so it’s possible that something was discovered in that way. It’s also possible the router isn’t being compromised via the Internet, but rather on the LAN side as it would be much easier for malware to scan the private subnet it sits on and attempt to use well known Continue reading
MikroTik CCR1072-1G-8S+ Review (Part 1) – hardware, specs and design use cases
[adrotate banner=”4″]
07/25/2015 – Thanks to Normunds @ MikroTik for sending over photos of the production CCR-1072-1G-8S+ which have now been included in the slideshow
[metaslider id=52]
CCR-1072-1G-8S+ available soon @
http://www.roc-noc.com/mikrotik/routerboard/CCR1072-1G-8Splus.html
UPDATE 7/10/2015 – MikroTik officially lists the CCR1072
http://routerboard.com/CCR1072-1G-8Splus
NOTE: The pictures in this review are of a pre-production CCR1072. The CCR1072 that is shipping has some minor differences on the mainboard and the case. MikroTik is sending updated pictures and we will post those as soon as they come in!
StubArea51.net prepares for CCR1072 performance testing in the Flowood, MS lab
Well, the long wait is finally over. According to Tom over at www.roc-noc.com, the CCR1072 will start shipping in the next 2 weeks and we will be adding it to our development lab in Flowood, MS. We were fortunate enough to get a significant amount of time with the new flagship router down in Miami at the 2015 USA MUM thanks to MT. The arrival of the CCR1072 and 80 Gbps of throughput opens up new doors for MikroTik. The CCR1072 positions MikroTik to break into larger markets and enables competition against industry players like Cisco and Continue reading