VPP MPLS – Part 2

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

I’ve deployed an MPLS core for IPng Networks, which allows me to provide L2VPN services, and at the same time keep an IPng Site Local network with IPv4 and IPv6 that is separate from the internet, based on hardware/silicon based forwarding at line rate and high availability. You can read all about my Centec MPLS shenanigans in [this article].

In the last article, I explored VPP’s MPLS implementation a little bit. All the while, @vifino has been tinkering with the Linux Control Plane and adding MPLS support to it, and together we learned a lot about how VPP does MPLS forwarding and how it sometimes differs to other implementations. During the process, we talked a bit about implicit-null and explicit-null. When my buddy Fred read the [previous article], he also talked about a feature called penultimate-hop-popping which Continue reading

VPP MPLS – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

I’ve deployed an MPLS core for IPng Networks, which allows me to provide L2VPN services, and at the same time keep an IPng Site Local network with IPv4 and IPv6 that is separate from the internet, based on hardware/silicon based forwarding at line rate and high availability. You can read all about my Centec MPLS shenanigans in [this article].

Ever since the release of the Linux Control Plane [ref] plugin in VPP, folks have asked “What about MPLS?” – I have never really felt the need to go this rabbit hole, because I figured that in this day and age, higher level IP protocols that do tunneling are just as performant, and a little bit less of an ‘art’ to get right. For example, the Centec switches I deployed perform VxLAN, GENEVE and GRE all at Continue reading

VPP – Monitoring

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

I’ve been working on the Linux Control Plane [ref], which you can read all about in my series on VPP back in 2021:

• [Part 1]: Punting traffic through TUN/TAP interfaces into Linux
• [Part 2]: Mirroring VPP interface configuration into Linux
• [Part 3]: Automatically creating sub-interfaces in Linux
• [Part 4]: Synchronize link state, MTU and addresses to Linux
• [Part 5]: Netlink Listener, synchronizing state from Linux to VPP
• [Part 6]: Observability with LibreNMS and VPP SNMP Agent
• [Part 7]: Productionizing and reference Supermicro fleet at IPng

With this, I can make a regular server running Linux use VPP as kind of a software ASIC for super fast forwarding, filtering, NAT, and so on, while keeping control of the interface state (links, addresses and routes) itself. With Continue reading

Case Study: Let’s Encrypt DNS-01

Last week I shared how IPng Networks deployed a loadbalanced frontend cluster of NGINX webservers that have public IPv4 / IPv6 addresses, but talk to a bunch of internal webservers that are in a private network which isn’t directly connected to the internet, so called IPng Site Local [ref] with addresses 198.19.0.0/16 and 2001:678:d78:500::/56.

I wrote in [that article] that IPng will be using ACME HTTP-01 validation, which asks the certificate authority, in this case Let’s Encrypt, to contact the webserver on a well-known URI for each domain that I’m requesting a certificate for. Unsurprisingly, several folks reached out to me asking “well what about DNS-01”, and one sentence caught their eye:

Some SSL certificate providers allow for wildcards (ie. *.ipng.ch), but I’m going to keep it relatively simple and use [Let’s Encrypt] which offers free certificates with a validity of three months.

I could’ve seen this one coming! The sentence can be read to imply it doesn’t, but of course Let’s Encrypt offers wildcard certificates. It just doesn’t satisfy my relatively simple qualifier of the second part of the sentence … So here I go, down the Continue reading

Case Study: Site Local NGINX

A while ago I rolled out an important change to the IPng Networks design: I inserted a bunch of [Centec MPLS] and IPv4/IPv6 capable switches underneath [AS8298], which gave me two specific advantages:

1. The entire IPng network is now capable of delivering L2VPN services, taking the form of MPLS point-to-point ethernet, and VPLS, as shown in a previous [deep dive], in addition to IPv4 and IPv6 transit provided by VPP in an elaborate and elegant [BGP Routing Policy].

2. A new internal private network becomes available to any device connected IPng switches, with addressing in 198.19.0.0/16 and 2001:678:d78:500::/56. This network is completely isolated from the Internet, with access controlled via N+2 redundant gateways/firewalls, described in more detail in a previous [deep dive] as well.

Overview

After rolling out this spiffy BGP Free [MPLS Core], I wanted to take a look at maybe conserving a few IP addresses here and there, as well as tightening access and protecting the more important machines that IPng Networks runs. You see, most enterprise networks will include a bunch of internal services, like databases, network attached storage, backup servers, network monitoring, Continue reading

Case Study: Centec MPLS Core

After receiving an e-mail from [Starry Networks], I had a chat with their founder and learned that the combination of switch silicon and software may be a good match for IPng Networks.

I got pretty enthusiastic when this new vendor claimed VxLAN, GENEVE, MPLS and GRE at 56 ports and line rate, on a really affordable budget ($4’200,- for the 56 port; and $1’650,- for the 26 port switch). This reseller is using a less known silicon vendor called [Centec], who have a lineup of ethernet chipsets. In this device, the CTC8096 (GoldenGate) is used for cost effective high density 10GbE/40GbE applications paired with 4x100GbE uplink capability. This is Centec’s fourth generation, so CTC8096 inherits the feature set from L2/L3 switching to advanced data center and metro Ethernet features with innovative enhancement. The switch chip provides up to 96x10GbE ports, or 24x40GbE, or 80x10GbE + 4x100GbE ports, inheriting from its predecessors a variety of features, including L2, L3, MPLS, VXLAN, MPLS SR, and OAM/APS. Highlights features include Telemetry, Programmability, Security and traffic management, and Network time synchronization.

After discussing basic L2, L3 and Overlay functionality in my [first post], and explored the functionality and Continue reading

Case Study: VPP at Coloclue, part 2

• Author: Pim van Pelt, Rogier Krieger
• Reviewers: Coloclue Network Committee
• Status: Draft - Review - Published

Almost precisely two years ago, in February of 2021, I created a loadtesting environment at [Coloclue] to prove that a provider of L2 connectivity between two datacenters in Amsterdam was not incurring jitter or loss on its services – I wrote up my findings in [an article], which demonstrated that the service provider indeed provides a perfect service. One month later, in March 2021, I briefly ran [VPP] on one of the routers at Coloclue, but due to lack of time and a few technical hurdles along the way, I had to roll back [ref].

The Problem

Over the years, Coloclue AS8283 continues to suffer from packet loss in its network. Taking a look at a simple traceroute, in this case from IPng AS8298, shows very high variance and packetlo when entering the network (at hop 5 in a router called eunetworks-2.router.nl.coloclue.net):

                                       My traceroute  [v0.94]                
squanchy.ipng.ch (194.1.193.90) -> 185.52.227.1                           2023-02-24T09:03:36+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                          Packets               Pings
  Continue reading

Review: Compulab Fitlet2

A while ago, in June 2021, we were discussing home routers that can keep up with 1G+ internet connections in the CommunityRack telegram channel. Of course at IPng Networks we are fond of the Supermicro Xeon D1518 [ref], which has a bunch of 10Gbit X522 and 1Gbit i350 and i210 intel NICs, but it does come at a certain price.

For smaller applications, PC Engines APU6 [ref] is kind of cool and definitely more affordable. But, in this chat, Patrick offered an alternative, the [Fitlet2] which is a small, passively cooled, and expandable IoT-esque machine.

Fast forward 18 months, and Patrick decided to sell off his units, so I bought one off of him, and decided to loadtest it. Considering the pricetag (the unit I will be testing will ship for around $400), and has the ability to use (1G/SFP) fiber optics, it may be a pretty cool one!

Executive Summary

TL/DR: Definitely a cool VPP router, 3x 1Gbit line rate, A- would buy again

With some care on the VPP configuration (notably RX/TX descriptors), this unit can handle L2XC at (almost) line rate in both directions (2.94Mpps out a theoretical 2.97Mpps), Continue reading

Review: S5648X-2Q4Z Switch – Part 2: MPLS

After receiving an e-mail from a newer [China based OEM], I had a chat with their founder and learned that the combination of switch silicon and software may be a good match for IPng Networks.

I got pretty enthusiastic when this new vendor claimed VxLAN, GENEVE, MPLS and GRE at 56 ports and line rate, on a really affordable budget ($4’200,- for the 56 port; and $1’650,- for the 26 port switch). This reseller is using a less known silicon vendor called [Centec], who have a lineup of ethernet silicon. In this device, the CTC8096 (GoldenGate) is used for cost effective high density 10GbE/40GbE applications paired with 4x100GbE uplink capability. This is Centec’s fourth generation, so CTC8096 inherits the feature set from L2/L3 switching to advanced data center and metro Ethernet features with innovative enhancement. The switch chip provides up to 96x10GbE ports, or 24x40GbE, or 80x10GbE + 4x100GbE ports, inheriting from its predecessors a variety of features, including L2, L3, MPLS, VXLAN, MPLS SR, and OAM/APS. Highlights features include Telemetry, Programmability, Security and traffic management, and Network time synchronization.

After discussing basic L2, L3 and Overlay functionality in my [previous post], I left Continue reading

Review: S5648X-2Q4Z Switch – Part 1: VxLAN/GENEVE/NvGRE

After receiving an e-mail from a newer [China based switch OEM], I had a chat with their founder and learned that the combination of switch silicon and software may be a good match for IPng Networks. You may recall my previous endeavors in the Fiberstore lineup, notably an in-depth review of the [S5860-20SQ] which sports 20x10G, 4x25G and 2x40G optics, and its larger sibling the S5860-48SC which comes with 48x10G and 8x100G cages. I use them in production at IPng Networks and their featureset versus price point is pretty good. In that article, I made one critical note reviewing those FS switches, in that they’e be a better fit if they allowed for MPLS or IP based L2VPN services in hardware.

I got cautiously enthusiastic (albeit suitably skeptical) when this new vendor claimed VxLAN, GENEVE, MPLS and GRE at 56 ports and line rate, on a really affordable budget (sub-$4K for the 56 port; and sub-$2K for the 26 port switch). This reseller is using a less known silicon vendor called [Centec], who have a lineup of ethernet silicon. In this device, the CTC8096 (GoldenGate) is used for cost effective high density 10GbE/40GbE applications Continue reading

Mastodon – Part 3 – statsd and Prometheus

About this series

I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for me it’s time to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.

In my [first post], I shared some thoughts on how I installed a Mastodon instance for myself. In a [followup post] I talked about its overall architecture and how one might use Prometheus to monitor vital backends like Redis, Postgres and Elastic. But Mastodon itself is also an application which can provide a wealth of telemetry using a protocol called [StatsD].

In this post, I’ll show how I tie these all together in a custom Grafana Mastodon dashboard!

Mastodon Statistics

I noticed in the [Mastodon docs], that there’s a one-liner breadcrumb that might be easy to overlook, Continue reading

Mastodon – Part 2 – Monitoring

About this series

I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for me it’s time to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.

In the [previous post], I shared some thoughts on how the overall install of a Mastodon instance went, making it a point to ensure my users’ (and my own!) data is somehow safe, and the machine runs on good hardware, and with good connectivity. Thanks IPng, for that 10G connection! In this post, I visit an old friend, [Borgmon], which has since reincarnated and become the de facto open source observability and signals ecosystem, and its incomparably awesome friend. Hello, Prometheus and Grafana!

Anatomy of Mastodon

Looking more closely at the architecture of Mastodon, it consists of a few moving Continue reading

Mastodon – Part 1 – Installing

About this series

I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for me it’s time to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.

This series details my findings starting a micro blogging website, which uses a new set of super interesting open interconnect protocols to share media (text, pictures, videos, etc) between producers and their followers, using an open source project called Mastodon.

Introduction

Similar to how blogging is the act of publishing updates to a website, microblogging is the act of publishing small updates to a stream of updates on your profile. You can publish text posts and optionally attach media such as pictures, audio, video, or polls. Mastodon lets you follow friends and discover new ones. It doesn’t do this in a centralized way, however.

Groups Continue reading

VPP Lab – Setup

Introduction

In a previous post (VPP Linux CP - Virtual Machine Playground), I wrote a bit about building a QEMU image so that folks can play with the Vector Packet Processor and the Linux Control Plane code. Judging by our access logs, this image has definitely been downloaded a bunch, and I myself use it regularly when I want to tinker a little bit, without wanting to impact the production routers at AS8298.

The topology of my tests has become a bit more complicated over time, and often just one router would not be enough. Yet, repeatability is quite important, and I found myself constantly reinstalling / recheckpointing the vpp-proto virtual machine I was using. I got my hands on some LAB hardware, so it’s time for an upgrade!

IPng Networks LAB - Physical

First, I specc’d out a few machines that will serve as hypervisors. From top to bottom in the picture here, two FS.com S5680-20SQ switches – I reviewed these earlier [ref], and I really like these, as they come with 20x10G, 4x25G and 2x40G ports, an OOB management port and serial to configure them. Under it, is its larger brother, with 48x10G Continue reading

VPP Configuration – Part2

About this series

I use VPP - Vector Packet Processor - extensively at IPng Networks. Earlier this year, the VPP community merged the Linux Control Plane plugin. I wrote about its deployment to both regular servers like the Supermicro routers that run on our AS8298, as well as virtual machines running in KVM/Qemu.

Now that I’ve been running VPP in production for about half a year, I can’t help but notice one specific drawback: VPP is a programmable dataplane, and by design it does not include any configuration or controlplane management stack. It’s meant to be integrated into a full stack by operators. For end-users, this unfortunately means that typing on the CLI won’t persist any configuration, and if VPP is restarted, it will not pick up where it left off. There’s one developer convenience in the form of the exec command-line (and startup.conf!) option, which will read a file and apply the contents to the CLI line by line. However, if any typo is made in the file, processing immediately stops. It’s meant as a convenience for VPP developers, and is certainly not a useful configuration method for all but the simplest topologies.

Luckily, VPP comes Continue reading

VPP Configuration – Part1

About this series

I use VPP - Vector Packet Processor - extensively at IPng Networks. Earlier this year, the VPP community merged the Linux Control Plane plugin. I wrote about its deployment to both regular servers like the Supermicro routers that run on our AS8298, as well as virtual machines running in KVM/Qemu.

Now that I’ve been running VPP in production for about half a year, I can’t help but notice one specific drawback: VPP is a programmable dataplane, and by design it does not include any configuration or controlplane management stack. It’s meant to be integrated into a full stack by operators. For end-users, this unfortunately means that typing on the CLI won’t persist any configuration, and if VPP is restarted, it will not pick up where it left off. There’s one developer convenience in the form of the exec command-line (and startup.conf!) option, which will read a file and apply the contents to the CLI line by line. However, if any typo is made in the file, processing immediately stops. It’s meant as a convenience for VPP developers, and is certainly not a useful configuration method for all but the simplest topologies.

Luckily, VPP comes Continue reading

Syslog to Telegram

Introduction

From time to time, I wish I could be made aware of failures earlier. There are two events, in particular, that I am interested to know about very quickly, as they may impact service at AS8298:

1. Open Shortest Path First (OSPF) adjacency removals. OSPF is a link-state protocol and it knows when a physical link goes down, that the peer (neighbor) is no longer reachable. It can then recompute paths to other routers fairly quickly. But if the link stays up but connectivity is interrupted, for example because there is a switch in the path, it can take a relatively long time to detect.
2. Bidirectional Forwarding Detection (BFD) session timeouts. BFD sets up a rapid (for example every 50ms or 20Hz) of a unidirectional UDP stream between two hosts. If a number of packets (for example 40 packets or 2 seconds) are not received, a link can be assumed to be dead.

Notably, BIRD, as many other vendors do, can combine the two. At IPng, each OSPF adjacency is protected by BFD. What happens is that once an OSPF enabled link comes up, OSPF Hello packets will be periodically transmitted (with a period called called a Hello Timer Continue reading

IPng Networks – Colocation

Introduction

As with most companies, it started with an opportunity. I got my hands on a location which has a raised floor at 60m2 and a significant power connection of 3x200A, and a metro fiber connection at 10Gbps. I asked my buddy Luuk ‘what would it take to turn this into a colo?’ and the rest is history. Thanks to Daedalean AG who benefit from this infrastructure as well, making this first small colocation site was not only interesting, but also very rewarding.

The colocation business is murder in Zurich - there are several very large datacenters (Equinix, NTT, Colozüri, Interxion) all directly in or around the city, and I’m known to dwell in most of these. The networking and service provider industry is quite small and well organized into Network Operator Groups, so I work under the assumption that everybody knows everybody. I definitely like to pitch in and share what I have built, both the physical bits but also the narrative.

This article describes the small serverroom I built at a partner’s premises in Zurich Albisrieden. The colo is open for business, that is to say: Please feel free to reach out if you’re interested.

Continue reading