VPP with loopback-only OSPFv3

Introduction

A few weeks ago I took a good look at the [Babel] protocol. I found a set of features there that I really appreciated. The first was a latency aware routing protocol - this is useful for mesh (wireless) networks but it is also a good fit for IPng’s usecase, notably because it makes use of carrier ethernet which, if any link in the underlying MPLS network fails, will automatically re-route but sometimes with much higher latency. In these cases, Babel can reconverge on its own to a topology that has the lowest end to end latency.

But a second really cool find, is that Babel can use IPv6 nexthops for IPv4 destinations - which is super useful because it will allow me to retire all of the IPv4 /31 point to point networks between my routers. AS8298 has about half of a /24 tied up in these otherwise pointless (pun intended) transit networks.

In the same week, my buddy Benoit asked a question about OSPFv3 on the Bird users mailinglist [ref] which may or may not have been because I had been messing around with Babel using only IPv4 loopback interfaces. And just a Continue reading

VPP with Babel – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. Thanks to the [Linux ControlPlane] plugin, higher level control plane software becomes available, that is to say: things like BGP, OSPF, LDP, VRRP and so on become quite natural for VPP.

IPng Networks is a small service provider that has built a network based entirely on open source: [Debian] servers with widely available Intel and Mellanox 10G/25G/100G network cards, paired with [VPP] for the dataplane, and [Bird2] for the controlplane.

As a small provider, I am well aware of the cost of IPv4 address space. Long gone are the times at which an initial allocation was a /19, and subsequent allocations usually a /20 based on justification. Then it watered down to a /22 for new Local Internet Registries, then that became a /24 for new LIRs, and ultimately we ran Continue reading

VPP on FreeBSD – Part 2

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. Over the years, folks have asked me regularly “What about BSD?” and to my surprise, late last year I read an announcement from the FreeBSD Foundation [ref] as they looked back over 2023 and forward to 2024:

Porting the Vector Packet Processor to FreeBSD

Vector Packet Processing (VPP) is an open-source, high-performance user space networking stack that provides fast packet processing suitable for software-defined networking and network function virtualization applications. VPP aims to optimize packet processing through vectorized operations and parallelism, making it well-suited for high-speed networking applications. In November of this year, the Foundation began a contract with Tom Jones, a FreeBSD developer specializing in network performance, to port VPP to FreeBSD. Under the contract, Tom will also allocate time for other tasks such as testing FreeBSD on common virtualization platforms to improve the desktop experience, improving Continue reading

VPP on FreeBSD – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. Over the years, folks have asked me regularly “What about BSD?” and to my surprise, late last year I read an announcement from the FreeBSD Foundation [ref] as they looked back over 2023 and forward to 2024:

Porting the Vector Packet Processor to FreeBSD

Vector Packet Processing (VPP) is an open-source, high-performance user space networking stack that provides fast packet processing suitable for software-defined networking and network function virtualization applications. VPP aims to optimize packet processing through vectorized operations and parallelism, making it well-suited for high-speed networking applications. In November of this year, the Foundation began a contract with Tom Jones, a FreeBSD developer specializing in network performance, to port VPP to FreeBSD. Under the contract, Tom will also allocate time for other tasks such as testing FreeBSD on common virtualization platforms to improve the desktop experience, improving Continue reading

VPP Python API

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

You’ll hear me talk about VPP being API centric, with no configuration persistence, and that’s by design. However, there is this also a CLI utility called vppctl, right, so what gives? In truth, the CLI is used a lot by folks to configure their dataplane, but it really was always meant to be a debug utility. There’s a whole wealth of programmability that is not exposed via the CLI at all, and the VPP community develops and maintains an elaborate set of tools to allow external programs to (re)configure the dataplane. One such tool is my own [vppcfg] which takes a YAML specification that describes the dataplane configuration, and applies it safely to a running VPP instance.

Introduction

In case you’re interested in writing your own automation, this article is for you! I’ll provide a deep dive into Continue reading

Debian on IPng’s VPP Routers

Introduction

When IPng Networks first built out a european network, I was running the Disaggregated Network Operating System [ref], initially based on AT&T’s “dNOS” software framework. Over time though, the DANOS project slowed down, and the developers with whom I had a pretty good relationship all left for greener pastures.

In 2019, Pierre Pfister (and several others) built a VPP router sandbox [ref], which graduated into a feature called the Linux Control Plane plugin [ref]. Lots of folks put in an effort for the Linux Control Plane, notably Neale Ranns from Cisco (these days Graphiant), and Matt Smith and Jon Loeliger from Netgate (who ship this as TNSR [ref], check it out!). I helped as well, by adding a bunch of Netlink handling and VPP->Linux synchronization code, which I’ve written about a bunch on this blog in the 2021 VPP development series [ref].

At the time, Ubuntu and CentOS were the supported platforms, so I installed a bunch of Ubuntu machines when doing the deploy with my buddy Fred from IP-Max [ref]. But as time went by, I fell back to my old habit of running Debian Continue reading

Debian on Mellanox SN2700 (32x100G)

Introduction

I’m still hunting for a set of machines with which I can generate 1Tbps and 1Gpps of VPP traffic, and considering a 100G network interface can do at most 148.8Mpps, I will need 7 or 8 of these network cards. Doing a loadtest like this with DACs back-to-back is definitely possible, but it’s a bit more convenient to connect them all to a switch. However, for this to work I would need (at least) fourteen or more HundredGigabitEthernet ports, and these switches tend to get expensive, real quick.

Or do they?

Hardware

I thought I’d ask the #nlnog IRC channel for advice, and of course the usual suspects came past, such as Juniper, Arista, and Cisco. But somebody mentioned “How about Mellanox, like SN2700?” and I remembered my buddy Eric was a fan of those switches. I looked them up on the refurbished market and I found one for EUR 1’400,- for 32x100G which felt suspiciously low priced… but I thought YOLO and I ordered it. It arrived a few days later via UPS from Denmark to Switzerland.

The switch specs are pretty impressive, with 32x100G QSFP28 ports, which can be broken out to a set of Continue reading

VPP IXP Gateway – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

There’s some really fantastic features in VPP, some of which are lesser well known, and not always very well documented. In this article, I will describe a unique usecase in which I think VPP will excel, notably acting as a gateway for Internet Exchange Points.

In this first article, I’ll take a closer look at three things that would make such a gateway possible: bridge domains, MAC address filtering and traffic shaping.

Introduction

Internet Exchanges are typically L2 (ethernet) switch platforms that allow their connected members to exchange traffic amongst themselves. Not all members share physical locations with the Internet Exchange itself, for example the IXP may be at NTT Zurich, but the member may be present in Interxion Zurich. For smaller clubs, like IPng Networks, it’s not always financially feasible (or desirable) to order a dark fiber between two adjacent Continue reading

Case Study: NGINX + Certbot with Ansible

About this series

In the distant past (to be precise, in November of 2009) I wrote a little piece of automation together with my buddy Paul, called PaPHosting. The goal was to be able to configure common attributes like servername, config files, webserver and DNS configs in a consistent way, tracked in Subversion. By the way despite this project deriving its name from the first two authors, our mutual buddy Jeroen also started using it, and has written lots of additional cool stuff in the repo, as well as helped to move from Subversion to Git a few years ago.

Michael DeHaan [ref] founded Ansible in 2012, and by then our little PaPHosting project, which was written as a set of bash scripts, had sufficiently solved our automation needs. But, as is the case with most home-grown systems, over time I kept on seeing more and more interesting features and integrations emerge, solid documentation, large user group, and eventually I had to reconsider our 1.5K LOC of Bash and ~16.5K files under maintenance, and in the end, I settled on Ansible.

commit c986260040df5a9bf24bef6bfc28e1f3fa4392ed
Author: Pim van Pelt <[email protected]>
Date:   Thu Nov 26 23:13:21 2009 +0000

 Continue reading

Pixelfed – Part 1 – Installing

About this series

I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for me it’s time to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.

After having written a fair bit about my Mastodon [install] and [monitoring], I’ve been using it every day. This morning, my buddy Ramón asked if he could make a second account on ublog.tech for his Campervan Adventures, and notably to post pics of where he and his family went.

But if pics is your jam, why not … [Pixelfed]!

Introduction

Similar to how blogging is the act of publishing updates to a website, microblogging is the act of publishing small updates to a stream of updates on your profile. Very similar to the relationship between Facebook and Continue reading

VPP MPLS – Part 4

About this series

Special Thanks: Adrian vifino Pistol for writing this code and for the wonderful collaboration!

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

In the last three articles, I thought I had described “all we need to know” to perform MPLS using the Linux Controlplane in VPP:

1. In the [first article] of this series, I took a look at MPLS in general.
2. In the [second article] of the series, I demonstrated a few special case labels (such as Explicit Null and Implicit Null which enables the fabled Penultimate Hop Popping behavior of MPLS.
3. Then, in the [third article], I worked with @vifino to implement the plumbing for MPLS in the Linux Control Plane plugin for VPP. He did most of the work, I just watched :)

As if in a state of premonition, I mentioned:

Caveat empor, outside of a modest functional and Continue reading

VPP MPLS – Part 3

About this series

Special Thanks: Adrian vifino Pistol for writing this code and for the wonderful collaboration!

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

In the [first article] of this series, I took a look at MPLS in general, and how setting up static Label Switched Paths can be done in VPP. A few details on special case labels (such as Implicit Null which enabled the fabled Penultimate Hop Popping) were missing, so I took a good look at them in the [second article] of the series.

This was all just good fun but also allowed me to buy some time for @vifino who has been implementing MPLS handling within the Linux Control Plane plugin for VPP! This final article in the series shows the engineering considerations that went in to writing the plugin, which is currently under review but reasonably complete. Considering the VPP Continue reading

VPP MPLS – Part 2

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

I’ve deployed an MPLS core for IPng Networks, which allows me to provide L2VPN services, and at the same time keep an IPng Site Local network with IPv4 and IPv6 that is separate from the internet, based on hardware/silicon based forwarding at line rate and high availability. You can read all about my Centec MPLS shenanigans in [this article].

In the last article, I explored VPP’s MPLS implementation a little bit. All the while, @vifino has been tinkering with the Linux Control Plane and adding MPLS support to it, and together we learned a lot about how VPP does MPLS forwarding and how it sometimes differs to other implementations. During the process, we talked a bit about implicit-null and explicit-null. When my buddy Fred read the [previous article], he also talked about a feature called penultimate-hop-popping which Continue reading

VPP MPLS – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

I’ve deployed an MPLS core for IPng Networks, which allows me to provide L2VPN services, and at the same time keep an IPng Site Local network with IPv4 and IPv6 that is separate from the internet, based on hardware/silicon based forwarding at line rate and high availability. You can read all about my Centec MPLS shenanigans in [this article].

Ever since the release of the Linux Control Plane [ref] plugin in VPP, folks have asked “What about MPLS?” – I have never really felt the need to go this rabbit hole, because I figured that in this day and age, higher level IP protocols that do tunneling are just as performant, and a little bit less of an ‘art’ to get right. For example, the Centec switches I deployed perform VxLAN, GENEVE and GRE all at Continue reading

VPP – Monitoring

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

I’ve been working on the Linux Control Plane [ref], which you can read all about in my series on VPP back in 2021:

• [Part 1]: Punting traffic through TUN/TAP interfaces into Linux
• [Part 2]: Mirroring VPP interface configuration into Linux
• [Part 3]: Automatically creating sub-interfaces in Linux
• [Part 4]: Synchronize link state, MTU and addresses to Linux
• [Part 5]: Netlink Listener, synchronizing state from Linux to VPP
• [Part 6]: Observability with LibreNMS and VPP SNMP Agent
• [Part 7]: Productionizing and reference Supermicro fleet at IPng

With this, I can make a regular server running Linux use VPP as kind of a software ASIC for super fast forwarding, filtering, NAT, and so on, while keeping control of the interface state (links, addresses and routes) itself. With Continue reading

Case Study: Let’s Encrypt DNS-01

Last week I shared how IPng Networks deployed a loadbalanced frontend cluster of NGINX webservers that have public IPv4 / IPv6 addresses, but talk to a bunch of internal webservers that are in a private network which isn’t directly connected to the internet, so called IPng Site Local [ref] with addresses 198.19.0.0/16 and 2001:678:d78:500::/56.

I wrote in [that article] that IPng will be using ACME HTTP-01 validation, which asks the certificate authority, in this case Let’s Encrypt, to contact the webserver on a well-known URI for each domain that I’m requesting a certificate for. Unsurprisingly, several folks reached out to me asking “well what about DNS-01”, and one sentence caught their eye:

Some SSL certificate providers allow for wildcards (ie. *.ipng.ch), but I’m going to keep it relatively simple and use [Let’s Encrypt] which offers free certificates with a validity of three months.

I could’ve seen this one coming! The sentence can be read to imply it doesn’t, but of course Let’s Encrypt offers wildcard certificates. It just doesn’t satisfy my relatively simple qualifier of the second part of the sentence … So here I go, down the Continue reading

Case Study: Site Local NGINX

A while ago I rolled out an important change to the IPng Networks design: I inserted a bunch of [Centec MPLS] and IPv4/IPv6 capable switches underneath [AS8298], which gave me two specific advantages:

1. The entire IPng network is now capable of delivering L2VPN services, taking the form of MPLS point-to-point ethernet, and VPLS, as shown in a previous [deep dive], in addition to IPv4 and IPv6 transit provided by VPP in an elaborate and elegant [BGP Routing Policy].

2. A new internal private network becomes available to any device connected IPng switches, with addressing in 198.19.0.0/16 and 2001:678:d78:500::/56. This network is completely isolated from the Internet, with access controlled via N+2 redundant gateways/firewalls, described in more detail in a previous [deep dive] as well.

Overview

After rolling out this spiffy BGP Free [MPLS Core], I wanted to take a look at maybe conserving a few IP addresses here and there, as well as tightening access and protecting the more important machines that IPng Networks runs. You see, most enterprise networks will include a bunch of internal services, like databases, network attached storage, backup servers, network monitoring, Continue reading

Case Study: Centec MPLS Core

After receiving an e-mail from [Starry Networks], I had a chat with their founder and learned that the combination of switch silicon and software may be a good match for IPng Networks.

I got pretty enthusiastic when this new vendor claimed VxLAN, GENEVE, MPLS and GRE at 56 ports and line rate, on a really affordable budget ($4’200,- for the 56 port; and $1’650,- for the 26 port switch). This reseller is using a less known silicon vendor called [Centec], who have a lineup of ethernet chipsets. In this device, the CTC8096 (GoldenGate) is used for cost effective high density 10GbE/40GbE applications paired with 4x100GbE uplink capability. This is Centec’s fourth generation, so CTC8096 inherits the feature set from L2/L3 switching to advanced data center and metro Ethernet features with innovative enhancement. The switch chip provides up to 96x10GbE ports, or 24x40GbE, or 80x10GbE + 4x100GbE ports, inheriting from its predecessors a variety of features, including L2, L3, MPLS, VXLAN, MPLS SR, and OAM/APS. Highlights features include Telemetry, Programmability, Security and traffic management, and Network time synchronization.

After discussing basic L2, L3 and Overlay functionality in my [first post], and explored the functionality and Continue reading

Case Study: VPP at Coloclue, part 2

• Author: Pim van Pelt, Rogier Krieger
• Reviewers: Coloclue Network Committee
• Status: Draft - Review - Published

Almost precisely two years ago, in February of 2021, I created a loadtesting environment at [Coloclue] to prove that a provider of L2 connectivity between two datacenters in Amsterdam was not incurring jitter or loss on its services – I wrote up my findings in [an article], which demonstrated that the service provider indeed provides a perfect service. One month later, in March 2021, I briefly ran [VPP] on one of the routers at Coloclue, but due to lack of time and a few technical hurdles along the way, I had to roll back [ref].

The Problem

Over the years, Coloclue AS8283 continues to suffer from packet loss in its network. Taking a look at a simple traceroute, in this case from IPng AS8298, shows very high variance and packetlo when entering the network (at hop 5 in a router called eunetworks-2.router.nl.coloclue.net):

                                       My traceroute  [v0.94]                
squanchy.ipng.ch (194.1.193.90) -> 185.52.227.1                           2023-02-24T09:03:36+0100
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                          Packets               Pings
  Continue reading

Review: Compulab Fitlet2

A while ago, in June 2021, we were discussing home routers that can keep up with 1G+ internet connections in the CommunityRack telegram channel. Of course at IPng Networks we are fond of the Supermicro Xeon D1518 [ref], which has a bunch of 10Gbit X522 and 1Gbit i350 and i210 intel NICs, but it does come at a certain price.

For smaller applications, PC Engines APU6 [ref] is kind of cool and definitely more affordable. But, in this chat, Patrick offered an alternative, the [Fitlet2] which is a small, passively cooled, and expandable IoT-esque machine.

Fast forward 18 months, and Patrick decided to sell off his units, so I bought one off of him, and decided to loadtest it. Considering the pricetag (the unit I will be testing will ship for around $400), and has the ability to use (1G/SFP) fiber optics, it may be a pretty cool one!

Executive Summary

TL/DR: Definitely a cool VPP router, 3x 1Gbit line rate, A- would buy again

With some care on the VPP configuration (notably RX/TX descriptors), this unit can handle L2XC at (almost) line rate in both directions (2.94Mpps out a theoretical 2.97Mpps), Continue reading