Review: R86S (Jasper Lake – N6005)

Introduction

I am always interested in finding new hardware that is capable of running VPP. Of course, a standard issue 19” rack mountable machine like a Dell, HPE or SuperMicro machine is an obvious choice. They come with redundant power supplies, PCIe v3.0 or better expansion slots, and can boot off of mSATA or NVME, with plenty of RAM. But for some people and in some locations, the power envelope or size/cost of these 19” rack mountable machines can be prohibitive. Sometimes, just having a smaller form factor can be very useful:
Enter the GoWin R86S!

I stumbled across this lesser known build from GoWin, which is an ultra compact but modern design, featuring three 2.5GbE ethernet ports and optionally two 10GbE, or as I’ll show here, two 25GbE ports. What I really liked about the machine is that it comes with 32GB of LPDDR4 memory and can boot off of an m.2 NVME – which makes it immediately an appealing device to put in the field. I noticed that the height of the machine is just a few millimeters smaller than 1U which is 1.75” (44.5mm), which gives me the bright idea to 3D Continue reading

Case Study: IPng at Coloclue

I have been a member of the Coloclue association in Amsterdam for a long time. This is a networking association in the social and technical sense of the word. [Coloclue] is based in Amsterdam with members throughout the Netherlands and Europe. Its goals are to facilitate learning about and operating IP based networks and services. It has about 225 members who, together, have built this network and deployed about 135 servers across 8 racks in 3 datacenters (Qupra, EUNetworks and NIKHEF). Coloclue is operating [AS8283] across several local and international internet exchange points.

A small while ago, one of our members, Sebas, shared their setup with the membership. It generated a bit of a show-and-tell response, with Sebas and other folks on our mailinglist curious as to how we all deployed our stuff. My buddy Tim pinged me on Telegram: “This is something you should share for IPng as well!”, so this article is a bit different than my usual dabbles. It will be more of a show and tell: how did I deploy and configure the Amsterdam Chapter of IPng Networks?

I’ll make this article a bit more picture-dense, to show the look-and-feel of Continue reading

VPP with loopback-only OSPFv3 – Part 2

Introduction

When I first built IPng Networks AS8298, I decided to use OSPF as an IPv4 and IPv6 internal gateway protocol. Back in March I took a look at two slightly different ways of doing this for IPng, notably against a backdrop of conserving IPv4 addresses. As the network grows, the little point to point transit networks between routers really start adding up.

I explored two potential solutions to this problem:

1. [Babel] can use IPv6 nexthops for IPv4 destinations - which is super useful because it would allow me to retire all of the IPv4 /31 point to point networks between my routers.
2. [OSPFv3] makes it difficult to use IPv6 nexthops for IPv4 destinations, but in a discussion with the Bird Users mailinglist, we found a way: by reusing a single IPv4 loopback address on adjacent interfaces

In May I ran a modest set of two canaries, one between the two routers in my house (chbtl0 and chbtl1), and another between a router at the Daedalean colocation and Interxion datacenters (ddln0 and chgtg0). AS8298 has about quarter of a /24 tied up in these otherwise pointless point-to-point transit networks (see what Continue reading

VPP with loopback-only OSPFv3 – Part 2

Introduction

When I first built IPng Networks AS8298, I decided to use OSPF as an IPv4 and IPv6 internal gateway protocol. Back in March I took a look at two slightly different ways of doing this for IPng, notably against a backdrop of conserving IPv4 addresses. As the network grows, the little point to point transit networks between routers really start adding up.

I explored two potential solutions to this problem:

1. [Babel] can use IPv6 nexthops for IPv4 destinations - which is super useful because it would allow me to retire all of the IPv4 /31 point to point networks between my routers.
2. [OSPFv3] makes it difficult to use IPv6 nexthops for IPv4 destinations, but in a discussion with the Bird Users mailinglist, we found a way: by reusing a single IPv4 loopback address on adjacent interfaces

In May I ran a modest set of two canaries, one between the two routers in my house (chbtl0 and chbtl1), and another between a router at the Daedalean colocation and Interxion datacenters (ddln0 and chgtg0). AS8298 has about quarter of a /24 tied up in these otherwise pointless point-to-point transit networks (see what Continue reading

Case Study: NAT64

Introduction

IPng’s network is built up in two main layers, (1) an MPLS transport layer, which is disconnected from the Internet, and (2) a VPP overlay, which carries the Internet. I created a BGP Free core transport network, which uses MPLS switches from a company called Centec. These switches offer IPv4, IPv6, VxLAN, GENEVE and GRE all in silicon, are very cheap on power and relatively affordable per port.

Centec switches allow for a modest but not huge amount of routes in the hardware forwarding tables. I loadtested them in [a previous article] at line rate (well, at least 8x10G at 64b packets and around 110Mpps), and they forward IPv4, IPv6 and MPLS traffic effortlessly, at 45 watts.

I wrote more about the Centec switches in [my review] of them back in 2022.

IPng Site Local

I leverage this internal transport network for more than just MPLS. The transport switches are perfectly capable of line rate (at 100G+) IPv4 and IPv6 forwarding as well. When designing IPng Site Local, I created a number plan that assigns IPv4 from the 198.19.0.0/16 prefix, and IPv6 from the 2001:678:d78:500::/56 prefix. Within these, I allocate blocks for Continue reading

Case Study: IPng’s mail servers

Intro

I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for the last few years, I’ve been more and more inclined to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.

First off - I love Google’s Workspace products. I started using GMail just after it launched, back in 2004. Its user interface is sleek, performant, and very intuitive. Its filtering, granted, could be a bit less … robotic, but that’s made up by labels and an incredibly comprehensive search function. I would dare say that between GMail and Photos, those are my absolute favorite products on the internet.

That said, I have been running e-mail servers since well before Google existed as a company. I started off at M.C.G.V. Stack, the computer club of the University of Continue reading

FreeIX – Remote

Introduction

Tier1 and aspiring Tier2 providers interconnect only in large metropolitan areas, due to commercial incentives and politics. They won’t often peer with smaller providers, because why peer with a potential customer? Due to this, it’s entirely likely that traffic between two parties in Thessaloniki is sent to Frankfurt or Milan and back.

One possible antidote to this is to connect to a local Internet Exchange point. Not all ISPs have access to large metropolitan datacenters where larger internet exchanges have a point of presence, and it doesn’t help that the datacenter operator is happy to charge a substantial amount of money each month, just for the privilege of having a passive fiber cross connect to the exchange. Many Internet Exchanges these days ask for per-month port costs and meter the traffic with policers and rate limiters, such that the total cost of peering starts to exceed what one might pay for transit, especially at low volumes, which further exacerbates the problem. Bah.

This is an unfortunate market effect (the race to the bottom), where transit providers are continuously lowering their prices to compete. And while transit providers can make up to some extent due to economies of scale, at Continue reading

VPP with loopback-only OSPFv3 – Part 1

Introduction

A few weeks ago I took a good look at the [Babel] protocol. I found a set of features there that I really appreciated. The first was a latency aware routing protocol - this is useful for mesh (wireless) networks but it is also a good fit for IPng’s usecase, notably because it makes use of carrier ethernet which, if any link in the underlying MPLS network fails, will automatically re-route but sometimes with much higher latency. In these cases, Babel can reconverge on its own to a topology that has the lowest end to end latency.

But a second really cool find, is that Babel can use IPv6 nexthops for IPv4 destinations - which is super useful because it will allow me to retire all of the IPv4 /31 point to point networks between my routers. AS8298 has about half of a /24 tied up in these otherwise pointless (pun intended) transit networks.

In the same week, my buddy Benoit asked a question about OSPFv3 on the Bird users mailinglist [ref] which may or may not have been because I had been messing around with Babel using only IPv4 loopback interfaces. And just a Continue reading

VPP with Babel – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. Thanks to the [Linux ControlPlane] plugin, higher level control plane software becomes available, that is to say: things like BGP, OSPF, LDP, VRRP and so on become quite natural for VPP.

IPng Networks is a small service provider that has built a network based entirely on open source: [Debian] servers with widely available Intel and Mellanox 10G/25G/100G network cards, paired with [VPP] for the dataplane, and [Bird2] for the controlplane.

As a small provider, I am well aware of the cost of IPv4 address space. Long gone are the times at which an initial allocation was a /19, and subsequent allocations usually a /20 based on justification. Then it watered down to a /22 for new Local Internet Registries, then that became a /24 for new LIRs, and ultimately we ran Continue reading

VPP on FreeBSD – Part 2

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. Over the years, folks have asked me regularly “What about BSD?” and to my surprise, late last year I read an announcement from the FreeBSD Foundation [ref] as they looked back over 2023 and forward to 2024:

Porting the Vector Packet Processor to FreeBSD

Vector Packet Processing (VPP) is an open-source, high-performance user space networking stack that provides fast packet processing suitable for software-defined networking and network function virtualization applications. VPP aims to optimize packet processing through vectorized operations and parallelism, making it well-suited for high-speed networking applications. In November of this year, the Foundation began a contract with Tom Jones, a FreeBSD developer specializing in network performance, to port VPP to FreeBSD. Under the contract, Tom will also allocate time for other tasks such as testing FreeBSD on common virtualization platforms to improve the desktop experience, improving Continue reading

VPP on FreeBSD – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation services router), VPP will look and feel quite familiar as many of the approaches are shared between the two. Over the years, folks have asked me regularly “What about BSD?” and to my surprise, late last year I read an announcement from the FreeBSD Foundation [ref] as they looked back over 2023 and forward to 2024:

Porting the Vector Packet Processor to FreeBSD

Vector Packet Processing (VPP) is an open-source, high-performance user space networking stack that provides fast packet processing suitable for software-defined networking and network function virtualization applications. VPP aims to optimize packet processing through vectorized operations and parallelism, making it well-suited for high-speed networking applications. In November of this year, the Foundation began a contract with Tom Jones, a FreeBSD developer specializing in network performance, to port VPP to FreeBSD. Under the contract, Tom will also allocate time for other tasks such as testing FreeBSD on common virtualization platforms to improve the desktop experience, improving Continue reading

VPP Python API

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

You’ll hear me talk about VPP being API centric, with no configuration persistence, and that’s by design. However, there is this also a CLI utility called vppctl, right, so what gives? In truth, the CLI is used a lot by folks to configure their dataplane, but it really was always meant to be a debug utility. There’s a whole wealth of programmability that is not exposed via the CLI at all, and the VPP community develops and maintains an elaborate set of tools to allow external programs to (re)configure the dataplane. One such tool is my own [vppcfg] which takes a YAML specification that describes the dataplane configuration, and applies it safely to a running VPP instance.

Introduction

In case you’re interested in writing your own automation, this article is for you! I’ll provide a deep dive into Continue reading

Debian on IPng’s VPP Routers

Introduction

When IPng Networks first built out a european network, I was running the Disaggregated Network Operating System [ref], initially based on AT&T’s “dNOS” software framework. Over time though, the DANOS project slowed down, and the developers with whom I had a pretty good relationship all left for greener pastures.

In 2019, Pierre Pfister (and several others) built a VPP router sandbox [ref], which graduated into a feature called the Linux Control Plane plugin [ref]. Lots of folks put in an effort for the Linux Control Plane, notably Neale Ranns from Cisco (these days Graphiant), and Matt Smith and Jon Loeliger from Netgate (who ship this as TNSR [ref], check it out!). I helped as well, by adding a bunch of Netlink handling and VPP->Linux synchronization code, which I’ve written about a bunch on this blog in the 2021 VPP development series [ref].

At the time, Ubuntu and CentOS were the supported platforms, so I installed a bunch of Ubuntu machines when doing the deploy with my buddy Fred from IP-Max [ref]. But as time went by, I fell back to my old habit of running Debian Continue reading

Debian on Mellanox SN2700 (32x100G)

Introduction

I’m still hunting for a set of machines with which I can generate 1Tbps and 1Gpps of VPP traffic, and considering a 100G network interface can do at most 148.8Mpps, I will need 7 or 8 of these network cards. Doing a loadtest like this with DACs back-to-back is definitely possible, but it’s a bit more convenient to connect them all to a switch. However, for this to work I would need (at least) fourteen or more HundredGigabitEthernet ports, and these switches tend to get expensive, real quick.

Or do they?

Hardware

I thought I’d ask the #nlnog IRC channel for advice, and of course the usual suspects came past, such as Juniper, Arista, and Cisco. But somebody mentioned “How about Mellanox, like SN2700?” and I remembered my buddy Eric was a fan of those switches. I looked them up on the refurbished market and I found one for EUR 1’400,- for 32x100G which felt suspiciously low priced… but I thought YOLO and I ordered it. It arrived a few days later via UPS from Denmark to Switzerland.

The switch specs are pretty impressive, with 32x100G QSFP28 ports, which can be broken out to a set of Continue reading

VPP IXP Gateway – Part 1

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

There’s some really fantastic features in VPP, some of which are lesser well known, and not always very well documented. In this article, I will describe a unique usecase in which I think VPP will excel, notably acting as a gateway for Internet Exchange Points.

In this first article, I’ll take a closer look at three things that would make such a gateway possible: bridge domains, MAC address filtering and traffic shaping.

Introduction

Internet Exchanges are typically L2 (ethernet) switch platforms that allow their connected members to exchange traffic amongst themselves. Not all members share physical locations with the Internet Exchange itself, for example the IXP may be at NTT Zurich, but the member may be present in Interxion Zurich. For smaller clubs, like IPng Networks, it’s not always financially feasible (or desirable) to order a dark fiber between two adjacent Continue reading

Case Study: NGINX + Certbot with Ansible

About this series

In the distant past (to be precise, in November of 2009) I wrote a little piece of automation together with my buddy Paul, called PaPHosting. The goal was to be able to configure common attributes like servername, config files, webserver and DNS configs in a consistent way, tracked in Subversion. By the way despite this project deriving its name from the first two authors, our mutual buddy Jeroen also started using it, and has written lots of additional cool stuff in the repo, as well as helped to move from Subversion to Git a few years ago.

Michael DeHaan [ref] founded Ansible in 2012, and by then our little PaPHosting project, which was written as a set of bash scripts, had sufficiently solved our automation needs. But, as is the case with most home-grown systems, over time I kept on seeing more and more interesting features and integrations emerge, solid documentation, large user group, and eventually I had to reconsider our 1.5K LOC of Bash and ~16.5K files under maintenance, and in the end, I settled on Ansible.

commit c986260040df5a9bf24bef6bfc28e1f3fa4392ed
Author: Pim van Pelt <[email protected]>
Date:   Thu Nov 26 23:13:21 2009 +0000

 Continue reading

Pixelfed – Part 1 – Installing

About this series

I have seen companies achieve great successes in the space of consumer internet and entertainment industry. I’ve been feeling less enthusiastic about the stronghold that these corporations have over my digital presence. I am the first to admit that using “free” services is convenient, but these companies are sometimes taking away my autonomy and exerting control over society. To each their own of course, but for me it’s time to take back a little bit of responsibility for my online social presence, away from centrally hosted services and to privately operated ones.

After having written a fair bit about my Mastodon [install] and [monitoring], I’ve been using it every day. This morning, my buddy Ramón asked if he could make a second account on ublog.tech for his Campervan Adventures, and notably to post pics of where he and his family went.

But if pics is your jam, why not … [Pixelfed]!

Introduction

Similar to how blogging is the act of publishing updates to a website, microblogging is the act of publishing small updates to a stream of updates on your profile. Very similar to the relationship between Facebook and Continue reading

VPP MPLS – Part 4

About this series

Special Thanks: Adrian vifino Pistol for writing this code and for the wonderful collaboration!

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

In the last three articles, I thought I had described “all we need to know” to perform MPLS using the Linux Controlplane in VPP:

1. In the [first article] of this series, I took a look at MPLS in general.
2. In the [second article] of the series, I demonstrated a few special case labels (such as Explicit Null and Implicit Null which enables the fabled Penultimate Hop Popping behavior of MPLS.
3. Then, in the [third article], I worked with @vifino to implement the plumbing for MPLS in the Linux Control Plane plugin for VPP. He did most of the work, I just watched :)

As if in a state of premonition, I mentioned:

Caveat empor, outside of a modest functional and Continue reading

VPP MPLS – Part 3

About this series

Special Thanks: Adrian vifino Pistol for writing this code and for the wonderful collaboration!

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

In the [first article] of this series, I took a look at MPLS in general, and how setting up static Label Switched Paths can be done in VPP. A few details on special case labels (such as Implicit Null which enabled the fabled Penultimate Hop Popping) were missing, so I took a good look at them in the [second article] of the series.

This was all just good fun but also allowed me to buy some time for @vifino who has been implementing MPLS handling within the Linux Control Plane plugin for VPP! This final article in the series shows the engineering considerations that went in to writing the plugin, which is currently under review but reasonably complete. Considering the VPP Continue reading

VPP MPLS – Part 2

About this series

Ever since I first saw VPP - the Vector Packet Processor - I have been deeply impressed with its performance and versatility. For those of us who have used Cisco IOS/XR devices, like the classic ASR (aggregation service router), VPP will look and feel quite familiar as many of the approaches are shared between the two.

I’ve deployed an MPLS core for IPng Networks, which allows me to provide L2VPN services, and at the same time keep an IPng Site Local network with IPv4 and IPv6 that is separate from the internet, based on hardware/silicon based forwarding at line rate and high availability. You can read all about my Centec MPLS shenanigans in [this article].

In the last article, I explored VPP’s MPLS implementation a little bit. All the while, @vifino has been tinkering with the Linux Control Plane and adding MPLS support to it, and together we learned a lot about how VPP does MPLS forwarding and how it sometimes differs to other implementations. During the process, we talked a bit about implicit-null and explicit-null. When my buddy Fred read the [previous article], he also talked about a feature called penultimate-hop-popping which Continue reading