Open sourcing Pingora: our Rust framework for building programmable network services

Today, we are proud to open source Pingora, the Rust framework we have been using to build services that power a significant portion of the traffic on Cloudflare. Pingora is released under the Apache License version 2.0.

As mentioned in our previous blog post, Pingora is a Rust async multithreaded framework that assists us in constructing HTTP proxy services. Since our last blog post, Pingora has handled nearly a quadrillion Internet requests across our global network.

We are open sourcing Pingora to help build a better and more secure Internet beyond our own infrastructure. We want to provide tools, ideas, and inspiration to our customers, users, and others to build their own Internet infrastructure using a memory safe framework. Having such a framework is especially crucial given the increasing awareness of the importance of memory safety across the industry and the US government. Under this common goal, we are collaborating with the Internet Security Research Group (ISRG) Prossimo project to help advance the adoption of Pingora in the Internet’s most critical infrastructure.

In our previous blog post, we discussed why and how we built Pingora. In this one, we will talk about why and how you might Continue reading

EVPN Terminology

Reading RFCs is a great source of information for understanding all the details of a protocol. Often they do require the reader to be quite technical and the terminology can be confusing if you aren’t used to the type of language and writing style used in RFCs. In this post, I go through some of the most important terminology in EVPN and VXLAN to help you build your understanding of the different forwarding constructs and how they interact.

The picture below shows some of the most important terminology in EVPN:

Let’s go through the terms used in the diagram and some additional ones:

  • Attachment circuit – An interface that is associated with a bridge table. The AC that the packet arrived on is determined by examining the port, and optionally VLAN tag.
  • Broadcast Domain – The Broadcast domain consists of all devices and hosts that would receive a broadcast frame when sent in that domain (assuming no ARP optimization features used). This is normally a VLAN, and it normally maps to one subnet. From a VXLAN perspective, it would be a L2 VNI. An EVI may contain one or more BDs depending on service model.
  • Bridge Table – Bridge Table Continue reading

DHCP Relaying on a Linux Host

Markku Leiniö sent me an interesting observation after writing a series of DHCP-relaying-related blog posts:

I was first using VyOS, but it uses the ISC DHCP relay, and that software relays unicast packets. The DHCP procedures eventually worked fine, but getting sensible outputs and explanations was a nightmare.

I quickly reproduced the behavior, but it took me almost half a year to turn it into a blog post. Engaging in a round of yak shaving (I wanted to implement DHCP in netlab first) didn’t exactly help, either.

Read more …

DHCP Relaying on a Linux Host

Markku Leiniö sent me an interesting observation after writing a series of DHCP-relaying-related blog posts:

I was first using VyOS, but it uses the ISC DHCP relay, and that software relays unicast packets. The DHCP procedures eventually worked fine, but getting sensible outputs and explanations was a nightmare.

I quickly reproduced the behavior, but it took me almost half a year to turn it into a blog post. Engaging in a round of yak shaving (I wanted to implement DHCP in netlab first) didn’t exactly help, either.

Read more …

Open-source network simulation roundup 2024

I updated the status of the open-source network simulation and network emulation projects tracked in this blog. I found no new projects to add to my lists in 2023 or early 2024. I also did not have to delete any projects because they all seem to have an acceptable level of support in 2024.

Read the rest of this post for information about the latest open-source network simulator releases, as of the end of February, 2024.

Network Simulators

  • cnet (No links; see below.)
    • The development team is re-writing the cnet code base and the old version is no longer available to download. A new version will be released soon: in, or after, March 2024. I will update the links on my web site at that time.
  • ns-3
    • ns-3 continues to be a major, active project. The most recent release was ns-3.41 on February 9, 2024. The ns-3 source code is on GitLab. at
  • OMNet++
    • OMNeT++ also continues to be a major project that is actively maintained. Release 6.0.3 was made available on February 24, 2024. The OMNET++ source code is on GitHub.
  • Shadow
    • Shadow is actively maintained with many fixed and features added in 2023. The Continue reading

Netflix Releases bpftop: An eBPF-Based Application Monitor

Extended Berkeley Packet Filter, eBPF to its friends, enables you to run sandboxed programs in a privileged context in the Linux kernel. Netflix has unveiled bpftop, a new open source command-line tool designed to enhance the performance optimization and monitoring of eBPF applications. As the streaming giant continues integrating eBPF technology into its infrastructure, ensuring these applications operate efficiently has become a top priority.

VXLAN Virtual Labs Have Never Been Easier

I stumbled upon an “I want to dive deep into VXLAN and plan to build a virtual lab” discussion on LinkedIn1. Of course, I suggested using netlab. After all, you have to build an IP core and VLAN access networks and connect a few clients to those access networks before you can start playing with VXLAN, and those things tend to be excruciatingly dull.

Now imagine you decide to use netlab. Out of the box, you get topology management, lab orchestration, IPAM, routing protocol design (OSPF, BGP, and IS-IS), and device configurations, including IP routing and VLANs.

Read more …

VXLAN Virtual Labs Have Never Been Easier

I stumbled upon an “I want to dive deep into VXLAN and plan to build a virtual lab” discussion on LinkedIn1. Of course, I suggested using netlab. After all, you have to build an IP core and VLAN access networks and connect a few clients to those access networks before you can start playing with VXLAN, and those things tend to be excruciatingly dull.

Now imagine you decide to use netlab. Out of the box, you get topology management, lab orchestration, IPAM, routing protocol design (OSPF, BGP, and IS-IS), and device configurations, including IP routing and VLANs.

Read more …

Human Generated Questions About AI Assistants

I’ve taken a number of briefings in the last few months that all mention how companies are starting to get into AI by building an AI virtual assistant. In theory this is the easiest entry point into the technology. Your network already has a ton of information about usage patterns and trouble spots. Network operations and engineering teams have learned over the years to read that information and provide analysis and feedback.

If marketing is to be believed, no one in the modern world has time to learn how to read all that data. Instead, AI provides a natural language way to ask simple questions and have the system provide the data back to you with proper context. It will highlight areas of concern and help you grasp what’s going on. Only you don’t need to get a CCNA to get there. Or, more likely, it’s more useful for someone on the executive team to ask questions and get answers without the need to talk to the network team.

I have some questions that I always like to ask when companies start telling me about their new AI assistant that help me understand how it’s being built.

Question 1: Laying Continue reading

Ansible Set Operations Do Not Preserve List Order

Here’s another Ansible quirk, this time caused by Python set behavior.

When I created the initial device configuration deployment playbook in netlab, I wanted to:

  • Be able to specify a list of modules to provision.1
  • Provision just the modules used in the topology and specified in the list of modules.

This allows you to use netlab initial to deploy all configuration modules used in a lab topology or netlab initial -m ospf to deploy just OSPF while surviving netlab initial -m foo (which would do nothing).

Read more …

Ansible Set Operations Do Not Preserve List Order

Here’s another Ansible quirk, this time caused by Python set behavior.

When I created the initial device configuration deployment playbook in netlab, I wanted to:

  • Be able to specify a list of modules to provision.1
  • Provision just the modules used in the topology and specified in the list of modules.

This allows you to use netlab initial to deploy all configuration modules used in a lab topology or netlab initial -m ospf to deploy just OSPF while surviving netlab initial -m foo (which would do nothing).

Read more …

Bridging Packet Walk In VXLAN/EVPN Network

In this post I walk you through all the steps and packets involved in two hosts communicating over a L2 VNI in a VXLAN/EVPN network. The topology below is the one we will be using:

The lab has the following characteristics:

  • OSPF in the underlay.
  • Ingress replication for BUM traffic through the use of EVPN.
  • ARP suppression is enabled.
  • ARP cache is cleared on Server-1 and Server-4 before initating the packet capture.
  • Server-1 is the host sourcing traffic by pinging Server-4.

Server-1 clears the ARP entry for Server-4 and initiates the ping:

sudo ip neighbor del 198.51.100.44 dev ens160
ping 198.51.100.44
PING 198.51.100.44 (198.51.100.44) 56(84) bytes of data.
64 bytes from 198.51.100.44: icmp_seq=1 ttl=64 time=6.38 ms
64 bytes from 198.51.100.44: icmp_seq=2 ttl=64 time=4.56 ms
64 bytes from 198.51.100.44: icmp_seq=3 ttl=64 time=4.60 ms

Below is packet capture showing the ARP request from Server-1:

Frame 7854: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface ens257, id 4
Ethernet II, Src: 00:50:56:ad:85:06, Dst: ff:ff:ff:ff:ff:ff
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type:  Continue reading
1 … 157 158 159 160 161 … 3,834