Archive

Category Archives for "Networking"

Tracing network packets with eBPF and pwru

pwru (packet, where are you?) is an open source tool from Cilium that used eBPF instrumentation in recent Linux kernels to trace network packets through the kernel.

In this article we will use Multipass to create a virtual machine to experiment with pwru. Multipass is a command line tool for running Ubuntu virtual machines on Mac or Windows. Multipass uses the native virtualization capabilities of the host operating system to simplify the creation of virtual machines.

multipass launch --name=ebpf noble
multipass exec ebpf -- sudo apt update
multipass exec ebpf -- sudo apt -y install git clang llvm make libbpf-dev flex bison golang
multipass exec ebpf -- git clone https://github.com/cilium/pwru.git
multipass exec ebpf --working-directory pwru -- make
multipass exec ebpf -- sudo ./pwru/pwru -h
Run the commands above to create the virtual machine and build pwru from sources.
multipass exec ebpf -- sudo ./pwru/pwru port https
Run pwru to trace https traffic on the virtual machine.
multipass exec ebpf -- curl https://sflow-rt.com
In a second window, run the above command to generate an https request from the virtual machine.
SKB                CPU PROCESS          NETNS      MARK/x        IFACE       PROTO  MTU   LEN   TUPLE FUNC
0xffff9fc40335a0e8 0   ~r/bin/curl:8966 4026531840 0               0          Continue reading

AI Metrics with InfluxDB Cloud

The InfluxDB AI Metrics dashboard shown above tracks performance metrics for AI/ML RoCEv2 network traffic, for example, large scale CUDA compute tasks using NVIDIA Collective Communication Library (NCCL) operations for inter-GPU communications: AllReduce, Broadcast, Reduce, AllGather, and ReduceScatter.

The metrics include:

  • Total Traffic Total traffic entering fabric
  • Operations Total RoCEv2 operations broken out by type
  • Core Link Traffic Histogram of load on fabric links
  • Edge Link Traffic Histogram of load on access ports
  • RDMA Operations Total RDMA operations
  • RDMA Bytes Average RDMA operation size
  • Credits Average number of credits in RoCEv2 acknowledgements
  • Period Detected period of compute / exchange activity on fabric (in this case just over 0.5 seconds)
  • Congestion Total ECN / CNP congestion messages
  • Errors Total ingress / egress errors
  • Discards Total ingress / egress discards
  • Drop Reasons Packet drop reasons
This article shows how to integrate with InfluxDB Cloud instead of running the services locally.

Note: InfluxDB Cloud has a free service tier that can be used to test this example.

Save the following compose.yml file on a system running Docker.

configs:
  config.telegraf:
    content: |
      [agent]
        interval = '15s'
        round_interval = true
        omit_hostname = true
      [[outputs.influxdb_v2]]
        urls = ['https://<INFLUXDB_CLOUD_INSTANCE>.cloud2.influxdata.com']
         Continue reading

Quicksilver v2: evolution of a globally distributed key-value store (Part 1)

Quicksilver is a key-value store developed internally by Cloudflare to enable fast global replication and low-latency access on a planet scale. It was initially designed to be a global distribution system for configurations, but over time it gained popularity and became the foundational storage system for many products in Cloudflare.

A previous post described how we moved Quicksilver to production and started replicating on all machines across our global network. That is what we called Quicksilver v1: each server has a full copy of the data and updates it through asynchronous replication. The design served us well for some time. However, as our business grew with an ever-expanding data center footprint and a growing dataset, it became more and more expensive to store everything everywhere.

We realized that storing the full dataset on every server is inefficient. Due to the uniform design, data accessed in one region or data center is replicated globally, even if it's never accessed elsewhere. This leads to wasted disk space. We decided to introduce a more efficient system with two new server roles: replica, which stores the full dataset and proxy, which acts as a persistent cache, evicting unused key-value pairs to free Continue reading

AWS Transit Gateway VPN Attachment and Resource Access Manager (VII)

AWS Transit Gateway VPN Attachment and Resource Access Manager (VII)

In the previous post, we covered the basics of Transit Gateway, what it is, what problem it solves, and we also looked at how to create one. We walked through attaching two VPCs to the TGW and establishing connectivity between them. We also covered the important concepts of TGW attachments, associations, and propagations.

AWS Transit Gateway Introduction
This complexity (when you have many VPCs) is why, in this post, we will look at AWS Transit Gateway (TGW). A Transit Gateway is an incredibly important networking resource in AWS that solves these scaling challenges.
AWS Transit Gateway VPN Attachment and Resource Access Manager (VII)

In this post, we will build on that knowledge and look at

  • Creating and attaching a Site-to-Site VPN to the Transit Gateway.
  • We will also cover sharing a single Transit Gateway across different AWS accounts using Resource Access Manager.
As always, if you find this post helpful, press the ‘clap’ button. It means a lot to me and helps me know you enjoy this type of content. If I get enough claps for this series, I’ll make sure to write more on this specific topic.

Transit Gateway AWS Site-to-Site VPN Attachment

We have already seen how to create a Site-to-Site Continue reading

How TimescaleDB helped us scale analytics and reporting

At Cloudflare, PostgreSQL and ClickHouse are our standard databases for transactional and analytical workloads. If you’re part of a team building products with configuration in our Dashboard, chances are you're using PostgreSQL. It’s fast, versatile, reliable, and backed by over 30 years of development and real-world use. It has been a foundational part of our infrastructure since the beginning, and today we run hundreds of PostgreSQL instances across a wide range of configurations and replication setups.

ClickHouse is a more recent addition to our stack. We started using it around 2017, and it has enabled us to ingest tens of millions of rows per second while supporting millisecond-level query performance. ClickHouse is a remarkable technology, but like all systems, it involves trade-offs.

In this post, I’ll explain why we chose TimescaleDB — a Postgres extension — over ClickHouse to build the analytics and reporting capabilities in our Zero Trust product suite.

Designing for future growth

After a decade in software development, I’ve grown to appreciate systems that are simple and boring. Over time, I’ve found myself consistently advocating for architectures with the fewest moving parts possible. Whenever I see a system diagram with more than three boxes, I ask: Why Continue reading

Calico Whisker & Staged Network Policies: Secure Kubernetes Workloads Without Downtime

Rolling out network policies in a live Kubernetes cluster can feel like swapping wings mid-flight—one typo or overly broad rule and critical traffic is grounded. Calico’s Staged Network Policies remove the turbulence by letting you deploy policies in staged mode, so you can observe their impact before enforcing anything. Add Whisker, the open-source policy enforcement and testing tool (introduced as part of Calico Open Source 3.30) that captures every flow and tags it with a policy verdict, and you’ve got a safety harness that proves your change is sound long before you flip the switch. In this post, we’ll walk you through how you can leverage these capabilities to tighten security, validate intent, and ship changes confidently—without a single packet of downtime.

Deploying a Kubernetes Cluster

Calico for Policy is a CNI agnostic tool. Refer to the Calico Open Source docs for a list of supported CNIs. The git repository for this blog post can be found here.

For this post, let’s deploy a simple AKS cluster with Azure CNI.

## Configure 
az group create --name calicooss --location eastus2

## Create a 3 node AKS cluster with Azure CNI
az aks create \
  --resource-group calicooss \
  --name  Continue reading

Introducing simple and secure egress policies by hostname in Cloudflare’s SASE platform

Cloudflare’s SASE platform is on a mission to strengthen our platform-wide support for hostname- and domain-based policies. This mission is being driven by enthusiastic demands from our customers, and boosted along the way by several interesting engineering challenges. Today, we’re taking a deep dive into the first milestone of this mission, which we recently released in open beta: egress policies by hostname, domain, content category, and application. Let’s dive right in! 

Egress policies and IP ACLs

Customers use our egress policies to control how their organization's Internet traffic connects to external services. An egress policy allows a customer to control the source IP address their traffic uses, as well as the geographic location that their traffic uses to egress onto the public Internet. Control of the source IP address is especially useful when accessing external services that apply policies to traffic based on source IPs, using IP Access Control Lists (ACLs). Some services use IP ACLs because they improve security, while others use them because they are explicitly required by regulation or compliance frameworks. 

(That said, it's important to clarify that we do not recommend relying on IP ACLs as the only security mechanism used to gate Continue reading

Ultra Ethernet

Introduction

Remote Direct Memory Access over Converged Ethernet (RoCE) is a transport model that extends InfiniBand semantics over Ethernet networks. It enables direct memory access between hosts by encapsulating InfiniBand transport headers—such as the InfiniBand Transport Header (IBTH) and the RDMA Extended Transport Header (RETH)—within Ethernet, IP, and UDP packets. In by book "Deep Learning for Network Engineers" Chapter 9, describes how RDMA NICs process application work requests, known as InfiniBand verbs, and how these are encoded into IBTH and RETH headers for delivery to remote targets using RoCEv2.

This post shifts focus to the Ultra Ethernet Transport (UET) model, developed by the Ultra Ethernet Consortium (UEC). UET defines an alternative RDMA transport architecture that operates over standard Ethernet networks, without relying on InfiniBand message formats or semantics. While both RoCEv2 and UET enable remote memory access between nodes, UET is not based on InfiniBand transport headers, and the term RoCE is not used in UET systems.

Instead, UET introduces a new Ultra Ethernet (UE) layer composed of several sublayers, including the Semantic Sublayer (SES) and the Packet Delivery Sublayer (PDS). These sublayers are responsible for encoding and transmitting RDMA operations—such as memory addresses, remote keys (RKEYs), operation codes, and Continue reading

AWS Transit Gateway Introduction (VI)

AWS Transit Gateway Introduction (VI)

In the previous post, we covered VPC Peering, which is a quick and easy way to create a connection between two VPCs. We also discussed its limitations, primarily that it is non-transitive. This means if VPC 'A' is peered with VPC 'B', and VPC 'B' is peered with VPC 'C', VPC 'A' cannot communicate with VPC 'C' through VPC 'B'. Because of this, to connect multiple VPCs together, you need to create a full mesh, where every VPC has a direct peering connection to every other VPC.

AWS VPC Peering
In this post, we will continue to expand on VPC connectivity by looking at what AWS VPC Peering is and how to configure one.
AWS Transit Gateway Introduction (VI)

This complexity (when you have many VPCs) is why, in this post, we will look at AWS Transit Gateway (TGW). A Transit Gateway is an incredibly important networking resource in AWS that solves these scaling challenges. You will see the TGW featured in many modern AWS architecture diagrams because of the flexibility and simplicity it provides.

As always, if you find this post helpful, press the ‘clap’ button. It means a lot to me and helps me know you enjoy Continue reading

AWS VPC Peering (V)

AWS VPC Peering (V)

Welcome back to the AWS Networking series. So far, we have covered a wide range of foundational topics. We started with the basics of building a VPC, creating subnets, configuring route tables, and providing Internet access with an Internet Gateway and a NAT Gateway. We then looked at the difference between stateful Security Groups attached to an instance's ENI and stateless Network ACLs applied at the subnet level. Most recently, we covered how to build a hybrid network using a Site-to-Site VPN.

In this post, we will continue to expand on VPC connectivity by looking at what AWS VPC Peering is and how to configure one.

If you are completely new to AWS networking, I highly recommend checking out our introductory posts linked below. However, if you are already familiar with the basics, you can carry on with this post.

AWS Networking Fundamentals
If you’re brand new to AWS, don’t worry. This post focuses on the basics of AWS networking. General networking knowledge is helpful but not required - I’ll try to explain things clearly so everyone can follow along.
AWS VPC Peering (V)
As always, if you find this post helpful, press the ‘clap’ button. Continue reading

Build Your Own Private Cloud at Home With Docker

If you’re like me, you depend on a lot of systems and services, even within your home LAN. Because I work from home, that’s amplified to the point where I need certain applications available to me that aren’t hosted by a third party, for flexibility, ease of use, reliability and security. Thankfully, Docker is there to make deploying those apps and services considerably easier; otherwise, I’d wind up having to first deploy a collection of virtual machines (VMs), keep them running and worry about upgrading/managing them efficiently. Yeah, Docker makes this entire process easier. Even better, I can spin up those apps and services in seconds, instead of having to go the traditional route, which can often take quite a bit longer to deploy. But what are the apps and services that I depend on for my LAN to keep me productive? Surprise, surprise: I have a list, and here it is. Nextcloud

AWS Site-to-Site VPN (IV)

AWS Site-to-Site VPN (IV)

So far in the AWS Networking series, we have covered VPCs, subnets, route tables, Internet Gateways, NAT Gateways, EC2 instances, Security Groups, Network ACLs, and Elastic Network Interfaces. In this post, we will look at using a Site-to-Site VPN in AWS so you can securely connect your on-premise workloads to and from your AWS environment. This is a very important aspect of AWS networking, and this is a service you will use almost always.

If you have been following the series, you can easily follow along with this post. If you just stumbled upon this post, you can still continue, assuming you are already familiar with AWS networking basics. However, if you are completely new to AWS, I highly recommend checking out the previous posts linked below.

AWS Networking Fundamentals
If you’re brand new to AWS, don’t worry. This post focuses on the basics of AWS networking. General networking knowledge is helpful but not required - I’ll try to explain things clearly so everyone can follow along.
AWS Site-to-Site VPN (IV)

Why Do We Need a VPN to AWS?

When we launch an instance in a public subnet with a public IP address, we have seen that we can connect to Continue reading

Minding the Gap: SDS That Carries You Through Last Mile

Data center modernization is inevitable to keep pace with today’s performance-intensive workloads, which are growing especially rapidly in the financial services and e-commerce sectors. Software-defined storage (SDS) offers the promise of greater agility, scalability and reduced costs, making it a compelling part of the platform architect’s vision of a streamlined, software-driven infrastructure. A common priority for modernization is retiring traditional Fibre Channel (FC) SANs, which are perceived as legacy and complex. While still adequate for many application workloads, performance gaps exist in SDS block storage solutions that are engineered into legacy FC SANs, which cannot support the organization’s mission-critical application workloads. Once reserved as the primary storage tier for traditional workloads,

Palo Alto Upgrade from 10.2 to 11.2 (PA-440)

Palo Alto Upgrade from 10.2 to 11.2 (PA-440)

I have PA-440 in my home lab and was happily running PAN-OS 10.2.10-h9. But with the recent announcement that PAN-OS 10.2 will enter limited support from 26th August 2025, I decided it was time to upgrade. I was deciding between 11.1 and 11.2 for a while, but after reading through a few forums and discussions, I ended up choosing 11.2, specifically PAN-OS 11.2.4-h7.

Since I was already on 10.2, I could upgrade directly to 11.2 without going through any intermediate versions. As per the upgrade guide, all I had to do was download the 11.2.0 base image, then download and install 11.2.4-h7.

Palo Alto Upgrade from 10.2 to 11.2 (PA-440)
💡
Just to clarify, you only need to download the 11.2.0 base image, there’s no need to install it.
Palo Alto Upgrade from 10.2 to 11.2 (PA-440)

After downloading both the base image and the target image, just click 'Install' on the target image. As usual, make sure to take a backup before starting. If you’re running in HA, you can upgrade the firewalls one at a time without any downtime.

Palo Alto Upgrade from 10.2 to 11.2 (PA-440)
Palo Alto Upgrade from 10.2 to 11.2 (PA-440)

The whole process took about 10 to 15 minutes, and now I'm running 11.2.4-h7. If I come across any issues, I'll be sure to update this post.

Palo Alto Upgrade from 10.2 to 11.2 (PA-440)

Worth Reading: Expert Generalists

Martin Fowler published an interesting article about Expert Generalists. Straight from the abstract:

As computer systems get more sophisticated we’ve seen a growing trend to value deep specialists. But we’ve found that our most effective colleagues have a skill in spanning many specialties.

Also:

There are two sides to real expertise. The first is the familiar depth: a detailed command of one domain’s inner workings. The second, crucial in our fast-moving field is the ability to learn quickly, spot the fundamentals that run beneath shifting tools and trends, and apply them wherever we land.

Remember how I told you to focus on the fundamentals? 😎

A Detailed Look at Calico Cloud Free Tier

Why Calico Cloud Free Tier?

As Kubernetes environments grow in scale and complexity, platform teams face increasing pressure to secure workloads without slowing down application delivery. But managing and enforcing network policies in Kubernetes is notoriously difficult—especially when visibility into pod-to-pod communication is limited or nonexistent. Teams are often forced to rely on manual traffic inspection, standalone logs, or trial-and-error policy changes, increasing the risk of misconfiguration and service disruption. Safe policy management and microsegmentation becomes a daunting task without clear knowledge or insight into which services should communicate with each other.

In this detailed look, we’ll explore how Calico Cloud Free Tier builds upon Calico Open Source, and helps platform teams visualize traffic with a dynamic service graph, simplifies policy management, and even analyzes actual traffic to recommend policies.

What is Calico Cloud Free Tier?

Calico Cloud Free Tier is a managed SaaS, no-cost offering that extends the capabilities of Calico Open Source 3.30 and higher to help Kubernetes teams improve network visibility, simplify policy management, and improve security by simplifying microsegmentation. Designed for single-cluster environments, it provides platform engineers and operators with powerful observability and policy management tools. With a seamless onboarding experience for users already Continue reading

1 2 3 3,444