Reclaiming CPU for free with Go’s Profile Guided Optimization

Golang 1.20 introduced support for Profile Guided Optimization (PGO) to the go compiler. This allows guiding the compiler to introduce optimizations based on the real world behaviour of your system. In the Observability Team at Cloudflare, we maintain a few Go-based services that use thousands of cores worldwide, so even the 2-7% savings advertised would drastically reduce our CPU footprint, effectively for free. This would reduce the CPU usage for our internal services, freeing up those resources to serve customer requests, providing measurable improvements to our customer experience. In this post, I will cover the process we created for experimenting with PGO – collecting representative profiles across our production infrastructure and then deploying new PGO binaries and measuring the CPU savings.

How does PGO work?

PGO itself is not a Go-specific tool, although it is relatively new. PGO allows you to take CPU profiles from a program running in production and use that to optimise the generated assembly for that program. This includes a bunch of different optimisations such as inlining heavily used functions more aggressively, reworking branch prediction to favour the more common branches, and rearranging the generated code to lump hot paths together to save on CPU Continue reading

EVPN Instance Deployment Scenario 1: L2-Only EVPN Instance

In this scenario, we are building a protected Broadcast Domain (BD), which we extend to the VXLAN Tunnel Endpoint (VTEP) switches of the EVPN Fabric, Leaf-101 and Leaf-102. Note that the VTEP operates in the Network Virtualization Edge (NVE) role for the VXLAN segment. The term NVE refers to devices that encapsulate data packets to transport them over routed IP infrastructure. Another example of an NVE device is the MPLS Provider Edge (MPLS-PE) router at the edge of the MPLS network, doing MPLS labeling. The term “Tenant System” (TS) refers to a physical host, virtual machine, or an intra-tenant forwarding component attached to one or more Tenant-specific Virtual Networks. Examples of TS forwarding components include firewalls, load balancers, switches, and routers. 

We begin by configuring L2 VLAN 10 to Leaf-101 and Leaf-102 and associate it with the vn-segment 10010. From the NVE perspective, this constitutes an L2-Only network segment, meaning we do not configure an Anycast Gateway (AGW) for the segment, and it does not have any VRF association.

Next, we deploy a Layer 2 EVPN Instance (EVI) with VXLAN Network Identifier (VNI) 10010. We utilize the 'auto' option to generate the Route Distinguisher (RD) and the Route Target (RT) import and export values for the EVI. The RD value is derived from the NVE Interface IP address and the VLAN Identifier (VLAN 10) associated with the EVI, added to the base value 32767 (e.g., 192.168.100.101:32777). The use of the VLAN ID as part of the automatically generated RD value is the reason why VLAN is configured before the EVPN Instance. Similarly, the RT values are derived from the BGP ASN and the VNI (e.g., 65000:10010).

As the final step for EVPN Instance deployment, we add EVI 10010 under the NVE interface configuration as a member vni with the Multicast Group 239.1.1.1 we are using for Broadcast, Unknown Unicast, and Multicast (BUM) traffic. 

For connecting TS1 and TS2 to the Broadcast domain, we will configure Leaf-101's interface Eth 1/5 and Leaf-102's interface Eth1/3 as access ports for VLAN 10.

A few words regarding the terminology utilized in Figure 3-2. '3-Stage Routed Clos Fabric' denotes both the physical topology of the network and the model for forwarding data packets. The 3-Stage Clos topology has three switches (ingress, spine, and egress) between the attached Tenant Systems. Routed, in turn, means that switches forward packets based on the destination IP address.

With the term VXLAN Segment, I refer to a stretched Broadcast Domain, identified by the VXLAN Network Identifier value defined under the EVPN Instance on Leaf switches.



Figure 3-2: L2-Only Intra VN Connection.

Continue reading

netlab 1.8.2: Bug Fixes, Usability Improvements

netlab release 1.8.2 contains dozens of bug fixes and minor tweaks to device configuration templates. We also added a few safeguards including:

  • Check for Vagrant boxes or Docker containers before starting the lab and display pointers to build recipes.
  • Check installed Ansible collections before trying to configure the lab devices.
  • Display a warning if the lab topology was modified after the lab was created
Read more …

netlab 1.8.2: Bug Fixes, Usability Improvements

netlab release 1.8.2 contains dozens of bug fixes and minor tweaks to device configuration templates. We also added a few safeguards including:

  • Check for Vagrant boxes or Docker containers before starting the lab and display pointers to build recipes.
  • Check installed Ansible collections before trying to configure the lab devices.
  • Display a warning if the lab topology was modified after the lab was created
Read more …

East African Internet connectivity again impacted by submarine cable cuts

On Sunday, May 12, issues with the ESSAy and Seacom submarine cables again disrupted connectivity to East Africa, impacting a number of countries previously affected by a set of cable cuts that occurred nearly three months earlier.

On February 24, three submarine cables that run through the Red Sea were damaged: the Seacom/Tata cable, the Asia Africa Europe-1 (AAE-1), and the Europe India Gateway (EIG). It is believed that the cables were cut by the anchor of the Rubymar, a cargo ship that was damaged by a ballistic missile on February 18. These cable cuts reportedly impacted countries in East Africa, including Tanzania, Kenya, Uganda, and Mozambique. As of this writing (May 13), these cables remain unrepaired.

Already suffering from reduced capacity due to the February cable cuts, these countries were impacted by a second set of cable cuts that occurred on Sunday, May 12. According to a social media post from Ben Roberts, Group CTIO at Liquid Intelligent Technologies in Kenya, faults on the EASSy and Seacom cables again disrupted connectivity to East Africa, as he noted “All sub sea capacity between East Africa and South Africa is down.” A BBC Continue reading

Network Configuration Diffing

Network Configuration Diffing

Lately, I was working on a Python script to help my team and me manage interface configurations on Cisco switches. One thing my team asked for was a way to see what changes the script was making to the configurations before and after it did its thing. My first thought was to use Python’s difflib module for this. It’s handy for comparing two sequences and showing the differences.

To give you an idea, here’s an example of what the output looked like using the difflib module.

import difflib

before = """
interface HundredGigE1/0/3
 description Dummy interface
 switchport mode access
 switchport access vlan 10
end
"""

after = """
interface HundredGigE1/0/3
 description Dummy interface
 switchport mode access
 switchport access vlan 20
end
"""

diff = difflib.ndiff(before.splitlines(), after.splitlines())
print("\n".join(diff))

The output of the above script is as below:

  interface HundredGigE1/0/3
   description Dummy interface
   switchport mode access
-  switchport access vlan 10
?                         ^

+  switchport access vlan 20
?                         ^

  end

The team wasn’t too thrilled with the output because it was kinda tricky to read. You really had to know what you were looking for to make sense of it.

While looking for a suitable solution, I came across Continue reading

netlab: VRF Instantiation on Lab Devices

In the previous blog post on this topic, I described how node and global VRFs work in netlab.

TL&DR: If you use the same VRF on multiple devices, it’s better to define it globally.

However, you might not need every VRF on every lab device in a more complex lab topology. Considering that, netlab tries to minimize the number of VRFs configured on lab devices using a simple rule: a VRF is configured on a lab device only if the device has at least one interface in that VRF.

Read more …

netlab: VRF Instantiation on Lab Devices

In the previous blog post on this topic, I described how node and global VRFs work in netlab.

TL&DR: If you use the same VRF on multiple devices, it’s better to define it globally.

However, you might not need every VRF on every lab device in a more complex lab topology. Considering that, netlab tries to minimize the number of VRFs configured on lab devices using a simple rule: a VRF is configured on a lab device only if the device has at least one interface in that VRF.

Read more …

Treasury and PNNL threat data now available for Financial sector customers to secure applications

Following the White House’s National Cybersecurity Strategy, which underscores the importance of fostering public-private partnerships to enhance the security of critical sectors, Cloudflare is happy to announce a strategic partnership with the United States Department of the Treasury and the Department of Energy’s Pacific Northwest National Laboratory (PNNL) to create Custom Indicator Feeds that enable customers to integrate approved threat intelligence feeds directly into Cloudflare's platform.

Our partnership with the Department of the Treasury and PNNL offers approved financial services institutions privileged access to threat data that was previously exclusive to the government. The feed, exposed as a Custom Indicator Feed, collects advanced insights from the Department of the Treasury and the federal government's exclusive sources. Starting today, financial institutions can create DNS filtering policies through Cloudflare’s Gateway product that leverage threat data directly from these government bodies. These policies are crucial for protecting organizations from malicious links and phishing attempts specifically targeting the financial sector.

This initiative not only supports the federal effort to strengthen cybersecurity within critical infrastructure including the financial sector, for which the Treasury is the designated lead agency, but also contributes directly to the ongoing improvement of our shared security capabilities.

Why we partnered Continue reading

KU055: KubeCon EU Review

Kristina attended KubeCon EU in March and she’s still trying to process it all. In today’s episode, Michael interviews her about what stood out most to her. They dive into the conference’s heavy emphasis on AI, particularly how Kubernetes can help with more efficient GPU utilization. Kristina also reports back on the United Nations hackathon... Read more »

Greasing The Skids To Move AI From InfiniBand To Ethernet

Just about everybody, including Nvidia, thinks that in the long run, most people running most AI training and inference workloads at any appreciable scale – hundreds to millions of datacenter devices – will want a cheaper alternative for networking AI accelerators than InfiniBand. …

Greasing The Skids To Move AI From InfiniBand To Ethernet was written by Timothy Prickett Morgan at The Next Platform.

Hedge 225: The CCNA

The CCNA has a long history as an important certification for network engineers. While the CCST has been created by Cisco “below” the CCNA, or as a different starting point, many network engineers begin their career with the CCNA. Join Jason Gooley, Wendell Odom, Tom, and Russ as we discuss the most recent updates to the CCNA, the way updates to the program are changing, and Jason’s and Wendell’s updated book on the CCNA.

download

MLAG Deep Dive: Dealing with LAG Member Failures

Craig Weinhold pointed me to a complex topic I managed to ignore in my MLAG Deep Dive series: how does an MLAG cluster reroute around a failure of a LAG member link?

In this blog post, we’ll focus on traditional MLAG cluster implementations using a peer link; another blog post will explore the implications of using VXLAN and EVPN to implement MLAG clusters.

We’ll also ignore the interesting question of “how is the LAG member link failure detected?”1 and focus on “what happens next?” using the sample MLAG topology:

Read more …
1 … 155 156 157 158 159 … 3,856