0

BGP EVPN in SONiC – Part II: BGP EVPN – Inter-Subnet Forwarding

Introduction

 

Though inter-subnet communication in a BGP EVPN-based network between hosts happens through the anycast default gateway of the subnet, it can use two different forwarding models: Asymmetric and Symmetric Integrated Routing and Bridging (IRB). The basic difference between these two IRB models is how the packet is forwarded between the ingress and egress VTEPs.

When an ingress VTEP receives a frame whose destination is in a different subnet from the source, the initial forwarding operations are common to both IRB models. The VTEP performs a MAC lookup in the local VLAN's MAC address table and resolves the anycast gateway (AGW) MAC address. The packet is then routed toward the tenant IP-VRF, where an IP lookup determines the route toward the destination IP address. The routing information identifies the remote VTEP as the overlay next hop. The remote VTEP's underlay IP address is then resolved for use as the outer VXLAN destination IP address. This resolution can involve recursive lookup through the underlay routing table.

The forwarding paths diverge after the ingress VTEP performs the IP lookup. In asymmetric IRB, the ingress VTEP performs an additional MAC lookup in the bridge table associated with the destination subnet. The resulting Continue reading

0

BGP EVPN in SONiC – Part I: L2VNI for Inter-Switch VLAN Extension

 

Introduction

 

The previous chapter explained how to build an IPv6-only multi-ASN Clos fabric where IPv4 prefixes can be advertised with IPv6 link-local next hops.

In this chapter, we extend VLAN 10 between Leaf-101 and Leaf-102 using BGP EVPN as the control plane and VXLAN as the data plane.

Figure 7-1 shows the main building blocks required to provide this functionality. The IP underlay, where leaf and spine switches use IPv6 link-local addresses for eBGP peering, has already been configured in the previous chapter. Loopback0 is used only as the BGP Router ID and is not advertised to any peer.

The IP address of Loopback1, in turn, is used as the VXLAN Tunnel Endpoint (VTEP) address. Each switch therefore advertises its Loopback1 IP address using BGP IPv4 Unicast so that the VTEP addresses are reachable across the fabric. This address identifies the local VTEP in the EVPN control plane and is used as the BGP next hop for EVPN routes. A VXLAN packet sent to a remote VTEP uses the remote VTEP address as the destination address in the outer IP header.

Each extended VLAN is mapped to an L2VNI (Layer-2 VXLAN Network Identifier). In Figure 7-1, VLAN 10 Continue reading

0

SONiC Deep Dive: BGP Update Message Processing

 

BGP Route Advertisement

The next step after configuring BGP peering using BGP Unnumbered with IPv6 link-local addresses is to advertise the host networks. The lower part of Figure 6-10 shows the sonic-cli configuration commands used to advertise the local VLAN subnet on each leaf. On Leaf-101, we advertise 10.0.10.0/24, which is the subnet for VLAN 10. On Leaf-102, we advertise 10.0.20.0/24, which is the subnet for VLAN 20.

By examining the BGP Loc-RIB on Leaf-102, for example, we can see that the next hop for 10.0.10.0/24, originated on Leaf-101, is the IPv6 link-local address of Spine-11's Ethernet0 interface.

Figure 6-10: IPv4 Routes Installed in the BGP Loc-RIB Table.

Figure 6-11 verifies that the host subnets learned through BGP have been installed in the IP routing table on all switches. For example, the destination prefix 10.0.10.0/24 is installed on Leaf-102 with the next hop fe80::e22:34ff:feb6:a and the outgoing interface Ethernet0.

This also illustrates the separation between the BGP control plane and the data plane. BGP uses IPv6 link-local addresses to establish the BGP session and exchange BGP messages carrying IPv4 NLRI, with an IPv6 link-local address specified as Continue reading

0

SONiC Deep Dive: BGP Peer Configuration – BGP Unnumbered

 

After enabling IPv6 on the Ethernet0 interface, we can configure BGP peering. Instead of statically defining a peer IP address and AS number, the BGP neighbor is defined at the interface level. This tells BGP to expect a neighbor and initialize the BGP peering over Ethernet0. Because our intent is to transport IPv4 traffic over the IPv6 network, we activate both the IPv4 and IPv6 address families. We also configure the peer as an external BGP (eBGP) neighbor by using a command remote-as external. This tells BGP that the peer must use an AS number different from the local AS number.

Because the network uses IPv6 as the transport for IPv4 traffic, IPv4 Network Layer Reachability Information (NLRI) exchanged through BGP use IPv6 next-hop addresses. BGP therefore needs the Extended Next Hop Encoding capability to support IPv4 routes with IPv6 next hops. In this configuration, the capability is explicitly enabled with the capability extended-nexthop command.

In FRRouting, the v6only option controls which IP version is used for interface-based BGP Unnumbered peering. Without v6only, FRR uses the interface's IPv6 link-local address for peering only when no suitable IPv4 address is configured on the interface. With v6only, FRR skips Continue reading

0

SONiC Deep Dive: BGP Unnumbered for IPv4

 

Chapter 6: BGP Unnumbered for IPv4

 

Introduction

 

This chapter explains how an IPv6-only transport network can provide connectivity between IPv4 networks by using IPv6 link-local addresses as next-hop addresses for IPv4 routes.

The first part of the chapter starts by showing how to configure interfaces to enable IPv6 and automatically generate IPv6 link-local addresses. These addresses can then be advertised to IPv6 peers using Router Advertisement (RA) messages defined by the IPv6 Neighbor Discovery Protocol (NDP). This provides the IPv6 link-local addressing required to establish BGP Unnumbered peering. Because the focus of the book is SONiC NOS, this section also shows how locally configured, interface-specific IPv6 information, as well as information received from neighboring devices, is represented in the SONiC Redis databases.

The second part of the chapter introduces how to configure external BGP peering using IPv6 link-local addresses without explicitly specifying the neighbor's IPv6 address or ASN. This simplifies automation by reducing the number of switch-specific configuration parameters. As in the first part, we examine the relevant Redis database tables after configuration and verification.

Figure 6-1 depicts the example topology used in this chapter. The physical topology is based on two-tiers Clos topology, with a single Continue reading

0

SONiC Deep Dive: Intra-Switch Inter-VLAN Routing and Packet Forwarding

 

The title of this section might seem a little overwhelming, but it reflects the focus of the chapter. The SONiC Redis database programming pipeline is examined through the configuration of two Layer-2 VLANs, VLAN 10 and VLAN 20, together with their respective Layer-3 routing interfaces.

VLAN 10 and VLAN 20 represent two separate Layer-2 broadcast domains, with each VLAN associated with its own IP subnet. VLAN 10 uses the 10.0.10.0/24 subnet, while VLAN 20 uses the 10.0.20.0/24 subnet. Because the two hosts belong to different broadcast domains and IP subnets, communication between them requires Layer-3 routing. In this example, the routing is performed locally by Leaf-1 through the routing interfaces configured for the two VLANs.

Figure 5-15 depicts the basic building blocks and IP addressing scheme used in this example. PC1, with IP address 10.0.10.101/24, belongs to VLAN 10 and is connected to the Leaf-1 Ethernet0 interface. PC2, with IP address 10.0.20.101/24, belongs to VLAN 20 and is connected to the Leaf-1 Ethernet1 interface. The routing interfaces for VLAN 10 and VLAN 20 use IP addresses 10.0.10.1/24 and 10.0.20.1/24, respectively.

The Continue reading

0

SONiC Deep Dive: Binding Interface to Vlan

Step 3: Associate Ethernet0 to VLAN 10

After configuring VLAN 10 and bringing up interface Ethernet0, the next step is to make Ethernet0 an untagged member of VLAN 10.

GONFIG_DB and APPL_DB

Figure 5-11 gives a conceptual view of the Redis database updates resulting from the SONiC command:

sudo config vlan member add 10 Ethernet0 --untagged

The command defines our intent: Ethernet0 should participate in VLAN 10 as an untagged (access) port. For an untagged VLAN member, frames received from the host arrive without an 802.1Q tag and are associated with VLAN 10 based on the port's VLAN membership. In the opposite direction, frames transmitted from VLAN 10 toward Ethernet0 are sent without an 802.1Q tag.

The command updates several related entries in CONFIG_DB. The PORT|Ethernet0 entry contains the port's access VLAN configuration:

PORT|Ethernet0

    access_vlan = 10

 

The VLAN|Vlan10 entry represents the VLAN itself and contains a members@ field that records Ethernet0 as a member:

 

VLAN|Vlan10

   members@ = Ethernet0   

   vlanid = 10

 

The @ character in members@ is part of the SONiC CONFIG_DB schema convention. The members@ value is a configuration representation of the relationship between the VLAN and its members.

The Continue reading

0

SONiC Deep Dive: Interface State Change Pipeline

Enable Interface Ethernet0

After creating VLAN 10, the next step is to enable interface Ethernet0. The baseline configuration for Ethernet0 already exists in the PORT table of CONFIG_DB. This configuration is populated during the SONiC switch startup process using the platform configuration and configuration files, including config_db.json. When the switch boots, the configuration from config_db.json is loaded into CONFIG_DB, which is implemented as a Redis database.

 

GONFIG_DB and APPL_DB

The SONiC command config interface startup Ethernet0 does not create the complete PORT|Ethernet0 entry to the CONFIG_DB. Instead, it changes the administrative state of the existing port configuration. In this example, the admin_status field of PORT|Ethernet0 is changed to up (1).

 

After admin_status is changed, Redis generates a notification for applications subscribed to the PORT table in CONFIG_DB (2). The Redis notification indicates that the configuration entry has changed. It identifies the affected key and the operation that occurred rather than containing the complete PORT|Ethernet0 configuration. portmgrd, running in the SWSS container, is one of the SONiC components involved in processing port configuration. portmgrd uses the notification to identify the changed key and retrieves the current field/value data from CONFIG_DB. It then processes the configuration and prepares Continue reading

0

SONiC Deep Dive: Configuring VLAN 10 from CLI to ASIC

 

Introduction

In this chapter, we examine what happens under the hood when we configure a Layer-2 VLAN on a SONiC switch, enable two host-facing interfaces, and associate them with the previously created VLAN as untagged access ports. Figure 5-1 illustrates the conceptual configuration in which Host-1 and Host-2 are connected to interfaces Eth1/1 and Eth1/2, respectively. Both interfaces are members of VLAN 10, which at this stage functions as a Layer-2 broadcast domain. The host-facing interfaces are single-lane 25-Gigabit Ethernet interfaces.

 

We begin by creating VLAN 10 and examining the complete SONiC processing sequence, from the user's VLAN configuration to ASIC programming. Next, we study what happens when the interfaces are enabled and associated with VLAN 10. Finally, we verify that Host-1 and Host-2 have IP connectivity through Leaf-1. Although this is a very basic configuration example, many SONiC components participate in the process. The database examples presented in this chapter are taken from the lab environment. The communication between processes is explained at a conceptual level because the focus is on SONiC operation rather than the underlying Linux communication mechanisms.

Figure 5-1: Host Connectivity.

Step 1: Configure Vlan 10

 

Figure 5-2 depicts the conceptual sequence of Continue reading

0

SONiC – ASIC Programming: Port Characteristics

 The previous section described at a high level how the config_db.json file is loaded into the Redis CONFIG_DB database in the Database container. Next we look at the topic in slightly more detail. First, we examine how physical port information, such as the port's default speed and lane configuration, is processed during the initial boot before it is programmed into the switch ASIC.

 

Step 1: Creating a Config_db.json and Updating CONFIG_DB by Sonic-cogent

 

Figure 4-2 illustrates how platform-specific port information becomes the PORT and BREAKOUT_CFG entries stored in the Redis CONFIG_DB database. During the image initialization process, sonic-cfggen uses the platform-specific port_config.ini file (1) together with other platform configuration files and templates to generate the port-related entries in the config_db.json file (2). The port_config.ini file defines the logical, expected physical characteristics of the switch ports, including the front-panel alias names, lane mappings, index values, subport values, and default port speeds. After the config_db.json file has been generated, its contents are loaded into the Redis CONFIG_DB database during the SONiC boot process (3).

 

Figure 4-2 also shows the DEVICE_METADATA table, which contains platform-specific information such as the base MAC address, switch_type, Continue reading

0

SONiC: Populating CONFIG_DB

Introduction

Before a switch can forward data traffic, its ASIC must first be programmed with the device's port configuration. This includes information such as the number of front-panel ports, their supported speeds, the number of lanes assigned to each port, the speed of each lane, and the administrative and operational state of every port.

SONiC relies heavily on a Redis-based database model. During startup, the configuration stored in config_db.json is loaded into CONFIG_DB. The *mgrd daemons running inside the SWSS container then receive the changes relevant to them through Redis Pub/Sub notifications. After a daemon has processed its part of the configuration, it publishes the corresponding application-level state to APPL_DB.

The orchagent process in the SWSS container monitors changes in APPL_DB, translates the application-level state into hardware objects suitable for ASIC programming, and publishes the resulting hardware objects to ASIC_DB. The actual ASIC programming takes place in the next stage: the syncd container monitors ASIC_DB changes and passes them through the SAI interface to the vendor-specific SDK, which interacts with the ASIC driver stack to program the physical switch ASIC.

This chapter focuses on port programming because it clearly illustrates the complete programming pipeline. Other configuration objects follow the Continue reading

0

Chapter 3: SONiC Startup Process

Figure 3-1 illustrates a simplified SONiC container startup sequence. First, systemd reads the service unit files and evaluates their dependency and ordering directives. It then starts docker.service, making the Docker daemon available. After that, SONiC service containers are started according to the dependencies and ordering rules defined in their service unit files.

In this simplified example, database.service starts early because many SONiC services rely on the Redis databases to exchange configuration, state, and event information. Services such as pmon, syncd, and swss are then started according to their own dependencies and ordering rules. Higher-level SONiC service containers, such as bgp, lldp, teamd, and snmp, are started after the lower-level services they depend on are available.

It is also important to distinguish between host-level service management and process management inside containers. systemd is responsible for starting the host-level service units that create and manage SONiC service containers. After a container has started, the processes inside the container are launched and supervised by the container's own initialization logic, which in many SONiC containers is based on supervisord. For example, after the bgp container has started, processes such as bgpd, zebra, and fpmsyncd are started inside the container. Similarly, the swss Continue reading

0

Chapter 2: Installing SONiC NOS

 

ONIE-Based SONiC Installation

Many switch vendors have added SONiC NOS support to at least part of their switch portfolio. Depending on the vendor and switch model, customers may be able to order a switch with a vendor-customized SONiC version that is supported at the same level as the vendor's own network operating system. Some vendors also allow customers to run the community-based SONiC distribution.

The support model for Community SONiC depends on the vendor. Some hardware vendors provide full support, while others provide no support at all. Compared with vendor-specific SONiC distributions, Community SONiC provides greater flexibility because it can be customized, rebuilt, and adapted to customer requirements. However, running a Community SONiC deployment without vendor support or in-house expertise is generally not a recommended operating model.

Community SONiC is typically installed by using ONIE (Open Network Install Environment) [1], a small open-source installation environment that provides a standardized method for installing network operating systems on supported switches. Figure 2-1 illustrates a conceptual ONIE-based Community SONiC installation process.

If the switch is delivered with a vendor-specific SONiC distribution already installed, it may boot directly into that operating system without requiring a separate ONIE installation workflow. For Community SONiC deployments, Continue reading

0

Chapter 1: SONiC Fundamentals

Introduction

SONiC (Software for Open Networking in the Cloud) is a Linux-based open-source network operating system that was originally developed at Microsoft and is now maintained by a broader open-source community. Its core idea is that the same network operating system can run on switch platforms from multiple hardware vendors. This reduces vendor lock-in and provides a more consistent operational model across different environments.

SONiC can also be viewed as an abstraction layer between network operators and the underlying switch hardware. Instead of learning and managing several vendor-specific operating systems, operators can use a common software architecture and management model across different switch platforms. This simplifies network operations, automation, monitoring, and telemetry collection. It can also reduce operational errors caused by configuration differences between platforms and make it easier to onboard new engineers.

Organizations can choose the hardware platform that best meets their technical, operational, and business requirements without being tied to a single software ecosystem. Some vendors provide commercially supported SONiC distributions together with professional support services, while others support community-based deployments or customer-tailored implementations. The appropriate model depends on the organization's operational requirements and support expectations.

From an architectural perspective, SONiC is a modular and container-based system. Major Continue reading

0

SONIC Part III: SONiC Introduction

SONiC is a vendor-neutral, Linux-based network operating system (NOS) that uses a database-driven architecture. Its software components run in multiple containers and exchange information through Redis. In SONiC, several named databases are defined for different functions, and these databases are mapped to Redis logical database IDs. Through this design, configuration data, application state, operational state, and ASIC-related state move between software layers by means of specialized processes.

Different hardware vendors may add their own platform integrations, transceiver support, monitoring utilities, or management workflows. However, the core SONiC architecture remains the same. This is one of the main reasons why SONiC knowledge, troubleshooting methods, and automation practices are transferable across different hardware platforms.

Vendor neutrality does not mean that every SONiC-based implementation behaves exactly the same in every operational detail. It means that different implementations follow the same architectural model. To organize information clearly, SONiC defines several named databases, each of which is mapped to a Redis logical database ID:

·       CONFIG_DB (Redis DB 4): Stores the user’s intended configuration.

·       APPL_DB (Redis DB 0): Stores application-level objects that are ready for processing by lower software layers.

·       STATE_DB (Redis DB 6): Stores operational state information about system Continue reading

0

SONiC Part II: Deploy a SONiC Switch Clos Topology

 

Introduction

 

This chapter explains how to create and deploy a simple SONiC-based Clos topology in WSL using Containerlab. First, we open VS Code from WSL to create and edit a topology definition file. Next, we build the topology by defining nodes (SONiC switches and Linux hosts) and the links between them. Before deploying the lab, we verify the wiring with Containerlab’s built-in topology graph. Finally, we deploy the topology and validate access to the nodes using both a Linux shell and the SONiC CLI (vtysh).

Phase 1: Integrate VS Code with WSL

There are a couple of ways to use VS Code with WSL. In this lab, we launch VS Code from the WSL terminal using code .. The first time you run this command, VS Code installs the VS Code Server components inside WSL and then opens a VS Code window connected to the Linux environment. After the installation completes, running code . from any directory opens that folder directly in VS Code.

0

SONiC Part 1: SONiC Lab Setup on Windows (Step-by-Step)

 

Introduction

 

This chapter explains how to build a SONiC virtual test environment on a Windows computer. First, we enable the required Windows features for WSL 2 and update and verify the WSL installation. Next, we install an Ubuntu distribution and validate that the Linux environment is working correctly, including basic resource checks (CPU, memory, and disk). After the Linux environment is ready, we install Docker Engine from Docker’s official repository and complete the required post-installation steps to run containers. We then install Containerlab, download the SONiC virtual switch image (docker-sonic-vs.gz), copy it into WSL, and load it into Docker. Finally, we install Visual Studio Code on Windows and connect it to WSL to make creating and editing the YAML topology files easier. The next chapter uses this environment to define and deploy a simple SONiC-based topology.

Phase 1: Enable Features for WSL

0

Packet trimming Deep Dive – Part IV

Receive Network Processing Unit (Rx NPU)

Figure 9-4 illustrates a simplified receive-side processing pipeline, starting from the moment a Packet Header Vector (PHV), constructed by the Rx IFG, is delivered to the Receive Network Processing Unit (Rx NPU).

When the PHV arrives at the Rx NPU, it is dispatched to one of the Run-to-Completion (RTC) cores in the Packet Processing Array (PPA). Each RTC core processes the packet within a single execution context, allowing parsing, classification, lookup, and queuing decisions to be resolved without intermediate handoffs between processing stages.

The first task of the RTC parser is to perform deep inspection of the packet headers. While the Rx IFG has already extracted basic Layer-2 and Layer-3 information, the RTC parser determines whether the packet is tunneled and whether the switch itself is the tunnel termination point. To demonstrate this behavior, consider a VXLAN-encapsulated packet. The outer Ethernet and IP headers are used to forward the packet through the underlay network. If the outer destination IP address matches one of the local switch IP addresses, the device identifies itself as the tunnel endpoint. The tunneling protocol is recognized by examining the UDP header, where destination port 4789 indicates VXLAN. After the Continue reading

0

Packet Trimming Deep Dive – Part III

Virtual Output Queue (VOQ)

The Silicon One VOQ Architecture

Instead of using dedicated deep interface buffers for packet queuing, Cisco Silicon One utilizes a Centralized Shared Memory architecture paired with a logical Virtual Output Queue (VOQ) mechanism. Because the VOQ concept is implemented within the Ingress (Rx) NPU entity, this queuing stage occurs after the initial ingress lookups but before the packet is switched across the internal fabric to the egress.

The VOQ model turns the traditional egress queuing model, where packets wait for serialization in a hardware buffer on the specific egress interface, upside down. While a VOQ is physically located on the ingress NPU, its ability to send traffic is controlled by the state of a small hardware Output Queue (OQ) on the egress interface.

Priority Mapping and Default State

As shown in Figure 9-3, a QoS policy can be created where a packet received on interface gi1/0/1 is assigned to Traffic Class 6 if the DSCP bits are set to EF (Expedited Forwarding). This configuration instantiates a VOQ specifically for that traffic class. In this hierarchy:

TC 7 (Control Plane/CS6): Mapped to OQ 1, the highest Strict Priority (Level 1).

TC 6 (DSCP-TRIMMED/EF): Mapped to OQ 2, Continue reading

0

Packet Trimming Deep Dive – Part II

Receive Interface Group (Rx IFG)

The Receive Interface Group (Rx IFG) is the ingress pre-processing stage that handles the incoming Ethernet bitstream before the packet enters the Packet Processing Array (PPA) of the Receive Network Processing Unit (Rx NPU) in the Cisco Silicon One architecture.

Processing begins at the Rx MAC. The Rx MAC reconstructs (“delimits”) the Ethernet frame from the Physical Coding Sublayer (PCS) bitstream and verifies frame integrity by computing a Frame Check Sequence (FCS) using the CRC-32 algorithm. If the computed FCS does not match the received FCS value, the frame is considered corrupted and is dropped immediately at ingress. If the CRC check succeeds, the frame is admitted for further processing. 

Shallow classification and Traffic Class mapping

After frame validation, the Rx IFG identifies the Ethernet MAC header and detects the presence of IEEE 802.1Q VLAN tags. The Rx IFG performs shallow classification to efficiently manage hardware resources before deeper protocol parsing and forwarding decisions are executed in the Rx NPU. When an IEEE 802.1Q VLAN tag is present, the Rx IFG extracts the Priority Code Point (PCP) bits from the VLAN tag and maps them to an Internal Continue reading