SONiC Deep Dive: Intra-Switch Inter-VLAN Routing and Packet Forwarding

The title of this section might seem a little overwhelming, but it reflects the focus of the chapter. The SONiC Redis database programming pipeline is examined through the configuration of two Layer-2 VLANs, VLAN 10 and VLAN 20, together with their respective Layer-3 routing interfaces.

VLAN 10 and VLAN 20 represent two separate Layer-2 broadcast domains, with each VLAN associated with its own IP subnet. VLAN 10 uses the 10.0.10.0/24 subnet, while VLAN 20 uses the 10.0.20.0/24 subnet. Because the two hosts belong to different broadcast domains and IP subnets, communication between them requires Layer-3 routing. In this example, the routing is performed locally by Leaf-1 through the routing interfaces configured for the two VLANs.

Figure 5-15 depicts the basic building blocks and IP addressing scheme used in this example. PC1, with IP address 10.0.10.101/24, belongs to VLAN 10 and is connected to the Leaf-1 Ethernet0 interface. PC2, with IP address 10.0.20.101/24, belongs to VLAN 20 and is connected to the Leaf-1 Ethernet1 interface. The routing interfaces for VLAN 10 and VLAN 20 use IP addresses 10.0.10.1/24 and 10.0.20.1/24, respectively.

The Continue reading

SONiC Deep Dive: Binding Interface to Vlan

Step 3: Associate Ethernet0 to VLAN 10

After configuring VLAN 10 and bringing up interface Ethernet0, the next step is to make Ethernet0 an untagged member of VLAN 10.

GONFIG_DB and APPL_DB

Figure 5-11 gives a conceptual view of the Redis database updates resulting from the SONiC command:

sudo config vlan member add 10 Ethernet0 --untagged

The command defines our intent: Ethernet0 should participate in VLAN 10 as an untagged (access) port. For an untagged VLAN member, frames received from the host arrive without an 802.1Q tag and are associated with VLAN 10 based on the port's VLAN membership. In the opposite direction, frames transmitted from VLAN 10 toward Ethernet0 are sent without an 802.1Q tag.

The command updates several related entries in CONFIG_DB. The PORT|Ethernet0 entry contains the port's access VLAN configuration:

PORT|Ethernet0

    access_vlan = 10

The VLAN|Vlan10 entry represents the VLAN itself and contains a members@ field that records Ethernet0 as a member:

VLAN|Vlan10

   members@ = Ethernet0   

   vlanid = 10

The @ character in members@ is part of the SONiC CONFIG_DB schema convention. The members@ value is a configuration representation of the relationship between the VLAN and its members.

The Continue reading

SONiC Deep Dive: Interface State Change Pipeline

Enable Interface Ethernet0

After creating VLAN 10, the next step is to enable interface Ethernet0. The baseline configuration for Ethernet0 already exists in the PORT table of CONFIG_DB. This configuration is populated during the SONiC switch startup process using the platform configuration and configuration files, including config_db.json. When the switch boots, the configuration from config_db.json is loaded into CONFIG_DB, which is implemented as a Redis database.

GONFIG_DB and APPL_DB

The SONiC command config interface startup Ethernet0 does not create the complete PORT|Ethernet0 entry to the CONFIG_DB. Instead, it changes the administrative state of the existing port configuration. In this example, the admin_status field of PORT|Ethernet0 is changed to up (1).

After admin_status is changed, Redis generates a notification for applications subscribed to the PORT table in CONFIG_DB (2). The Redis notification indicates that the configuration entry has changed. It identifies the affected key and the operation that occurred rather than containing the complete PORT|Ethernet0 configuration. portmgrd, running in the SWSS container, is one of the SONiC components involved in processing port configuration. portmgrd uses the notification to identify the changed key and retrieves the current field/value data from CONFIG_DB. It then processes the configuration and prepares Continue reading

SONiC Deep Dive: Configuring VLAN 10 from CLI to ASIC

Introduction

In this chapter, we examine what happens under the hood when we configure a Layer-2 VLAN on a SONiC switch, enable two host-facing interfaces, and associate them with the previously created VLAN as untagged access ports. Figure 5-1 illustrates the conceptual configuration in which Host-1 and Host-2 are connected to interfaces Eth1/1 and Eth1/2, respectively. Both interfaces are members of VLAN 10, which at this stage functions as a Layer-2 broadcast domain. The host-facing interfaces are single-lane 25-Gigabit Ethernet interfaces.

We begin by creating VLAN 10 and examining the complete SONiC processing sequence, from the user's VLAN configuration to ASIC programming. Next, we study what happens when the interfaces are enabled and associated with VLAN 10. Finally, we verify that Host-1 and Host-2 have IP connectivity through Leaf-1. Although this is a very basic configuration example, many SONiC components participate in the process. The database examples presented in this chapter are taken from the lab environment. The communication between processes is explained at a conceptual level because the focus is on SONiC operation rather than the underlying Linux communication mechanisms.

Figure 5-1: Host Connectivity.

Step 1: Configure Vlan 10

Figure 5-2 depicts the conceptual sequence of Continue reading

SONiC – ASIC Programming: Port Characteristics

 The previous section described at a high level how the config_db.json file is loaded into the Redis CONFIG_DB database in the Database container. Next we look at the topic in slightly more detail. First, we examine how physical port information, such as the port's default speed and lane configuration, is processed during the initial boot before it is programmed into the switch ASIC.

Step 1: Creating a Config_db.json and Updating CONFIG_DB by Sonic-cogent

Figure 4-2 illustrates how platform-specific port information becomes the PORT and BREAKOUT_CFG entries stored in the Redis CONFIG_DB database. During the image initialization process, sonic-cfggen uses the platform-specific port_config.ini file (1) together with other platform configuration files and templates to generate the port-related entries in the config_db.json file (2). The port_config.ini file defines the logical, expected physical characteristics of the switch ports, including the front-panel alias names, lane mappings, index values, subport values, and default port speeds. After the config_db.json file has been generated, its contents are loaded into the Redis CONFIG_DB database during the SONiC boot process (3).

Figure 4-2 also shows the DEVICE_METADATA table, which contains platform-specific information such as the base MAC address, switch_type, Continue reading

SONiC: Populating CONFIG_DB

Introduction

Before a switch can forward data traffic, its ASIC must first be programmed with the device's port configuration. This includes information such as the number of front-panel ports, their supported speeds, the number of lanes assigned to each port, the speed of each lane, and the administrative and operational state of every port.

SONiC relies heavily on a Redis-based database model. During startup, the configuration stored in config_db.json is loaded into CONFIG_DB. The *mgrd daemons running inside the SWSS container then receive the changes relevant to them through Redis Pub/Sub notifications. After a daemon has processed its part of the configuration, it publishes the corresponding application-level state to APPL_DB.

The orchagent process in the SWSS container monitors changes in APPL_DB, translates the application-level state into hardware objects suitable for ASIC programming, and publishes the resulting hardware objects to ASIC_DB. The actual ASIC programming takes place in the next stage: the syncd container monitors ASIC_DB changes and passes them through the SAI interface to the vendor-specific SDK, which interacts with the ASIC driver stack to program the physical switch ASIC.

This chapter focuses on port programming because it clearly illustrates the complete programming pipeline. Other configuration objects follow the Continue reading

Chapter 3: SONiC Startup Process

Figure 3-1 illustrates a simplified SONiC container startup sequence. First, systemd reads the service unit files and evaluates their dependency and ordering directives. It then starts docker.service, making the Docker daemon available. After that, SONiC service containers are started according to the dependencies and ordering rules defined in their service unit files.

In this simplified example, database.service starts early because many SONiC services rely on the Redis databases to exchange configuration, state, and event information. Services such as pmon, syncd, and swss are then started according to their own dependencies and ordering rules. Higher-level SONiC service containers, such as bgp, lldp, teamd, and snmp, are started after the lower-level services they depend on are available.

It is also important to distinguish between host-level service management and process management inside containers. systemd is responsible for starting the host-level service units that create and manage SONiC service containers. After a container has started, the processes inside the container are launched and supervised by the container's own initialization logic, which in many SONiC containers is based on supervisord. For example, after the bgp container has started, processes such as bgpd, zebra, and fpmsyncd are started inside the container. Similarly, the swss Continue reading

Chapter 2: Installing SONiC NOS

ONIE-Based SONiC Installation

Many switch vendors have added SONiC NOS support to at least part of their switch portfolio. Depending on the vendor and switch model, customers may be able to order a switch with a vendor-customized SONiC version that is supported at the same level as the vendor's own network operating system. Some vendors also allow customers to run the community-based SONiC distribution.

The support model for Community SONiC depends on the vendor. Some hardware vendors provide full support, while others provide no support at all. Compared with vendor-specific SONiC distributions, Community SONiC provides greater flexibility because it can be customized, rebuilt, and adapted to customer requirements. However, running a Community SONiC deployment without vendor support or in-house expertise is generally not a recommended operating model.

Community SONiC is typically installed by using ONIE (Open Network Install Environment) [1], a small open-source installation environment that provides a standardized method for installing network operating systems on supported switches. Figure 2-1 illustrates a conceptual ONIE-based Community SONiC installation process.

If the switch is delivered with a vendor-specific SONiC distribution already installed, it may boot directly into that operating system without requiring a separate ONIE installation workflow. For Community SONiC deployments, Continue reading

Chapter 1: SONiC Fundamentals

Introduction

SONiC (Software for Open Networking in the Cloud) is a Linux-based open-source network operating system that was originally developed at Microsoft and is now maintained by a broader open-source community. Its core idea is that the same network operating system can run on switch platforms from multiple hardware vendors. This reduces vendor lock-in and provides a more consistent operational model across different environments.

SONiC can also be viewed as an abstraction layer between network operators and the underlying switch hardware. Instead of learning and managing several vendor-specific operating systems, operators can use a common software architecture and management model across different switch platforms. This simplifies network operations, automation, monitoring, and telemetry collection. It can also reduce operational errors caused by configuration differences between platforms and make it easier to onboard new engineers.

Organizations can choose the hardware platform that best meets their technical, operational, and business requirements without being tied to a single software ecosystem. Some vendors provide commercially supported SONiC distributions together with professional support services, while others support community-based deployments or customer-tailored implementations. The appropriate model depends on the organization's operational requirements and support expectations.

From an architectural perspective, SONiC is a modular and container-based system. Major Continue reading

SONIC Part III: SONiC Introduction

SONiC is a vendor-neutral, Linux-based network operating system (NOS) that uses a database-driven architecture. Its software components run in multiple containers and exchange information through Redis. In SONiC, several named databases are defined for different functions, and these databases are mapped to Redis logical database IDs. Through this design, configuration data, application state, operational state, and ASIC-related state move between software layers by means of specialized processes.

Different hardware vendors may add their own platform integrations, transceiver support, monitoring utilities, or management workflows. However, the core SONiC architecture remains the same. This is one of the main reasons why SONiC knowledge, troubleshooting methods, and automation practices are transferable across different hardware platforms.

Vendor neutrality does not mean that every SONiC-based implementation behaves exactly the same in every operational detail. It means that different implementations follow the same architectural model. To organize information clearly, SONiC defines several named databases, each of which is mapped to a Redis logical database ID:

·       CONFIG_DB (Redis DB 4): Stores the user’s intended configuration.

·       APPL_DB (Redis DB 0): Stores application-level objects that are ready for processing by lower software layers.

·       STATE_DB (Redis DB 6): Stores operational state information about system Continue reading

SONiC Part II: Deploy a SONiC Switch Clos Topology

Introduction

This chapter explains how to create and deploy a simple SONiC-based Clos topology in WSL using Containerlab. First, we open VS Code from WSL to create and edit a topology definition file. Next, we build the topology by defining nodes (SONiC switches and Linux hosts) and the links between them. Before deploying the lab, we verify the wiring with Containerlab’s built-in topology graph. Finally, we deploy the topology and validate access to the nodes using both a Linux shell and the SONiC CLI (vtysh).

Phase 1: Integrate VS Code with WSL

There are a couple of ways to use VS Code with WSL. In this lab, we launch VS Code from the WSL terminal using code .. The first time you run this command, VS Code installs the VS Code Server components inside WSL and then opens a VS Code window connected to the Linux environment. After the installation completes, running code . from any directory opens that folder directly in VS Code.

SONiC Part 1: SONiC Lab Setup on Windows (Step-by-Step)

Introduction

This chapter explains how to build a SONiC virtual test environment on a Windows computer. First, we enable the required Windows features for WSL 2 and update and verify the WSL installation. Next, we install an Ubuntu distribution and validate that the Linux environment is working correctly, including basic resource checks (CPU, memory, and disk). After the Linux environment is ready, we install Docker Engine from Docker’s official repository and complete the required post-installation steps to run containers. We then install Containerlab, download the SONiC virtual switch image (docker-sonic-vs.gz), copy it into WSL, and load it into Docker. Finally, we install Visual Studio Code on Windows and connect it to WSL to make creating and editing the YAML topology files easier. The next chapter uses this environment to define and deploy a simple SONiC-based topology.

Phase 1: Enable Features for WSL

WSL 2 requires two Windows features to be enabled. The first feature, Microsoft-Windows-Subsystem-Linux (Example 1-1), enables WSL. The second feature, VirtualMachinePlatform (Example 1-2), is required to run WSL 2.

In this example, both features are enabled using Microsoft PowerShell (Run as Administrator) with the dism.exe command. The options used are:

·         Continue reading

Packet trimming Deep Dive – Part IV

Receive Network Processing Unit (Rx NPU)

Figure 9-4 illustrates a simplified receive-side processing pipeline, starting from the moment a Packet Header Vector (PHV), constructed by the Rx IFG, is delivered to the Receive Network Processing Unit (Rx NPU).

When the PHV arrives at the Rx NPU, it is dispatched to one of the Run-to-Completion (RTC) cores in the Packet Processing Array (PPA). Each RTC core processes the packet within a single execution context, allowing parsing, classification, lookup, and queuing decisions to be resolved without intermediate handoffs between processing stages.

The first task of the RTC parser is to perform deep inspection of the packet headers. While the Rx IFG has already extracted basic Layer-2 and Layer-3 information, the RTC parser determines whether the packet is tunneled and whether the switch itself is the tunnel termination point. To demonstrate this behavior, consider a VXLAN-encapsulated packet. The outer Ethernet and IP headers are used to forward the packet through the underlay network. If the outer destination IP address matches one of the local switch IP addresses, the device identifies itself as the tunnel endpoint. The tunneling protocol is recognized by examining the UDP header, where destination port 4789 indicates VXLAN. After the Continue reading

Packet Trimming Deep Dive – Part III

Virtual Output Queue (VOQ)

The Silicon One VOQ Architecture

Instead of using dedicated deep interface buffers for packet queuing, Cisco Silicon One utilizes a Centralized Shared Memory architecture paired with a logical Virtual Output Queue (VOQ) mechanism. Because the VOQ concept is implemented within the Ingress (Rx) NPU entity, this queuing stage occurs after the initial ingress lookups but before the packet is switched across the internal fabric to the egress.

The VOQ model turns the traditional egress queuing model, where packets wait for serialization in a hardware buffer on the specific egress interface, upside down. While a VOQ is physically located on the ingress NPU, its ability to send traffic is controlled by the state of a small hardware Output Queue (OQ) on the egress interface.

Priority Mapping and Default State

As shown in Figure 9-3, a QoS policy can be created where a packet received on interface gi1/0/1 is assigned to Traffic Class 6 if the DSCP bits are set to EF (Expedited Forwarding). This configuration instantiates a VOQ specifically for that traffic class. In this hierarchy:

TC 7 (Control Plane/CS6): Mapped to OQ 1, the highest Strict Priority (Level 1).

TC 6 (DSCP-TRIMMED/EF): Mapped to OQ 2, Continue reading

Packet Trimming Deep Dive – Part II

Receive Interface Group (Rx IFG)

The Receive Interface Group (Rx IFG) is the ingress pre-processing stage that handles the incoming Ethernet bitstream before the packet enters the Packet Processing Array (PPA) of the Receive Network Processing Unit (Rx NPU) in the Cisco Silicon One architecture.

Processing begins at the Rx MAC. The Rx MAC reconstructs (“delimits”) the Ethernet frame from the Physical Coding Sublayer (PCS) bitstream and verifies frame integrity by computing a Frame Check Sequence (FCS) using the CRC-32 algorithm. If the computed FCS does not match the received FCS value, the frame is considered corrupted and is dropped immediately at ingress. If the CRC check succeeds, the frame is admitted for further processing. 

Shallow classification and Traffic Class mapping

After frame validation, the Rx IFG identifies the Ethernet MAC header and detects the presence of IEEE 802.1Q VLAN tags. The Rx IFG performs shallow classification to efficiently manage hardware resources before deeper protocol parsing and forwarding decisions are executed in the Rx NPU. When an IEEE 802.1Q VLAN tag is present, the Rx IFG extracts the Priority Code Point (PCP) bits from the VLAN tag and maps them to an Internal Continue reading

Packet Trimming Deep Dive – Part I

 Introduction

The previous chapter introduced the Ultra Ethernet (UE) Transport Layer and its endpoint-centric congestion control mechanisms: Network Signaled Congestion Control (NSCC) and Receiver Credit-based Congestion Control (RCCC). This chapter moves down to the UE Network Layer and introduces Packet Trimming (PT).

While node-based approaches rely on NIC-to-NIC feedback loops, Packet Trimming allows network switches to actively intervene during periods of high utilization. Instead of silently dropping packets under congestion, the network provides an explicit and fast signal that enables immediate recovery.

The primary goal of Packet Trimming is to prevent incast congestion, a situation in which multiple ingress ports simultaneously overwhelm a single egress port. In AI and HPC workloads, many-to-one traffic patterns are common—for example, when multiple workers send data to a single parameter server. Under these conditions, egress buffers can be exhausted very quickly. In a best-effort network, this typically results in tail drops. The receiver then waits for a retransmission timeout, which introduces long tail latency and disrupts synchronization across distributed workloads. Packet Trimming replaces this silent packet loss with an explicit congestion signal that travels faster than the data itself.

The process begins at the source UE node. The NIC marks outgoing data packets with Continue reading

Ultra Ethernet: Receiver Credit-based Congestion Control (RCCC)

 Introduction

Receiver Credit-Based Congestion Control (RCCC) is a cornerstone of the Ultra Ethernet transport architecture, specifically designed to eliminate incast congestion. Incast occurs at the last-hop switch when the aggregate data rate from multiple senders exceeds the egress interface capacity of the target’s link. This mismatch leads to rapid buffer exhaustion on the outgoing interface, resulting in packet drops and severe performance degradation.

The RCCC Mechanism

Figure 8-1 illustrates the operational flow of the RCCC algorithm. In a standard scenario without credit limits, source Rank 0 and Rank 1 might attempt to transmit at their full 100G line rates simultaneously. If the backbone fabric consists of 400G inter-switch links, the core utilization remains a comfortable 50% (200G total traffic). However, because the target host link is only 100G, the last-hop switch (Leaf 1B-1) becomes an immediate bottleneck. The switch is forced to queue packets that cannot be forwarded at the 100G egress rate, eventually triggering incast congestion and buffer overflows.

While "incast" occurs at the egress interface and can resemble head-of-line blocking, it is fundamentally a "fan-in" problem where multiple sources converge on a single receiver. Under RCCC, standard Explicit Congestion Notification (ECN) on the last-hop switch's egress interface is Continue reading

Ultra Ethernet: NSCC Destination Flow Control

Figure 6-14 depicts a demonstrative event where Rank 4 receives seven simultaneous flows (1). As these flows are processed by their respective PDCs and handed over to the Semantic Sublayer (2), the High-Bandwidth Memory (HBM) Controller becomes congested. Because HBM must arbitrate multiple fi_write RMA operations requiring concurrent memory bank access and state updates, the incoming packet rate quickly exceeds HBM’s transactional retirement rate. 

This causes internal buffers at the memory interface to fill, creating a local congestion event (3). To prevent buffer overflow, which would lead to dropped packets and expensive RMA retries, the receiver utilizes NSCC to move the queuing "pain" back to the source. This is achieved by using pds.rcv_cwnd_pend parameter of the ACK_CC header (4). The parameter operates on a scale of 0 to 127; while zero is ignored, a value of 127 triggers the maximum possible rate decrement. In this scenario, a value of 64 is utilized, resulting in a 50% penalty relative to the newly acknowledged data.

Rather than directly computing a new transport rate, the mechanism utilizes a three-phase process to define a restricted Congestion Window (CWND). This reduction in CWND inherently forces the source to drain its inflight bucket to Continue reading

Ultra Ethernet: Inflight Bytes and CWND Adjustment

Inflight Packet Adjustment

Figure 6-12 depicts the ACK_CC header structure and fields. When NSCC is enabled in the UET node, the PDS must use the pds.type ACK_CC in the prologue header, which serves as the common header structure for all PDS messages. Within the actual PDS ACK_CC header, the pds.cc_type must be set to CC_NSCC. The pds.ack_cc_state field describes the values and states for service_time, rc (restore congestion CWND), rcv_cwnd_pend, and received_bytes. The source specifically utilizes the received_bytes parameter to calculate the updated state for inflight packets.

The CCC computes the reduction in the inflight state by subtracting the rcvd_bytes value received in previous ACK_CC messages from the rcvd_bytes value carried within the latest ACK_CC message. As illustrated in Figure 6-12, the inflight state is decreased by 4,096 bytes, which is the delta between 16,384 and 12,288 bytes.

Recap: In order to transport data to network, the Inflight bytes must be less than CWND size.

Figure 6-12: NSCC: Inflight Bytes adjustment.

CWND Adjustment

A single, shared Congestion Window (CWND) regulates the total volume of bytes across all PDCs that are permitted for transmission to the backend network. The transport rate and network performance are continuously monitored and Continue reading

Ultra Ethernet: Network-Signaled Congestion Control (NSCC) – Overview

Network-Signaled Congestion Control (NSCC)

The Network-Signaled Congestion Control (NSCC) algorithm operates on the principle that the network fabric itself is the best source of truth regarding congestion. Rather than waiting for packet loss to occur, NSCC relies on proactive feedback from switches to adjust transmission rates in real time. The primary mechanism for this feedback is Explicit Congestion Notification (ECN) marking. When a switch interface's egress queue begins to build up, it employs a Random Early Detection (RED) logic to mark specific packets. Once the buffer’s Minimum Threshold is crossed, the switch begins randomly marking packets by setting the last two bits of the IP header’s Type of Service (ToS) field to the CE (11) state. If the congestion worsens and the Maximum Threshold is reached, every packet passing through that interface is marked, providing a clear and urgent signal to the endpoints.

The practical impact of this mechanism is best illustrated by a hash collision event, such as the one shown in Figure 6-10. In this scenario, multiple GPUs on the left-hand side of the fabric transmit data at line rate. Due to the specific entropy of these flows, the ECMP hashing algorithms on leaf switches 1A-1 and 1A-2 Continue reading