Archive

Category Archives for "sFlow"

Multipass

Multipass is a command line tool for running Ubuntu virtual machines on Mac or Windows. Multipass uses the native virtualization capabilities of the host operating system to simplify the creation of virtual machines.

Docker testbed and Docker DDoS testbed describe how to use containers to experiment with network visibility and control. However, not all software is amenable to running in containers, and so the ability to quickly create and configure virtual machines is a useful complement. This article demonstrates how to use Multipass to quickly build a virtual machine to run Mininet network emulation software.
multipass launch --name=mininet bionic
multipass exec mininet -- sudo apt update
multipass exec mininet -- sudo apt -y install mininet python-ryu
multipass exec mininet -- sudo apt -y install default-jre python-requests hping3
multipass exec mininet -- wget https://inmon.com/products/sFlow-RT/sflow-rt.tar.gz
multipass exec mininet -- tar -xzf sflow-rt.tar.gz
multipass exec mininet -- ./sflow-rt/get-app.sh sflow-rt mininet-dashboard

Run the above commands in a terminal to create the virtual machine. Multipass commands can easily be scripted to automate the creation and configuration of virtual machines.

multipass list
List the virtual machines.
Name                    State             IPv4             Image
test Running 192.168.64.2 Ubuntu 18.04 LTS

Continue reading

sFlow Monitoring for AI


A Proposal towards sFlow Monitoring Dashboards for AI-controlled NRENs is a recent talk by Mariam Kiran (Esnet) presented at the recent GÉANT Telemetry and Big Data Workshop.
In the talk, Miram describes the set open source tools (Netdata, Prometheus, Zabbix, Ntopng, and PerfSONAR) that they attempted to synthesize a complete picture of the network.
A number of tools were combined since each tool provides a different subset of the measurements needed to drive the AI controller. 
Integrating the data from the different sources was a challenge, but they were able to pull the data together into a single Grafana dashboard. Unfortunately, there was a lot of noise in legacy measurement schemes, making the data set unsuitable for training the AI controller.
The team decided to go toward sFlow, replacing the legacy monitoring tools with sFlow enabled devices, in order to generate the very clean data needed for machine learning.

For background, the talk, Real-time network telemetry for automation, describes why sFlow is uniquely suited to automation, providing the comprehensive, real-time, system-wide, visibility needed to make networked systems observable.

Real-time network telemetry for automation


The video discusses telemetry and requirements for network automation, providing an overview of sFlow measurement architecture and a discussion of recently added packet drop monitoring functionality, and ending with a live demonstration of GPU compute cluster analytics. The slides from the video are available here.

The video is part of recent talk Using Advanced Telemetry to Correlate GPU and Network Performance Issues [A21870] presented at the NVIDIA GTC conference

DENT


Introducing DENT OS, switchdev NOS for the rest of us talk, presented at the recent Open Source Summit, describes the Linux Foundation DENT project. The slides from the presentation are available.

Linux switchdev is an in-kernel driver model for switch devices which offload the forwarding (data) plane from the kernel. Integrating switch ASIC drivers in the Linux kernel makes switch ports appear as additional Linux network interfaces that can be configured and managed using standard Linux tools. 

DENT is an Ubuntu based Linux distribution that packages the drivers for switch hardware (fans, temperature sensors, ASIC, etc) along with the open source FRRouting routing protocol suite which includes protocol daemons for BGP, IS-IS, LDP, OSPF, PIM, and RIP. The FRRouting software uses the Linux netlink API to program Linux kernel packet forwarding, which on a hardware switch platform is offloaded by the switchdev driver to the ASIC for line rate forwarding.

A major benefit of DENT's approach to making Linux into the network operating system is that the same tools used to configure, manage and monitor Linux servers can also be used to manage network switches. In addition, a DENT virtual machine behaves in exactly the same way as Continue reading

DDoS protection of local address space


Docker DDoS testbed describes how to use Docker Desktop to experiment with Real-time DDoS mitigation using BGP RTBH and FlowSpec. In this article, Real-time BGP route analytics are used to automatically classify address space, replacing the manually configured classification in the previous example.

Routers supporting the sFlow extended_gateway extension include BGP routing information as part of the exported telemetry stream.  Real-time DDoS mitigation using BGP RTBH and FlowSpec describes how to configure an Arista router.
sflow sample 16384
sflow polling-interval 30
sflow extension bgp
sflow destination 10.0.0.70
sflow run
Adding the highlighted command to the sFlow configuration above enables the extended_gateway extension.

The alternative if the router doesn't support the extended_gateway extension, or doesn't support sFlow at all, sFlow-RT can be configured to match up sFlow streams from switches with routes discovered via BGP from routers in order to perform the route analytics needed to automatically classify DDoS attacks. The Docker DDoS testbed has separate sFlow and BGP agents, and so requires the use of this technique.

Start a Host sFlow agent using the pre-built sflow/host-sflow image:
docker run --rm -d -e "COLLECTOR=host.docker.internal" -e "SAMPLING=10" \
--net=host -v /var/run/docker.sock:/var/run/docker.sock:ro \
--name=host-sflow sflow/host-sflow
Continue reading

Docker DDoS testbed


Docker testbed describes how to use Docker Desktop to build a test network to experiment with real-time sFlow streaming telemetry and analytics. This article extends the testbed to experiment with distributed denial of service (DDoS) detection and mitigation techniques described in Real-time DDoS mitigation using BGP RTBH and FlowSpec.

Start a Host sFlow agent using the pre-built sflow/host-sflow image:
docker run --rm -d -e "COLLECTOR=host.docker.internal" -e "SAMPLING=10" \
--net=host -v /var/run/docker.sock:/var/run/docker.sock:ro \
--name=host-sflow sflow/host-sflow
Start ExaBGP using the pre-built sflow/exabgp image. ExaBGP connects to the sFlow-RT analytics software and displays BGP RTBH / Flowspec controls sent by sFlow-RT:
docker run --rm sflow/exabgp
In a second terminal window, start an instance of the sFlow-RT analytics software using the pre-built sflow/ddos-protect image:
GW=`docker network inspect bridge -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}'`

SUBNET=`docker network inspect bridge -f '{{range .IPAM.Config}}{{.Subnet}}{{end}}'`

docker run --rm -p 6343:6343/udp -p 8008:8008 -p 1179:1179 --name=sflow-rt \
sflow/ddos-protect -Dddos_protect.router=$GW -Dddos_protect.as=65001 \
-Dddos_protect.enable.flowspec=yes -Dddos_protect.group.local=$SUBNET \
-Dddos_protect.mode=automatic \
-Dddos_protect.udp_amplification.action=filter \
-Dddos_protect.udp_amplification.threshold=5000
Open the sFlow-RT dashboard at http://localhost:8008/
The sFlow Agents gauge confirms that sFlow is being received from the Host sFlow agent. Now access the Continue reading

Broadcom Mirror on Drop (MoD)

Networking Field Day 23 included a presentation by Bhaskar Chinni describing Broadcom's Mirror-on-Drop (MOD) capability. MOD capable hardware can generate a notification whenever a packet is dropped by the ASIC, reporting the packet header and the reason that the packet was dropped. MOD is supported by Trident 3, Tomahawk 3,  and Jericho 2 or later ASICs that are included in popular data center switches and widely deployed in data centers.

The recently published sFlow Dropped Packet Notification Structures specification adds drop notifications to industry standard sFlow telemetry export, complementing the existing push based counter and packet sampling measurements. The inclusion of drop monitoring in sFlow will allow the benefits of MOD to be fully realized, ensuring consistent end-to-end visibility into dropped packets across multiple vendors and network operating systems.

Using Advanced Telemetry to Correlate GPU and Network Performance Issues demonstrates how packet drop notifications from NVIDA Mellanox switches forms part of an integrated sFlow telemetry stream that provides the system wide observability needed to drive automation.

MOD instrumentation on Broadcom based switches provides the foundation needed for network vendors to integrate the Continue reading

Using Advanced Telemetry to Correlate GPU and Network Performance Issues


The image above was captured from the recent talk Using Advanced Telemetry to Correlate GPU and Network Performance Issues [A21870] presented at the NVIDIA GTC conference. The talk includes a demonstration of monitoring a high performance GPU compute cluster in real-time. The real-time dashboard provides an up to the second view of key performance metrics for the cluster.

This diagram shows the elements of the GPU compute cluster that was demonstrated. Cumulus Linux running on the switches reduces operational complexity by allowing you to run the same Linux operating system on the network devices as is run on the compute servers. sFlow telemetry is generated by the open source Host sFlow agent that runs on the servers and the switches, using standard Linux APIs to enable instrumentation and gather measurements. On switches, the measurements are offloaded to the ASIC to provide line rate monitoring.

Telemetry from all the switches and servers in the cluster is streamed to an sFlow-RT analyzer, which builds a real-time view of performance that can be used to drive operational dashboards and automation.

The Real-time GPU and network telemetry dashboard combines measurements from all the devices to provide view of cluster performance. Each of the three Continue reading

Cumulus Linux 4.2

Cumulus Linux is a network operating system for open networking hardware. Cumulus VX is a free virtual appliance that allows network engineers to experiment with Cumulus Linux and verify configurations before deploying into production. 
The Cumulus VX documentation describes how to build network topologies in KVM, VirtualBox, using VMWare hypervisors. If you want to run virtual machines locally, Cumulus in the Cloud is a free service that will allow you to access pre-built networks in the public cloud.

A key feature of Cumulus Linux is the use of the Linux kernel as the authoritative repository of network state. A result of this approach is that the behavior of a Cumulus Linux VX virtual appliance is the same as Cumulus Linux running on a hardware switch. For example, the open source FRR routing daemon shipped with Cumulus Linux uses the Linux netlink API to push routes to the kernel, which forwards packets in the virtual appliance. On a physical switch, routes are still pushed to the kernel, but kernel routing configuration is then offloaded to the switch ASIC so that packets bypass the kernel and are routed by hardware.

Cumulus Linux includes the open source Host sFlow agent. Here again, Continue reading

Real-time trending of dropped packets

Discard Browser is a recently released open source application running on the sFlow-RT real-time analytics engine. The software uses streaming analytics to trend dropped packets.
Using sFlow to monitor dropped packets describes the recently added packet drop monitoring functionality added to the open source Host sFlow agent. The article describes how to install and configure the agent on Linux-based platforms and stream industry standard sFlow telemetry to an sFlow collector.

Visibility into dropped packets describes instrumentation, recently added to the Linux kernel, that provides visibility into packets dropped by the kernel data path on a host, or dropped by a switch ASIC when packets are forwarded in hardware.  Extending sFlow to provide visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.

Packet discard monitoring complements sFlow's existing counter polling and packet sampling mechanisms and shares a common data model so that all three sources of data can be correlated.  For example, if packets are being discarded because of buffer exhaustion, the discard records don't necessarily Continue reading

Using sFlow to monitor dropped packets

Visibility into dropped packets describes instrumentation, recently added to the Linux kernel, that provides visibility into packets dropped by the kernel data path on a host, or dropped by a switch ASIC when packets are forwarded in hardware. This article describes integration of drop monitoring in the open source Host sFlow agent and inclusion of drop reporting as part of industry standard sFlow telemetry.

Extending sFlow to provide visibility into dropped packets offers significant benefits for network troubleshooting, providing real-time network-wide visibility into the specific packets that were dropped as well the reason the packet was dropped. This visibility instantly reveals the root cause of drops and the impacted connections.

Packet discard monitoring complements sFlow's existing counter polling and packet sampling mechanisms and shares a common data model so that all three sources of data can be correlated.  For example, if packets are being discarded because of buffer exhaustion, the discard records don't necessarily tell the whole story. The discarded packets may represent mice flows that are victims of an elephant flow. Packet samples will reveal the traffic that isn't being dropped and provide a more complete picture. Counter data adds additional information such as CPU load, interface speed, Continue reading

Visibility into dropped packets

Dropped packets have a profound impact on network performance and availability. Packet discards due to congestion can significantly impact application performance. Dropped packets due to black hole routes, expired TTLs, MTU mismatches, etc. can result in insidious connection failures that are time consuming and difficult to diagnose.

Devlink Trap describes recent changes to the Linux drop monitor service that provide visibility into packets dropped by switch ASIC hardware. When a packet is dropped by the ASIC, an event is generated that includes the header of the dropped packet and the reason why it was dropped. A hardware policer is used to limit the number of events generated by the ASIC to a rate that can be handled by the Linux kernel. The events are delivered to userspace applications using the Linux netlink service.

Running the dropwatch command line tool on an Ubuntu 20 system demonstrates the instrumentation:
[email protected]:~$ sudo dropwatch
Initializing null lookup method
dropwatch> set alertmode packet
Setting alert mode
Alert mode successfully set
dropwatch> start
Enabling monitoring...
Kernel monitoring activated.
Issue Ctrl-C to stop monitoring
drop at: __udp4_lib_rcv+0xae5/0xbb0 (0xffffffffb05ead95)
origin: software
input port ifindex: 2
timestamp: Wed Jul 15 23:57:36 2020 223253465 nsec
protocol: 0x800
length: 128
original Continue reading

Large flow marking using BGP Flowspec

Elephant Detection in Virtual Switches & Mitigation in Hardware discusses a VMware and Cumulus demonstration, Elephants and Mice, in which the virtual switch on a host detects and marks large "Elephant" flows and the hardware switch enforces priority queueing to prevent Elephant flows from adversely affecting latency of small "Mice" flows.

SDN and WAN optimization describes a presentation by Amin Vahdat describing Google's SDN based wide area network traffic engineering solution in which traffic prioritization allows Google to reduce costs by fully utilizing WAN bandwidth.

Deconstructing Datacenter Packet Transport describes how priority marking of packets associated with large flows can improve completion times for flows crossing the data center fabric. Simulation results presented in the paper show that prioritization of short flows over large flows can significantly improve throughput (reducing flow completion times by a factor of 5 or more at high loads).

This article demonstrates a self contained real-time Elephant flow marking solution that leverages the real-time visibility and control features available using commodity switch hardware.

The diagram shows the elements of the solution. An instance of the sFlow-RT real-time analytics engine receives streaming sFlow telemetry from a pair of edge routers. A mix of many small flows mixed Continue reading

Real-time network and system metrics as a service

The sFlow-RT real-time analytics engine receives industry standard sFlow telemetry as a continuous stream from network and host devices and coverts the raw data into useful measurements that can be be queried through a REST API. A single sFlow-RT instance can monitor the entire data center, providing a comprehensive view of performance, not just of the individual components, but of the data center as a whole.

This article is an interactive tutorial intended to familiarize the reader with the REST API. The examples can be run on a laptop using recorded data so that access to a live network is not required.

The data was captured from the leaf and spine test network shown above (described in Fabric View).
curl -O https://raw.githubusercontent.com/sflow-rt/fabric-view/master/demo/ecmp.pcap
First, download the captured sFlow data.

You will need to have a system with Java or Docker to run the sFlow-RT software.
curl -O https://inmon.com/products/sFlow-RT/sflow-rt.tar.gz
tar -xzf sflow-rt.tar.gz
./sflow-rt/get-app.sh sflow-rt browse-metrics
./sflow-rt/get-app.sh sflow-rt browse-flows
./sflow-rt/get-app.sh sflow-rt prometheus
./sflow-rt/start.sh -Dsflow.file=$PWD/ecmp.pcap
The above commands download and run sFlow-RT, with browse-metrics, browse-flows, and prometheus applications on a system with Java 1.8+ installed.
docker  Continue reading

NVIDIA, Mellanox, and Cumulus

Recent press releases, Riding a Cloud: NVIDIA Acquires Network-Software Trailblazer Cumulus and NVIDIA Completes Acquisition of Mellanox, Creating Major Force Driving Next-Gen Data Centers, describe NVIDIA's moves to provide high speed data center networks to connect compute clusters that use of their GPUs to accelerate big data workloads, including: deep learning, climate modeling, animation, data visualization, physics, molecular dynamics etc.

Real-time visibility into compute, network, and GPU infrastructure is required manage and optimize the unified infrastructure. This article explores how the industry standard sFlow technology supported by all three vendors can deliver comprehensive visibility.

Cumulus Linux simplifies operations, providing the same operating system, Linux, that runs on the servers. Cumulus Networks and Mellanox have a long history of working with the Linux community to integrate support for switches. The latest Linux kernels now include native support for network ASICs, seamlessly integrating with standard Linux routing (FRR, Quagga, Bird, etc), configuration (Puppet, Chef, Ansible, etc) and monitoring (collectd, netstat, top, etc) tools.

Linux 4.11 kernel extends packet sampling support describes enhancements to the Linux kernel to support industry standard sFlow instrumentation in network ASICs. Cumulus Linux and Mellanox both support the new Linux APIs. Cumulus Linux uses the open source Continue reading

Monitoring DDoS mitigation

Real-time DDoS mitigation using BGP RTBH and FlowSpec and Pushing BGP Flowspec rules to multiple routers describe how to deploy the ddos-protect application. This article focuses on how to monitor DDoS activity and control actions.

The diagram shows the elements of the solution. Routers stream standard sFlow telemetry to an instance of the sFlow-RT real-time analytics engine running the ddos-protect application. The instant a DDoS attack is detected, RTBH and / or Flowspec actions are pushed via BGP to the routers to mitigate the attack. Key metrics are published using the Prometheus exporter format over HTTP and events are sent using the standard syslog protocol.
The sFlow-RT DDoS Protect dashboard, shown above, makes use of the Prometheus time series database and the Grafana metrics visualization tool to track DDoS attack mitigation actions.
The sFlow-RT Countries and Networks dashboard, shown above, breaks down traffic by origin network and country to provide an indication of the source of attacks.  Flow metrics with Prometheus and Grafana describes how to build additional dashboards to provide additional insight into network traffic.
In this example, syslog events are directed to an Elasticsearch, Logstash, and Kibana (ELK) stack where they are archived, queried, and analyzed. Grafana Continue reading

Pushing BGP Flowspec rules to multiple routers

Real-time DDoS mitigation using BGP RTBH and Flowspec describes the open source DDoS Protect application. The software runs on the sFlow-RT real-time analytics engine, which receives industry standard sFlow telemetry from routers and pushes controls using BGP. A recent enhancement to the application pushes controls to multiple routers in order to protect networks with redundant edge routers.
ddos_protect.router=10.0.0.96,10.0.0.97
Configuring multiple BGP connections is simple, the ddos_protect.router configuration option has been extended to accept a comma separated list of IP addresses for the routers that will be connecting to the controller.
Alternatively, a BGP Flowspec/RTBH reflector can be used to propagate the controls. Flowspec is a recent addition to open source BGP software, FRR and Bird, and it should be possible to use this software to reflect Flowspec controls. A reflector can be a useful place to implement policies that direct controls to specific enforcement devices.

Support for multiple BGP connections in the DDoS Protect application reduces the complexity of simple deployments by removing the requirement for a reflector. Controls are pushed to all devices, but differentiated policies can still be implemented by configuring each device's response to controls.

Kubernetes testbed

The sFlow-RT real-time analytics platform receives a continuous telemetry stream from sFlow Agents embedded in network devices, hosts and applications and converts the raw measurements into actionable metrics, accessible through open APIs, see Writing Applications.

Application development is greatly simplified if you can emulate the infrastructure you want to monitor on your development machine. Docker testbed describes a simple way to develop sFlow based visibility solutions. This article describes how to build a Kubernetes testbed to develop and test configurations before deploying solutions into production.
Docker Desktop provides a convenient way to set up a single node Kubernetes cluster, just select the Enable Kubernetes setting and click on Apply & Restart.

Create the following sflow-rt.yml file:
apiVersion: v1
kind: Service
metadata:
name: sflow-rt-sflow
spec:
type: NodePort
selector:
name: sflow-rt
ports:
- protocol: UDP
port: 6343
---
apiVersion: v1
kind: Service
metadata:
name: sflow-rt-rest
spec:
type: LoadBalancer
selector:
name: sflow-rt
ports:
- protocol: TCP
port: 8008
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: sflow-rt
spec:
replicas: 1
selector:
matchLabels:
name: sflow-rt
template:
metadata:
labels:
name: sflow-rt
spec:
containers:
- name: sflow-rt
image: sflow/prometheus:latest
ports:
- name: http
protocol: TCP
containerPort: 8008
- name: sflow
protocol: UDP
containerPort: 6343
Run the Continue reading

SFMIX San Francisco shelter in place

A shelter in place order restricted San Francisco residents to their homes beginning at 12:01 a.m. on March 17, 2020. Many residents work for Bay Area technology companies such as Salesforce, Facebook, Twitter, Google, Netflix and Apple. Employees from these companies are able to, and have been instructed to, work remotely from their homes. In addition, other housebound residents are making use of social networking to keep in touch with friends and family as well as streaming media and online gaming for entertainment.

The traffic trend chart above from the San Francisco Metropolitan Internet Exchange (SFMIX) shows the change in network traffic that has resulted from the shelter in place order. Peak traffic has increased by around 10Gbit/s (a 25% increase) and continues throughout the day (whereas peaks previously occurred in the evenings).

The SFMIX network directly connects a number of data centers in the Bay Area and the member organizations that peer from those data centers.  Peering through the exchange network keeps traffic local by directly connecting companies with their employees and customers and avoiding potentially congested service provider networks.
SFMIX recently finished a network upgrade to 100Gbit/s Arista switches and all fiber Continue reading

Ubuntu 18.04

Ubuntu 18.04 comes with Linux kernel version 4.15. This version of the kernel includes efficient in-kernel packet sampling that can be used to provide network visibility for production servers running network heavy workloads, see Berkeley Packet Filter (BPF).
This article provides instructions for installing and configuring the open source Host sFlow agent to remotely monitor servers using the industry standard sFlow protocol. The sFlow-RT real-time analyzer is used to demonstrate the capabilities of sFlow telemetry.

Find the latest Host sFlow version on the Host sFlow download page.
wget https://github.com/sflow/host-sflow/releases/download/v2.0.25-3/hsflowd-ubuntu18_2.0.25-3_amd64.deb
sudo dpkg -i hsflowd-ubuntu18_2.0.25-3_amd64.deb
sudo systemctl enable hsflowd
The above commands download and install the software.
sflow {
collector { ip=10.0.0.30 }
pcap { speed=1G-1T }
tcp { }
systemd { }
}
Edit the /etc/hsflowd.conf file. The above example sends sFlow to a collector at 10.0.0.30, enables packet sampling on all network adapters, adds TCP performance information, and exports metrics for Linux services. See Configuring Host sFlow for Linux for the complete set of configuration options.
sudo systemctl restart hsflowd
Restart the Host sFlow daemon to start streaming telemetry to Continue reading
1 2 3 11