New Feature in Cumulus Linux 2.2: sFlow
sFlow is an open protocol, newly supported in Cumulus Linux 2.2, that enables a collector to determine what is going on in a complex network.
It is used to collect statistics, such as packet counts, error counts, CPU usage, etc from a large number of individual switches. What is especially interesting is that it can be used to collect sampled packets (usually only the first n bytes, containing the header), along with some metadata about those packets.
Bringing sFlow to Cumulus Linux was particuarly easy, because “hsflowd” was already available for implementing sFlow support on Linux servers. We were able to reuse that existing code, with extremely minimal modification, to implement sFlow on our Linux based switches.
sFlow allows a collector to get a statistical view of what is going on in a collection of switches, approaching per-flow granularity. This is extremely useful information to present to users for capacity planning and debugging purposes, but things really get interesting when the collector can make decisions based on the information.
For example, our friends at inMon implemented detection of elephant flows (high bandwidth), followed by marking those flows on the switch at network ingress for special QoS handling. This nearly Continue reading
DDoS mitigation with Cumulus Linux
 |
| Figure 1: Real-time SDN Analytics for DDoS mitigation |
This article builds on the test setup described in RESTful control of Cumulus Linux ACLs in order to implement the ONS 2014 SDN Idol winning distributed denial of service (DDoS) mitigation solution - Real-time SDN Analytics for DDoS mitigation.
The following sFlow-RT application implements basic DDoS mitigation functionality:
include('extras/json2.js');
// Define large flow as greater than 100Mbits/sec for 1 second or longer
var bytes_per_second = 100000000/8;
var duration_seconds = 1;
var id = 0;
var controls = {};
setFlow('udp_target',
{keys:'ipdestination,udpsourceport', value:'bytes',
filter:'direction=egress', t:duration_seconds}
);
setThreshold('attack',
{metric:'udp_target', value:bytes_per_second, byFlow:true, timeout:4,
filter:{ifspeed:[1000000000]}}
);
setEventHandler(function(evt) {
if(controls[evt.flowKey]) return;
var rulename = 'ddos' + id++;
var keys = evt.flowKey.split(',');
var acl = [
'[iptables]',
'# block UDP reflection attack',
'-A FORWARD --in-interface swp+ -d ' + keys[0]
+ ' -p udp --sport ' + keys[1] + ' -j DROP'
];
http('http://'+evt.agent+':8080/acl/'+rulename,
'put','application/json',JSON.stringify(acl));
controls[evt.flowKey] = {
agent:evt.agent,
dataSource:evt.dataSource,
rulename:rulename,
Continue reading
Docker performance monitoring
 |
| Visibility and the software defined data center |
The talk provides additional background on the sFlow standard and case studies. The remainder of this article describes how to use Host sFlow to monitor a Docker server pool.
First, download, compile and install the Host sFlow agent on a Docker host (Note: The agent needs to Continue reading
Microsoft Office 365 outage
6/24/2014 Information Week - Microsoft Exchange Online Suffers Service Outage, "Service disruptions with Microsoft's Exchange Online left many companies with no email on Tuesday."The following entry on the Microsoft 365 community forum describes the incident:
====================================The closure summary shows that operators took 8 hour 39 minutes to manually diagnose and remediate the problem with degraded networking infrastructure. The network related outage described in this example is not an isolated incident; other incidents described on this blog include: Packet loss, Amazon EC2 outage, Gmail outage, Delay vs utilization for Continue reading
Closure Summary: On Tuesday, June 24, 2014, at approximately 1:11 PM UTC, engineers received reports of an issue in which some customers were unable to access the Exchange Online service. Investigation determined that a portion of the networking infrastructure entered into a degraded state. Engineers made configuration changes on the affected capacity to remediate end-user impact. The issue was successfully fixed on Tuesday, June 24, 2014, at 9:50 PM UTC.
Customer Impact: Affected customers were unable to access the Exchange Online service.
Incident Start Time: Tuesday, June 24, 2014, at 1:11 PM UTC
Incident End Time: Tuesday, June 24, 2014, at 9:50 PM UTC
=====================================
RESTful control of Cumulus Linux ACLs
 |
| Figure 1: Elephants and Mice |
This article demonstrates a self contained real-time Elephant flow marking solution that leverages the visibility and control features of Cumulus Linux.
SDN fabric controller for commodity data center switches provides some background on the capabilities of the commodity switch hardware used to run Cumulus Linux. The article describes how the measurement and control capabilities of the hardware can be used to maximize data center fabric performance:
- Visibility: Cumulus recently added sFlow support, providing the network wide visibility needed for control - see Cumulus Networks, sFlow and data center automation.
- Control: Cumulus Linux Netfilter (ACLs) describes how standard Linux ebtables / iptables configuration files are used to manage the ACL/policy capabilities of the hardware.
For example, the following command creates a filter called Continue reading
Cumulus Networks, sFlow and data center automation
The Cumulus Linux platform makes it possible to run the same open source agent on switches, servers, and hypervisors - providing unified end-to-end visibility across the data center. The open networking model that Cumulus is pioneering offers exciting opportunities. Cumulus Linux allows popular open source server orchestration tools to also manage the network, and the combination of real-time, data center wide analytics with orchestration make it possible to create self-optimizing data centers.
Install and configure Host sFlow agent
The following command installs the Host sFlow agent on a Cumulus Linux switch:sudo apt-get install hsflowdNote: Network managers may find this command odd since it is usually not possible to install third party software on switch hardware. However, what is even more radical is that Cumulus Linux allows users to download source Continue reading
SDN fabric controller for commodity data center switches
 |
| Figure 1: Rise of merchant silicon |
This article will examine some of the factors leading to commoditization of network hardware and the role that software defined networking (SDN) plays in coordinating hardware resources to deliver increased network efficiency.
 |
| Figure 2: Fabric: A Retrospective on Evolving SDN |
 |
| Table 1: Edge vs Fabric Functionality |
Load balancing large flows on multi-path networks
 |
| Figure 1: Active control of large flows in a multi-path topology |
 |
| Figure 2: Two path topology |
To ensure that packets in a flow arrive in order at their destination, switch s3 computes a hash function over selected fields in the packets (e.g. source and destination IP addresses Continue reading
Mininet integrated hybrid OpenFlow testbed
 |
| Figure 1: Hybrid Programmable Forwarding Planes |
A number of vendors support sFlow and integrated hybrid OpenFlow today, examples described on this blog include: Alcatel-Lucent, Brocade, and Hewlett-Packard. However, building a physical testbed is expensive and time consuming. This article describes how to build an sFlow and hybrid OpenFlow testbed using free Mininet network emulation software. The testbed emulates ECMP leaf and spine data center fabrics and provides a platform for experimenting with analytics driven feedback control using the sFlow-RT hybrid OpenFlow controller.
First build an Ubuntu 13.04 / 13.10 virtual machine then follow instructions for installing Mininet - Option 3: Installation from Packages.
Next, install an Apache web server:
sudo apt-get install apache2Install the sFlow-RT integrated hybrid OpenFlow controller, either on the Mininet virtual machine, or on a different system (Java 1.6+ is required to run sFlow-RT):
Continue reading
Configuring Mellanox switches
The following commands configure a Mellanox switch (10.0.0.252) to sample packets at 1-in-10000, poll counters every 30 seconds and send sFlow to an analyzer (10.0.0.50) using the default sFlow port 6343:sflow enableFor each interface:
sflow agent-ip 10.0.0.252
sflow collector-ip 10.0.0.50
sflow sampling-rate 10000
sflow counter-poll-interval 30
interface ethernet 1/1 sflow enableA previous posting discussed the selection of sampling rates. Additional information can be found on the Mellanox web site.
See Trying out sFlow for suggestions on getting started with sFlow monitoring and reporting.
DDoS mitigation hybrid OpenFlow controller
This article describes an additional example, using the sFlow-RT controller to implement the ONS 2014 SDN Idol winning distributed denial of service (DDoS) mitigation solution - Real-time SDN Analytics for DDoS mitigation.
 |
| Figure 1: ISP/IX Market Segment |
 |
| Figure 2: Novel DDoS Mitigation solution using Real-time SDN Analytics |
// Define large flow Continue reading
Cisco, ACI, OpFlex and OpenDaylight
In, Cisco Submits Its (Very Different) SDN to IETF & OpenDaylight, SDNCentral editor Craig Matsumoto comments, "You know how, early on, people were all worried Cisco would 'take over' OpenDaylight? This is pretty much what they were talking about. It’s not a 'takeover,' literally, but OpFlex and the group policy concept steer OpenDaylight into a new direction that it otherwise wouldn’t have, one that Cisco happens to already have taken."
CIMI Corp. President, Tom Nolle, remarks "We’re all in business to make money, and if Cisco takes a position in a key market like SDN that seems to favor…well…doing nothing much different, you have to assume they have good reason to believe that their approach will resonate with buyers." - Cisco’s OpFlex: We Have Sound AND Fury
This article will look at some of the architectural issues raised by Cisco's announcement based on the following documents:
- OpenDaylight: Project Proposals:Group Based Policy Plugin
- IETF: OpFlex Control Protocol
Integrated hybrid OpenFlow control of HP switches
The previous article provided an example of large flow marking using an Alcatel-Lucent OmniSwitch 6900 switch. This article discusses how to replicate the example using HP Networking switches.
At present, the following HP switch models are listed as having OpenFlow support:
- FlexFabric 12900 Switch Series
- 12500 Switch Series
- FlexFabric 11900 Switch Series
- 8200 zl Switch Series
- HP FlexFabric 5930 Switch Series
- 5920 Switch Series
- 5900 Switch Series
- 5400 zl Switch Series
- 3800 Switch Series
- HP 3500 and 3500 yl Switch Series
- 2920 Switch Series
HP's OpenFlow implementation supports integrated hybrid mode - provided the OpenFlow controller pushes a default low priority OpenFlow rule that matches all packets and applies the NORMAL action (i.e. instructs the switch to apply default switching / routing forwarding to the packets).
In Continue reading
Performance optimizing hybrid OpenFlow controller
This article discusses the evolving architecture of software defined networking (SDN) and the role of analytics and traffic engineering. InMon's sFlow-RT controller is used to provide practical examples of the architecture.
 |
| Figure 1: Fabric: A Retrospective on Evolving SDN |
First generation SDN controllers were designed before the edge / core split became Continue reading
ONS2014 SDN Idol finalist demonstrations
The video of the ONS 2014 SDN Idol final demonstrations has been released (the demonstrations were presented live at the Open Networking Summit on Monday, March 3, 02:30P - 04:00P).
The first demo presented is Real-time SDN Analytics for DDoS mitigation, a joint Brocade / InMon solution that combines real-time sFlow analytics and OpenFlow with SDN so that service providers can deliver large scale distributed denial of service (DDoS) attack mitigation services to their enterprise customers using their existing network infrastructure. DDoS mitigation is particularly topical, two weeks ago, a large attack was targeted at CloudFlare, DDoS Attack Hits 400 Gbit/s, Breaks Record, and this past week, Meetup.com has been hit with a large persistent attack, Meetup Suffering Significant DDoS Attack, Taking It Offline For Days. The SDN DDoS mitigation solution can address these large attacks by leveraging the multi-Terabit, line-rate, monitoring and filtering capabilities in the network switches.
ONS2014 Announces Finalists for SDN Idol 2014 provides some sFlow related trivia relating to the finalists.An expert panel of judges selected the finalists:
ONS2014 Announces Finalists for SDN Idol 2014
- Brocade and InMon: Real-time SDN Analytics for DDoS mitigation
- HP: HP SDN App Store and Open SDN Ecosystem
- OpenDaylight: Hydrogen
- Pica8: Pica8’s Open SDN Starter Kit
- Radware: Adaptive Network, Application and APT Protection
Real-time SDN Analytics for DDoS mitigation is an example of a performance aware SDN controller that combines sFlow and OpenFlow for the visibility and control needed to build self optimizing networks that automatically adapt to changing traffic conditions. A number of other use cases were outlined by Brocade at the recent OpenDaylight Summit - see Flow-aware Real-time SDN Analytics (FRSA)
There are interesting links with other finalists:
There are interesting links with other finalists:
- OpenDaylight Hydrogen The Brocade is a Platinum member of the OpenDaylight project, and the Brocade/InMon DDoS mitigation solution employs OpenDaylight Hydrogen as an OpenFlow controller. Like Brocade, many of the OpenDaylight project members also support sFlow in their networking equipment, including: Brocade, Cisco, IBM, Juniper, NEC, A10 Networks, Arista, Dell, HP, Huawei, Intel, and ZTE. One might expect to see other vendors start to build traffic aware solutions on OpenDaylight in the coming months.
- HP SDN App Store and Open SDN Continue reading
Dell, Cumulus, Open Source, Open Standards, and Unified Management
On Thursday, at Network Field Day 7, Arpit Joshipura described Dell's networking strategy. He started by polling the delegates to see which topics were most on their mind.
Arpit suggests that customers will choose Cumulus Linux as the operating system for the layer 3 features and because they can use the same expertise and tools (Puppet, Chef etc.) to manage Linux servers and the switches connecting them. He also suggested that customers would choose FTOS for legacy networks and layer 2 features. Support for the Open Networking Install Environment Continue reading
#NFD7 Real Time SDN and NFV Analytics for DDoS Mitigation
Today, at Networking Field Day 7, Ramki Krishnan of Brocade Networks demonstrated how the sFlow and OpenFlow standards can be combined to deliver DDoS mitigation as a service. Ramki is a co-author of related Internet Drafts: Large Flow Use Cases for I2RS PBR and QoS and Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks.
Flow-aware Real-time SDN Analytics (FRSA)
This article will review the slides and discuss selected topics in detail.
Large flow marking using hybrid OpenFlow
 |
| Figure 1: Marking large flows |
 |
| Figure 2: Large flow marking controller results |
ping -f 10.0.0.238 -s 1400Figure 2 shows the results, the left half of the chart shows traffic when the controller is disabled and the right half shows traffic when the controller is enabled. The blue line trends the largest unmarked flow seen in the network and the gold line shows the largest marked flow. When controller is disabled, none of the traffic is marked. When the controller is enabled, sFlow-RT detects the large flow Continue reading