Peter

Author Archives: Peter

HP proposes hybrid OpenFlow discussion at Open Daylight design forum

Hewlett-Packard, an Open Daylight platinum member, is proposing a discussion of integrated hybrid OpenFlow at the upcoming Open Daylight Developer Design Forum, September 29 - 30, 2014, Santa Clara.

Topics for ODL Design Summit from HP contains the following proposal, making the case for integrated hybrid OpenFlow:
We would like to share our experiences with Customer SDN deployments that require OpenFlow hybrid mode. Why it matters, implementation considerations, and how to achieve better support for it in ODL

OpenFlow-compliant switches come in two types: OpenFlow-only, and OpenFlow-hybrid. OpenFlow-only switches support only OpenFlow operation, in those switches all packets are processed by the OpenFlow pipeline, and cannot be processed otherwise. OpenFlow-hybrid switches support both OpenFlow operation and normal Ethernet switching operation, i.e. traditional L2 Ethernet switching, VLAN isolation, L3 routing (IPv4 routing, IPv6 routing...), ACL and QoS processing

The rationale for supporting hybrid mode is twofold:
  1. Controlled switches have decades of embedded traditional networking logic. The controller does not add value to a solution if it replicates traditional forwarding logic. One alternative controller responsibility is that provides forwarding decisions when it wants to override the traditional data-plane forwarding decision.
  2. Controllers can be gradually incorporated into a traditional network. Continue reading

DDoS mitigation with Cumulus Linux

Figure 1: Real-time SDN Analytics for DDoS mitigation
Figure 1 shows how service providers are ideally positioned to mitigate large flood attacks directed at their customers. The mitigation solution involves an SDN controller that rapidly detects and filters out attack traffic and protects the customer's Internet access.

This article builds on the test setup described in RESTful control of Cumulus Linux ACLs in order to implement the ONS 2014 SDN Idol winning distributed denial of service (DDoS) mitigation solution - Real-time SDN Analytics for DDoS mitigation.

The following sFlow-RT application implements basic DDoS mitigation functionality: 
include('extras/json2.js');

// Define large flow as greater than 100Mbits/sec for 1 second or longer
var bytes_per_second = 100000000/8;
var duration_seconds = 1;

var id = 0;
var controls = {};

setFlow('udp_target',
 {keys:'ipdestination,udpsourceport', value:'bytes',
  filter:'direction=egress', t:duration_seconds}
);

setThreshold('attack',
 {metric:'udp_target', value:bytes_per_second, byFlow:true, timeout:4,
  filter:{ifspeed:[1000000000]}}
);

setEventHandler(function(evt) {
 if(controls[evt.flowKey]) return;

 var rulename = 'ddos' + id++;
 var keys = evt.flowKey.split(',');
 var acl = [
'[iptables]',
'# block UDP reflection attack',
'-A FORWARD --in-interface swp+ -d ' + keys[0]
+ ' -p udp --sport ' + keys[1] + ' -j DROP'
 ];
 http('http://'+evt.agent+':8080/acl/'+rulename,
      'put','application/json',JSON.stringify(acl));
 controls[evt.flowKey] = {
   agent:evt.agent,
   dataSource:evt.dataSource,
   rulename:rulename,
    Continue reading

Docker performance monitoring

IT’S HERE: DOCKER 1.0 recently announced the first production release of the Docker Linux container platform. Docker is seeing explosive growth and has already been embraced by IBM, RedHat and RackSpace. Today the open source Host sFlow project released support for Docker, exporting standard sFlow performance metrics for Linux containers and unifying Linux containers with the broader sFlow ecosystem.
Visibility and the software defined data center
Host sFlow Docker support simplifies data center performance management by unifying monitoring of Linux containers with monitoring of virtual machines (Hyper-V, KVM/libvirt, Xen/XCP/XenServer), virtual switches (Open vSwitch, Hyper-V Virtual Switch, IBM Distributed Virtual Switch, HP FlexFabric Virtual Switch), servers (Linux, Windows, Solaris, AIX, FreeBSD), and physical networks (over 40 vendors, including: A10, Arista, Alcatel-Lucent, Arista, Brocade, Cisco, Cumulus, Extreme, F5, Hewlett-Packard, Hitachi, Huawei, IBM, Juniper, Mellanox, NEC, ZTE). In addition, standardizing metrics allows allows measurements to be shared among different tools, further reducing operational complexity.


The talk provides additional background on the sFlow standard and case studies. The remainder of this article describes how to use Host sFlow to monitor a Docker server pool.

First, download, compile and install the Host sFlow agent on a Docker host (Note: The agent needs to Continue reading

Microsoft Office 365 outage

6/24/2014 Information Week - Microsoft Exchange Online Suffers Service Outage, "Service disruptions with Microsoft's Exchange Online left many companies with no email on Tuesday."

The following entry on the Microsoft 365 community forum describes the incident:
====================================

Closure Summary: On Tuesday, June 24, 2014, at approximately 1:11 PM UTC, engineers received reports of an issue in which some customers were unable to access the Exchange Online service. Investigation determined that a portion of the networking infrastructure entered into a degraded state. Engineers made configuration changes on the affected capacity to remediate end-user impact. The issue was successfully fixed on Tuesday, June 24, 2014, at 9:50 PM UTC.

Customer Impact: Affected customers were unable to access the Exchange Online service.

Incident Start Time: Tuesday, June 24, 2014, at 1:11 PM UTC

Incident End Time: Tuesday, June 24, 2014, at 9:50 PM UTC

=====================================
The closure summary shows that operators took 8 hour 39 minutes to manually diagnose and remediate the problem with degraded networking infrastructure. The network related outage described in this example is not an isolated incident; other incidents described on this blog include: Packet loss, Amazon EC2 outage, Gmail outage, Delay vs utilization for Continue reading

RESTful control of Cumulus Linux ACLs

Figure 1: Elephants and Mice
Elephant Detection in Virtual Switches & Mitigation in Hardware discusses a VMware and Cumulus demonstration, Elephants and Mice, in which the virtual switch on a host detects and marks large "Elephant" flows and the hardware switch enforces priority queueing to prevent Elephant flows from adversely affecting latency of small "Mice" flows.

This article demonstrates a self contained real-time Elephant flow marking solution that leverages the visibility and control features of Cumulus Linux.

SDN fabric controller for commodity data center switches provides some background on the capabilities of the commodity switch hardware used to run Cumulus Linux. The article describes how the measurement and control capabilities of the hardware can be used to maximize data center fabric performance:
Exposing the ACL configuration files through a RESTful API offers a straightforward method of remotely creating, reading, updating, deleting and listing ACLs.

For example, the following command creates a filter called Continue reading

Cumulus Networks, sFlow and data center automation

Cumulus Networks and InMon Corp have ported the open source Host sFlow agent to the upcoming Cumulus Linux 2.1 release. The Host sFlow agent already supports Linux, Windows, FreeBSD, Solaris, and AIX operating systems and KVM, Xen, XCP, XenServer, and Hyper-V hypervisors, delivering a standard set of performance metrics from switches, servers, hypervisors, virtual switches, and virtual machines - see Visibility and the software defined data center

The Cumulus Linux platform makes it possible to run the same open source agent on switches, servers, and hypervisors - providing unified end-to-end visibility across the data center. The open networking model that Cumulus is pioneering offers exciting opportunities. Cumulus Linux allows popular open source server orchestration tools to also manage the network, and the combination of real-time, data center wide analytics with orchestration make it possible to create self-optimizing data centers.

Install and configure Host sFlow agent

The following command installs the Host sFlow agent on a Cumulus Linux switch: 
sudo apt-get install hsflowd
Note: Network managers may find this command odd since it is usually not possible to install third party software on switch hardware. However, what is even more radical is that Cumulus Linux allows users to download source Continue reading

SDN fabric controller for commodity data center switches

Figure 1: Rise of merchant silicon
Figure 1 illustrates the rapid transition to merchant silicon among leading data center network vendors, including: Alcatel-Lucent, Arista, Cisco, Cumulus, Dell, Extreme, Juniper, Hewlett-Packard, and IBM.

This article will examine some of the factors leading to commoditization of network hardware and the role that software defined networking (SDN) plays in coordinating hardware resources to deliver increased network efficiency.
Figure 2: Fabric: A Retrospective on Evolving SDN
The article, Fabric: A Retrospective on Evolving SDN by Martin Casado, Teemu Koponen, Scott Shenker, and Amin Tootoonchian, makes the case for a two tier SDN architecture; comprising a smart edge and an efficient core.
Table 1: Edge vs Fabric Functionality
Virtualization and advances in the networking capability of x86 based servers are drivers behind this separation. Virtual machines are connected to each other and to the physical network using a software virtual switch. The software switch provides the flexibility to quickly develop and deploy advanced features like network virtualization, tenant isolation, distributed firewalls, etc. Network function virtualization (NFV) is moving firewall, load balancing, routing, etc. functions from dedicated appliances to virtual machines or embedding them within the virtual switches. The increased importance of network centric software has Continue reading

Load balancing large flows on multi-path networks

Figure 1: Active control of large flows in a multi-path topology
Figure 1 shows initial results from the Mininet integrated hybrid OpenFlow testbed demonstrating that active steering of large flows using a performance aware SDN controller significantly improves network throughput of multi-path network topologies.
Figure 2: Two path topology
The graph in Figure 1 summarizes results from topologies with 2, 3 and 4 equal cost paths. For example, the Mininet topology in Figure 2 has two equal cost paths of 10Mbit/s (shown in blue and red). The iperf traffic generator was used to create a continuous stream of 20 second flows from h1 to h3 and from h2 to h4. If traffic were perfectly balanced, each flow would achieve 10Mbit/s throughput. However, Figure 1 shows that the throughput obtained using hash based ECMP load balancing is approximately 6.8Mbit/s. Interestingly, the average link throughput decreases as additional paths are added, dropping to approximately 6.2Mbit/s with four equal cost paths (see the blue bars in Figure 1).

To ensure that packets in a flow arrive in order at their destination, switch s3 computes a hash function over selected fields in the packets (e.g. source and destination IP addresses Continue reading

Mininet integrated hybrid OpenFlow testbed

Figure 1: Hybrid Programmable Forwarding Planes
Integrated hybrid OpenFlow combines OpenFlow and existing distributed routing protocols to deliver robust software defined networking (SDN) solutions. Performance optimizing hybrid OpenFlow controller describes how the sFlow and OpenFlow standards combine to deliver visibility and control to address challenges including: DDoS mitigation, ECMP load balancing, LAG load balancing, and large flow marking.

A number of vendors support sFlow and integrated hybrid OpenFlow today, examples described on this blog include: Alcatel-Lucent, Brocade, and Hewlett-Packard. However, building a physical testbed is expensive and time consuming. This article describes how to build an sFlow and hybrid OpenFlow testbed using free Mininet network emulation software. The testbed emulates ECMP leaf and spine data center fabrics and provides a platform for experimenting with analytics driven feedback control using the sFlow-RT hybrid OpenFlow controller.

First build an Ubuntu 13.04 / 13.10 virtual machine then follow instructions for installing Mininet - Option 3: Installation from Packages.

Next, install an Apache web server:
sudo apt-get install apache2
Install the sFlow-RT integrated hybrid OpenFlow controller, either on the Mininet virtual machine, or on a different system (Java 1.6+ is required to run sFlow-RT):
 Continue reading

Configuring Mellanox switches

The following commands configure a Mellanox switch (10.0.0.252) to sample packets at 1-in-10000, poll counters every 30 seconds and send sFlow to an analyzer (10.0.0.50) using the default sFlow port 6343: 
sflow enable
sflow agent-ip 10.0.0.252
sflow collector-ip 10.0.0.50
sflow sampling-rate 10000
sflow counter-poll-interval 30
For each interface: 
interface ethernet 1/1 sflow enable
A previous posting discussed the selection of sampling rates. Additional information can be found on the Mellanox web site.

See Trying out sFlow for suggestions on getting started with sFlow monitoring and reporting.

DDoS mitigation hybrid OpenFlow controller

Performance optimizing hybrid OpenFlow controller describes the growing split in the SDN controller market between edge controllers using virtual switches to deliver network virtualization (e.g. VMware NSX, Nuage Networks, Juniper Contrail, etc.) and fabric controllers that optimize performance of the physical network. The article provides an example using InMon's sFlow-RT controller to detect and mark large "elephant" flows so that they don't interfere with latency sensitive small "mice" flows.

This article describes an additional example, using the sFlow-RT controller to implement the ONS 2014 SDN Idol winning distributed denial of service (DDoS) mitigation solution - Real-time SDN Analytics for DDoS mitigation.
Figure 1: ISP/IX Market Segment
Figure 1 shows how service providers are ideally positioned to mitigate large flood attacks directed at their customers. The mitigation solution involves an SDN controller that rapidly detects and filters out attack traffic and protects the customer's Internet access.
Figure 2: Novel DDoS Mitigation solution using Real-time SDN Analytics
Figure 2 shows the elements of the control system in the SDN Idol demonstration. The addition of an embedded OpenFlow controller in sFlow-RT allows the entire DDoS mitigation system to be collapsed into the following sFlow-RT JavaScript application:
// Define large flow  Continue reading

Cisco, ACI, OpFlex and OpenDaylight

Cisco's April 2nd, 2014 announcement - Cisco and Industry Leaders Will Deliver Open, Multi-Vendor, Standards-Based Networks for Application Centric Infrastructure with OpFlex Protocol - has drawn mixed reviews from industry commentators.

In, Cisco Submits Its (Very Different) SDN to IETF & OpenDaylight, SDNCentral editor Craig Matsumoto comments, "You know how, early on, people were all worried Cisco would 'take over' OpenDaylight? This is pretty much what they were talking about. It’s not a 'takeover,' literally, but OpFlex and the group policy concept steer OpenDaylight into a new direction that it otherwise wouldn’t have, one that Cisco happens to already have taken."

CIMI Corp. President, Tom Nolle, remarks "We’re all in business to make money, and if Cisco takes a position in a key market like SDN that seems to favor…well…doing nothing much different, you have to assume they have good reason to believe that their approach will resonate with buyers." - Cisco’s OpFlex: We Have Sound AND Fury

This article will look at some of the architectural issues raised by Cisco's announcement based on the following documents:
The diagram at the top of this article illustrates the Continue reading

Integrated hybrid OpenFlow control of HP switches

Performance optimizing hybrid OpenFlow controller describes InMon's sFlow-RT controller. The controller makes use of the sFlow and OpenFlow standards and is optimized for real-time traffic engineering applications that managing large traffic flows, including: DDoS mitigation, ECMP load balancing, LAG load balancing, large flow marking etc.

The previous article provided an example of large flow marking using an Alcatel-Lucent OmniSwitch 6900 switch. This article discusses how to replicate the example using HP Networking switches.

At present, the following HP switch models are listed as having OpenFlow support:
  • FlexFabric 12900 Switch Series
  • 12500 Switch Series
  • FlexFabric 11900 Switch Series
  • 8200 zl Switch Series
  • HP FlexFabric 5930 Switch Series
  • 5920 Switch Series
  • 5900 Switch Series
  • 5400 zl Switch Series
  • 3800 Switch Series
  • HP 3500 and 3500 yl Switch Series
  • 2920 Switch Series 
Note: All of the above HP switches (and many others) support the sFlow standard - see sFlow Products: Network Equipment @ sFlow.org.

HP's OpenFlow implementation supports integrated hybrid mode - provided the OpenFlow controller pushes a default low priority OpenFlow rule that matches all packets and applies the NORMAL action (i.e. instructs the switch to apply default switching / routing forwarding to the packets).

In Continue reading

Performance optimizing hybrid OpenFlow controller

The latest release of InMon's sFlow-RT controller adds integrated hybrid OpenFlow support - optimized for real-time traffic engineering applications that manage large traffic flows, including: DDoS mitigation, ECMP load balancing, LAG load balancing, large flow marking etc.

This article discusses the evolving architecture of software defined networking (SDN) and the role of analytics and traffic engineering. InMon's sFlow-RT controller is used to provide practical examples of the architecture.
Figure 1: Fabric: A Retrospective on Evolving SDN
The article, Fabric: A Retrospective on Evolving SDN by Martin Casado, Teemu Koponen, Scott Shenker, and Amin Tootoonchian, makes the case for a two tier software defined networking (SDN) architecture; comprising a smart edge and an efficient core. The article, Pragmatic software defined networking on this blog, examines how the edge is moving into virtual switches, with tunneling (VxLAN, NVGRE, GRE, STT) used to virtualize the network and decouple the edge from the core. As complex policy decisions move to the network edge, the core fabric is left with the task of efficiently managing physical resources in order to deliver low latency, high bandwidth connectivity between edge switches.

First generation SDN controllers were designed before the edge / core split became Continue reading

ONS2014 SDN Idol finalist demonstrations


The video of the ONS 2014 SDN Idol final demonstrations has been released (the demonstrations were presented live at the Open Networking Summit on Monday, March 3, 02:30P - 04:00P).

The first demo presented is Real-time SDN Analytics for DDoS mitigation, a joint Brocade / InMon solution that combines real-time sFlow analytics and OpenFlow with SDN so that service providers can deliver large scale distributed denial of service (DDoS) attack mitigation services to their enterprise customers using their existing network infrastructure. DDoS mitigation is particularly topical, two weeks ago, a large attack was targeted at CloudFlare, DDoS Attack Hits 400 Gbit/s, Breaks Record, and this past week, Meetup.com has been hit with a large persistent attack, Meetup Suffering Significant DDoS Attack, Taking It Offline For Days. The SDN DDoS mitigation solution can address these large attacks by leveraging the multi-Terabit, line-rate, monitoring and filtering capabilities in the network switches.
ONS2014 Announces Finalists for SDN Idol 2014 provides some sFlow related trivia relating to the finalists. 
An expert panel of judges selected the finalists:

The finalists were selected based on the following criteria:
Voting is open to ONS delegates and will occur during this evenings reception and Continue reading

ONS2014 Announces Finalists for SDN Idol 2014

Today the Open Networking Summit announced the five finalists for the SDN Idol 2014 competition:
Real-time SDN Analytics for DDoS mitigation is an example of a performance aware SDN controller that combines sFlow and OpenFlow for the visibility and control needed to build self optimizing networks that automatically adapt to changing traffic conditions. A number of other use cases were outlined by Brocade at the recent OpenDaylight Summit - see Flow-aware Real-time SDN Analytics (FRSA)

There are interesting links with other finalists:
  • OpenDaylight Hydrogen The Brocade is a Platinum member of the OpenDaylight project, and the Brocade/InMon DDoS mitigation solution employs OpenDaylight Hydrogen as an OpenFlow controller. Like Brocade, many of the OpenDaylight project members also support sFlow in their networking equipment, including: Brocade, Cisco, IBM, Juniper, NEC, A10 Networks, Arista, Dell, HP, Huawei, Intel, and ZTE. One might expect to see other vendors start to build traffic aware solutions on OpenDaylight in the coming months.
  • HP SDN App Store and Open SDN Continue reading

Dell, Cumulus, Open Source, Open Standards, and Unified Management


On Thursday, at Network Field Day 7, Arpit Joshipura described Dell's networking strategy. He started by polling the delegates to see which topics were most on their mind.
The first topic raised by many of the delegates was the recently announced Dell/Cumulus partnership (listed as Open NW on the white board), see Dell Unlocks New Era for Open Networking, Decouples Hardware and Software. Next on the list was an interest in Dell's Open Source networking strategy, understanding Dell's Differentiation strategy, and plans for L3.
Dell's open networking strategy is described at time marker 14:55 in the video. Dell was one of the first vendors to move to merchant silicon, now they are opening up the switch platform, allowing customers to choose from standard merchant silicon based switch platforms (Broadcom, Intel) and switch software (currently FTOS / Cumulus).

Arpit suggests that customers will choose Cumulus Linux as the operating system for the layer 3 features and because they can use the same expertise and tools (Puppet, Chef etc.) to manage Linux servers and the switches connecting them. He also suggested that customers would choose FTOS for legacy networks and layer 2 features. Support for the Open Networking Install Environment Continue reading

#NFD7 Real Time SDN and NFV Analytics for DDoS Mitigation


Today, at Networking Field Day 7, Ramki Krishnan of Brocade Networks demonstrated how the sFlow and OpenFlow standards can be combined to deliver DDoS mitigation as a service. Ramki is a co-author of related Internet Drafts: Large Flow Use Cases for I2RS PBR and QoS and Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks.
The talk starts by outlining the growing problem of DDoS attacks and the market opportunity for mitigation solutions, referencing the articles, Prolexic Publishes Top 10 DDoS Attack Trends for 2013, World's largest DDoS strikes US, Europe.
The diagram shows the unique position occupied by Internet Service Provider (ISP) and Internet Exchange (IX) networks, allowing them to filter large flood attacks and prevent them from overwhelming Enterprise customer connections - provided they can use their network to efficiently detect attacks and automatically filter traffic for their customers.
This diagram shows how standard sFlow enabled in the switches and routers provides a continuous stream of measurement data to InMon sFlow-RT, which provided real-time detection and notification of DDoS attacks to the DDoS Mitigation SDN Application. The DDoS Mitigation SDN Application selects a mitigation action and instructs the SDN Controller to push the action to Continue reading

Flow-aware Real-time SDN Analytics (FRSA)

Today at the OpenDaylight Summit in Santa Clara, Ram (Ramki) Krishnan of Brocade Communications presented a framework and set of use cases for applying software defined networking (SDN) techniques control large (elephant) flows. Ramki is a co-author of related Internet Drafts: Large Flow Use Cases for I2RS PBR and QoS and Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks. The slides from the talk are available on the OpenDaylight Summit web site.

This article will review the slides and discuss selected topics in detail.
The FRSA framework identifies four classes of traffic flow based on flow rate and flow duration and identifies long lived large flows as amenable to SDN based control since they can be readily observed, consume significant resources, and last long enough to be effectively controlled. The article, SDN and large flows, discusses the opportunity presented by large flow control in greater detail.
The two elements required in the FRSA framework are real-time traffic analytics - to rapidly identify the large flows (within seconds) and a control mechanism such as integrated hybrid OpenFlow, that allows the normal switch forwarding protocols to handle traffic, but offers a way for the controller to intervene and determine Continue reading

Large flow marking using hybrid OpenFlow

Top of rack switches are in a unique position at the edge of the network to implement traffic engineering controls. Marking large flows describes a use case for dynamically detecting and marking large flows as they enter the network:
Figure 1: Marking large flows
Physical switch hybrid OpenFlow example described how real-time sFlow analytics can be used to trigger OpenFlow controls to block denial of service attacks. This article will describe how the sFlow-RT, Floodlight OpenFlow controller, and Alcatel-Lucent OmniSwitch hybrid OpenFlow SDN controller setup can be programmed to dynamically detect and mark large (Elephant) flows as they enter the network.
Figure 2: Large flow marking controller results
In the experimental setup, a flood ping is used to generate a large flow: 
ping -f 10.0.0.238 -s 1400
Figure 2 shows the results, the left half of the chart shows traffic when the controller is disabled and the right half shows traffic when the controller is enabled. The blue line trends the largest unmarked flow seen in the network and the gold line shows the largest marked flow. When controller is disabled, none of the traffic is marked. When the controller is enabled, sFlow-RT detects the large flow Continue reading
1 … 12 13 14 15