0

Microsoft Office 365 outage

6/24/2014 Information Week - Microsoft Exchange Online Suffers Service Outage, "Service disruptions with Microsoft's Exchange Online left many companies with no email on Tuesday."

The following entry on the Microsoft 365 community forum describes the incident:

====================================

Closure Summary: On Tuesday, June 24, 2014, at approximately 1:11 PM UTC, engineers received reports of an issue in which some customers were unable to access the Exchange Online service. Investigation determined that a portion of the networking infrastructure entered into a degraded state. Engineers made configuration changes on the affected capacity to remediate end-user impact. The issue was successfully fixed on Tuesday, June 24, 2014, at 9:50 PM UTC.

Customer Impact: Affected customers were unable to access the Exchange Online service.

Incident Start Time: Tuesday, June 24, 2014, at 1:11 PM UTC

Incident End Time: Tuesday, June 24, 2014, at 9:50 PM UTC

=====================================

The closure summary shows that operators took 8 hour 39 minutes to manually diagnose and remediate the problem with degraded networking infrastructure. The network related outage described in this example is not an isolated incident; other incidents described on this blog include: Packet loss, Amazon EC2 outage, Gmail outage, Delay vs utilization for Continue reading

0

RESTful control of Cumulus Linux ACLs

Figure 1: Elephants and Mice

Elephant Detection in Virtual Switches & Mitigation in Hardware discusses a VMware and Cumulus demonstration, Elephants and Mice, in which the virtual switch on a host detects and marks large "Elephant" flows and the hardware switch enforces priority queueing to prevent Elephant flows from adversely affecting latency of small "Mice" flows.

This article demonstrates a self contained real-time Elephant flow marking solution that leverages the visibility and control features of Cumulus Linux.

SDN fabric controller for commodity data center switches provides some background on the capabilities of the commodity switch hardware used to run Cumulus Linux. The article describes how the measurement and control capabilities of the hardware can be used to maximize data center fabric performance:

• Visibility: Cumulus recently added sFlow support, providing the network wide visibility needed for control - see Cumulus Networks, sFlow and data center automation.
• Control: Cumulus Linux Netfilter (ACLs) describes how standard Linux ebtables / iptables configuration files are used to manage the ACL/policy capabilities of the hardware.

Exposing the ACL configuration files through a RESTful API offers a straightforward method of remotely creating, reading, updating, deleting and listing ACLs.

For example, the following command creates a filter called Continue reading

0

Cumulus Networks, sFlow and data center automation

Cumulus Networks and InMon Corp have ported the open source Host sFlow agent to the upcoming Cumulus Linux 2.1 release. The Host sFlow agent already supports Linux, Windows, FreeBSD, Solaris, and AIX operating systems and KVM, Xen, XCP, XenServer, and Hyper-V hypervisors, delivering a standard set of performance metrics from switches, servers, hypervisors, virtual switches, and virtual machines - see Visibility and the software defined data center

The Cumulus Linux platform makes it possible to run the same open source agent on switches, servers, and hypervisors - providing unified end-to-end visibility across the data center. The open networking model that Cumulus is pioneering offers exciting opportunities. Cumulus Linux allows popular open source server orchestration tools to also manage the network, and the combination of real-time, data center wide analytics with orchestration make it possible to create self-optimizing data centers.

Install and configure Host sFlow agent

The following command installs the Host sFlow agent on a Cumulus Linux switch:

sudo apt-get install hsflowd

Note: Network managers may find this command odd since it is usually not possible to install third party software on switch hardware. However, what is even more radical is that Cumulus Linux allows users to download source Continue reading

0

SDN fabric controller for commodity data center switches

Figure 1: Rise of merchant silicon

Figure 1 illustrates the rapid transition to merchant silicon among leading data center network vendors, including: Alcatel-Lucent, Arista, Cisco, Cumulus, Dell, Extreme, Juniper, Hewlett-Packard, and IBM.

This article will examine some of the factors leading to commoditization of network hardware and the role that software defined networking (SDN) plays in coordinating hardware resources to deliver increased network efficiency.

Figure 2: Fabric: A Retrospective on Evolving SDN

The article, Fabric: A Retrospective on Evolving SDN by Martin Casado, Teemu Koponen, Scott Shenker, and Amin Tootoonchian, makes the case for a two tier SDN architecture; comprising a smart edge and an efficient core.

Table 1: Edge vs Fabric Functionality

Virtualization and advances in the networking capability of x86 based servers are drivers behind this separation. Virtual machines are connected to each other and to the physical network using a software virtual switch. The software switch provides the flexibility to quickly develop and deploy advanced features like network virtualization, tenant isolation, distributed firewalls, etc. Network function virtualization (NFV) is moving firewall, load balancing, routing, etc. functions from dedicated appliances to virtual machines or embedding them within the virtual switches. The increased importance of network centric software has Continue reading

0

Load balancing large flows on multi-path networks

Figure 1: Active control of large flows in a multi-path topology

Figure 1 shows initial results from the Mininet integrated hybrid OpenFlow testbed demonstrating that active steering of large flows using a performance aware SDN controller significantly improves network throughput of multi-path network topologies.

Figure 2: Two path topology

The graph in Figure 1 summarizes results from topologies with 2, 3 and 4 equal cost paths. For example, the Mininet topology in Figure 2 has two equal cost paths of 10Mbit/s (shown in blue and red). The iperf traffic generator was used to create a continuous stream of 20 second flows from h1 to h3 and from h2 to h4. If traffic were perfectly balanced, each flow would achieve 10Mbit/s throughput. However, Figure 1 shows that the throughput obtained using hash based ECMP load balancing is approximately 6.8Mbit/s. Interestingly, the average link throughput decreases as additional paths are added, dropping to approximately 6.2Mbit/s with four equal cost paths (see the blue bars in Figure 1).

To ensure that packets in a flow arrive in order at their destination, switch s3 computes a hash function over selected fields in the packets (e.g. source and destination IP addresses Continue reading

0

Mininet integrated hybrid OpenFlow testbed

Figure 1: Hybrid Programmable Forwarding Planes

Integrated hybrid OpenFlow combines OpenFlow and existing distributed routing protocols to deliver robust software defined networking (SDN) solutions. Performance optimizing hybrid OpenFlow controller describes how the sFlow and OpenFlow standards combine to deliver visibility and control to address challenges including: DDoS mitigation, ECMP load balancing, LAG load balancing, and large flow marking.

A number of vendors support sFlow and integrated hybrid OpenFlow today, examples described on this blog include: Alcatel-Lucent, Brocade, and Hewlett-Packard. However, building a physical testbed is expensive and time consuming. This article describes how to build an sFlow and hybrid OpenFlow testbed using free Mininet network emulation software. The testbed emulates ECMP leaf and spine data center fabrics and provides a platform for experimenting with analytics driven feedback control using the sFlow-RT hybrid OpenFlow controller.

First build an Ubuntu 13.04 / 13.10 virtual machine then follow instructions for installing Mininet - Option 3: Installation from Packages.

Next, install an Apache web server:

sudo apt-get install apache2

Install the sFlow-RT integrated hybrid OpenFlow controller, either on the Mininet virtual machine, or on a different system (Java 1.6+ is required to run sFlow-RT):

 Continue reading

0

Configuring Mellanox switches

The following commands configure a Mellanox switch (10.0.0.252) to sample packets at 1-in-10000, poll counters every 30 seconds and send sFlow to an analyzer (10.0.0.50) using the default sFlow port 6343:

sflow enable
sflow agent-ip 10.0.0.252
sflow collector-ip 10.0.0.50
sflow sampling-rate 10000
sflow counter-poll-interval 30

For each interface:

interface ethernet 1/1 sflow enable

A previous posting discussed the selection of sampling rates. Additional information can be found on the Mellanox web site.

See Trying out sFlow for suggestions on getting started with sFlow monitoring and reporting.

0

DDoS mitigation hybrid OpenFlow controller

Performance optimizing hybrid OpenFlow controller describes the growing split in the SDN controller market between edge controllers using virtual switches to deliver network virtualization (e.g. VMware NSX, Nuage Networks, Juniper Contrail, etc.) and fabric controllers that optimize performance of the physical network. The article provides an example using InMon's sFlow-RT controller to detect and mark large "elephant" flows so that they don't interfere with latency sensitive small "mice" flows.

This article describes an additional example, using the sFlow-RT controller to implement the ONS 2014 SDN Idol winning distributed denial of service (DDoS) mitigation solution - Real-time SDN Analytics for DDoS mitigation.

Figure 1: ISP/IX Market Segment

Figure 1 shows how service providers are ideally positioned to mitigate large flood attacks directed at their customers. The mitigation solution involves an SDN controller that rapidly detects and filters out attack traffic and protects the customer's Internet access.

Figure 2: Novel DDoS Mitigation solution using Real-time SDN Analytics

Figure 2 shows the elements of the control system in the SDN Idol demonstration. The addition of an embedded OpenFlow controller in sFlow-RT allows the entire DDoS mitigation system to be collapsed into the following sFlow-RT JavaScript application:

// Define large flow  Continue reading

0

Cisco, ACI, OpFlex and OpenDaylight

Cisco's April 2nd, 2014 announcement - Cisco and Industry Leaders Will Deliver Open, Multi-Vendor, Standards-Based Networks for Application Centric Infrastructure with OpFlex Protocol - has drawn mixed reviews from industry commentators.

In, Cisco Submits Its (Very Different) SDN to IETF & OpenDaylight, SDNCentral editor Craig Matsumoto comments, "You know how, early on, people were all worried Cisco would 'take over' OpenDaylight? This is pretty much what they were talking about. It’s not a 'takeover,' literally, but OpFlex and the group policy concept steer OpenDaylight into a new direction that it otherwise wouldn’t have, one that Cisco happens to already have taken."

CIMI Corp. President, Tom Nolle, remarks "We’re all in business to make money, and if Cisco takes a position in a key market like SDN that seems to favor…well…doing nothing much different, you have to assume they have good reason to believe that their approach will resonate with buyers." - Cisco’s OpFlex: We Have Sound AND Fury

This article will look at some of the architectural issues raised by Cisco's announcement based on the following documents:

• OpenDaylight: Project Proposals:Group Based Policy Plugin
• IETF: OpFlex Control Protocol

The diagram at the top of this article illustrates the Continue reading

0

Integrated hybrid OpenFlow control of HP switches

Performance optimizing hybrid OpenFlow controller describes InMon's sFlow-RT controller. The controller makes use of the sFlow and OpenFlow standards and is optimized for real-time traffic engineering applications that managing large traffic flows, including: DDoS mitigation, ECMP load balancing, LAG load balancing, large flow marking etc.

The previous article provided an example of large flow marking using an Alcatel-Lucent OmniSwitch 6900 switch. This article discusses how to replicate the example using HP Networking switches.

At present, the following HP switch models are listed as having OpenFlow support:

• FlexFabric 12900 Switch Series
• 12500 Switch Series
• FlexFabric 11900 Switch Series
• 8200 zl Switch Series
• HP FlexFabric 5930 Switch Series
• 5920 Switch Series
• 5900 Switch Series
• 5400 zl Switch Series
• 3800 Switch Series
• HP 3500 and 3500 yl Switch Series
• 2920 Switch Series 

Note: All of the above HP switches (and many others) support the sFlow standard - see sFlow Products: Network Equipment @ sFlow.org.

HP's OpenFlow implementation supports integrated hybrid mode - provided the OpenFlow controller pushes a default low priority OpenFlow rule that matches all packets and applies the NORMAL action (i.e. instructs the switch to apply default switching / routing forwarding to the packets).

In Continue reading

0

Performance optimizing hybrid OpenFlow controller

The latest release of InMon's sFlow-RT controller adds integrated hybrid OpenFlow support - optimized for real-time traffic engineering applications that manage large traffic flows, including: DDoS mitigation, ECMP load balancing, LAG load balancing, large flow marking etc.

This article discusses the evolving architecture of software defined networking (SDN) and the role of analytics and traffic engineering. InMon's sFlow-RT controller is used to provide practical examples of the architecture.

Figure 1: Fabric: A Retrospective on Evolving SDN

The article, Fabric: A Retrospective on Evolving SDN by Martin Casado, Teemu Koponen, Scott Shenker, and Amin Tootoonchian, makes the case for a two tier software defined networking (SDN) architecture; comprising a smart edge and an efficient core. The article, Pragmatic software defined networking on this blog, examines how the edge is moving into virtual switches, with tunneling (VxLAN, NVGRE, GRE, STT) used to virtualize the network and decouple the edge from the core. As complex policy decisions move to the network edge, the core fabric is left with the task of efficiently managing physical resources in order to deliver low latency, high bandwidth connectivity between edge switches.

First generation SDN controllers were designed before the edge / core split became Continue reading

0

ONS2014 SDN Idol finalist demonstrations

The video of the ONS 2014 SDN Idol final demonstrations has been released (the demonstrations were presented live at the Open Networking Summit on Monday, March 3, 02:30P - 04:00P).

The first demo presented is Real-time SDN Analytics for DDoS mitigation, a joint Brocade / InMon solution that combines real-time sFlow analytics and OpenFlow with SDN so that service providers can deliver large scale distributed denial of service (DDoS) attack mitigation services to their enterprise customers using their existing network infrastructure. DDoS mitigation is particularly topical, two weeks ago, a large attack was targeted at CloudFlare, DDoS Attack Hits 400 Gbit/s, Breaks Record, and this past week, Meetup.com has been hit with a large persistent attack, Meetup Suffering Significant DDoS Attack, Taking It Offline For Days. The SDN DDoS mitigation solution can address these large attacks by leveraging the multi-Terabit, line-rate, monitoring and filtering capabilities in the network switches.

ONS2014 Announces Finalists for SDN Idol 2014 provides some sFlow related trivia relating to the finalists. 

An expert panel of judges selected the finalists:

The finalists were selected based on the following criteria:
Voting is open to ONS delegates and will occur during this evenings reception and Continue reading

0

ONS2014 Announces Finalists for SDN Idol 2014

Today the Open Networking Summit announced the five finalists for the SDN Idol 2014 competition:

0

Dell, Cumulus, Open Source, Open Standards, and Unified Management

On Thursday, at Network Field Day 7, Arpit Joshipura described Dell's networking strategy. He started by polling the delegates to see which topics were most on their mind.
The first topic raised by many of the delegates was the recently announced Dell/Cumulus partnership (listed as Open NW on the white board), see Dell Unlocks New Era for Open Networking, Decouples Hardware and Software. Next on the list was an interest in Dell's Open Source networking strategy, understanding Dell's Differentiation strategy, and plans for L3.
Dell's open networking strategy is described at time marker 14:55 in the video. Dell was one of the first vendors to move to merchant silicon, now they are opening up the switch platform, allowing customers to choose from standard merchant silicon based switch platforms (Broadcom, Intel) and switch software (currently FTOS / Cumulus).

Arpit suggests that customers will choose Cumulus Linux as the operating system for the layer 3 features and because they can use the same expertise and tools (Puppet, Chef etc.) to manage Linux servers and the switches connecting them. He also suggested that customers would choose FTOS for legacy networks and layer 2 features. Support for the Open Networking Install Environment Continue reading

0

#NFD7 Real Time SDN and NFV Analytics for DDoS Mitigation

Today, at Networking Field Day 7, Ramki Krishnan of Brocade Networks demonstrated how the sFlow and OpenFlow standards can be combined to deliver DDoS mitigation as a service. Ramki is a co-author of related Internet Drafts: Large Flow Use Cases for I2RS PBR and QoS and Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks.
The talk starts by outlining the growing problem of DDoS attacks and the market opportunity for mitigation solutions, referencing the articles, Prolexic Publishes Top 10 DDoS Attack Trends for 2013, World's largest DDoS strikes US, Europe.
The diagram shows the unique position occupied by Internet Service Provider (ISP) and Internet Exchange (IX) networks, allowing them to filter large flood attacks and prevent them from overwhelming Enterprise customer connections - provided they can use their network to efficiently detect attacks and automatically filter traffic for their customers.
This diagram shows how standard sFlow enabled in the switches and routers provides a continuous stream of measurement data to InMon sFlow-RT, which provided real-time detection and notification of DDoS attacks to the DDoS Mitigation SDN Application. The DDoS Mitigation SDN Application selects a mitigation action and instructs the SDN Controller to push the action to Continue reading

0

Flow-aware Real-time SDN Analytics (FRSA)

Today at the OpenDaylight Summit in Santa Clara, Ram (Ramki) Krishnan of Brocade Communications presented a framework and set of use cases for applying software defined networking (SDN) techniques control large (elephant) flows. Ramki is a co-author of related Internet Drafts: Large Flow Use Cases for I2RS PBR and QoS and Mechanisms for Optimal LAG/ECMP Component Link Utilization in Networks. The slides from the talk are available on the OpenDaylight Summit web site.

This article will review the slides and discuss selected topics in detail.
The FRSA framework identifies four classes of traffic flow based on flow rate and flow duration and identifies long lived large flows as amenable to SDN based control since they can be readily observed, consume significant resources, and last long enough to be effectively controlled. The article, SDN and large flows, discusses the opportunity presented by large flow control in greater detail.
The two elements required in the FRSA framework are real-time traffic analytics - to rapidly identify the large flows (within seconds) and a control mechanism such as integrated hybrid OpenFlow, that allows the normal switch forwarding protocols to handle traffic, but offers a way for the controller to intervene and determine Continue reading

0

Large flow marking using hybrid OpenFlow

Top of rack switches are in a unique position at the edge of the network to implement traffic engineering controls. Marking large flows describes a use case for dynamically detecting and marking large flows as they enter the network:

Figure 1: Marking large flows

Physical switch hybrid OpenFlow example described how real-time sFlow analytics can be used to trigger OpenFlow controls to block denial of service attacks. This article will describe how the sFlow-RT, Floodlight OpenFlow controller, and Alcatel-Lucent OmniSwitch hybrid OpenFlow SDN controller setup can be programmed to dynamically detect and mark large (Elephant) flows as they enter the network.

Figure 2: Large flow marking controller results

In the experimental setup, a flood ping is used to generate a large flow:

ping -f 10.0.0.238 -s 1400

Figure 2 shows the results, the left half of the chart shows traffic when the controller is disabled and the right half shows traffic when the controller is enabled. The blue line trends the largest unmarked flow seen in the network and the gold line shows the largest marked flow. When controller is disabled, none of the traffic is marked. When the controller is enabled, sFlow-RT detects the large flow Continue reading

0

sFlow leads convergence of multi-vendor application, server, and network performance management

Over the last six months, leading Application Delivery Controller (ADC) vendors F5 and A10 have added support for the sFlow standard to their respective TMOS and ACOS operating systems, making multi-vendor, real-time application layer visibility available in approximately 50% of commercial ADC market.

Figure 1: Best of Velocity 2012, The sFlow Standard

Equally important is the availability of sFlow support in leading open source web servers, load balancers, applications servers, hypervisors and operating systems, including: Apache, NGINX, Tomcat, Java, HAproxy, Hyper-V, Xen, KVM, Linux, Windows, Solaris, FreeBSD and AIX. The combination sFlow in ADCs and the application infrastructure behind them provides comprehensive end to end visibility in multi-tier, scale-out, application architectures.

Figure 1 shows the strategic role that ADCs (load balancers) play in controlling the flow of application requests, regulating admission, filtering, directing loads, and virtualizing services. RESTful control of ADCs combined with real-time visibility provides a powerful capability for flexing resources as demand changes, reducing costs and increasing performance as resources are closely matched to workloads.

What is unusual about diagram is the inclusion of the network. Application architects often give little thought to the network since its complexity is conveniently hidden behind APIs. Unfortunately, it is in the Continue reading

0

Physical switch hybrid OpenFlow example

Alcatel-Lucent OmniSwitch analytics driven control provided an example with a physical switch, using the Web Services API to send CLI controls to the switch as HTTP requests, the following screen shot shows the results:

Figure 1: Controller using HTTP / REST API

Integrated hybrid OpenFlow describes how the combination of normal forwarding combined with OpenFlow for control of large flows provides a scaleable and practical solution for traffic engineering. The article used the Mininet testbed to develop a DDoS mitigation controller consisting of the sFlow-RT real-time analytics engine to detect large flows and the Floodlight OpenFlow controller to push control rules to the software virtual switch in the testbed.

Figure 2: Performance aware software defined networking

The OmniSwitch supports hybrid mode OpenFlow and this article will evaluate the performance of a physical switch hybrid OpenFlow solution using the OmniSwitch. The following results were obtained when repeating the DDoS attack test using Floodlight and OpenFlow as the control mechanism:

Figure 3: OmniSwitch controller using hybrid OpenFlow

Figure 3 shows that implementing traffic controls using OpenFlow is considerably faster than those obtained using the HTTP API shown in Figure 1, cutting the time to implement controls from seconds to milliseconds.

Figure Continue reading

0

Alcatel-Lucent OmniSwitch analytics driven control

There are a many articles on this blog that demonstrate how real-time sFlow analytics driven control of switches using a Mininet testbed. This article is the first of a series that will shift the focus to physical switches and demonstrate different techniques for adapting network behavior to changing traffic.

Performance Aware SDN describes the theory behind analytics driven orchestration. The talk describes how fast controller response, programmatic configuration interfaces and consistent instrumentation of all the elements being orchestrated are pre-requisites for feedback control.

This article uses an Alcatel-Lucent OmniSwitch 6900 as an example. The switch has hardware sFlow support for line rate visibility on all ports, and support for OpenFlow and a RESTful configuration API to deploy control actions. In this example a basic DDoS mitigation filtering function will be triggered when large flood attacks are detected. The script is based on the version described in the article Integrated hybrid OpenFlow, but modified to use the OmniSwitch RESTful API.
RESTful control of switches describes how RESTFul configuration access to switches can be used to develop simple, controller-less SDN solutions. In this example the controller application is implemented using JavaScript that runs within the sFlow-RT analytics engine. The script has Continue reading