Peter
Author Archives: Peter
- Home → Author's archive:
Peter
CloudFlare DDoS Mitigation Pipeline
The Usenix Enigma 2016 talk from Marek Majkowski describes CloudFlare's automated DDoS mitigation solution. CloudFlare provides reverse proxy services for millions of web sites and their customers are frequently targets of DDoS attacks. The talk is well worth watching in its entirety to learn about their experiences.
Elements of the CloudFlare solution are readily accessible to anyone interested in building DDoS mitigation solutions. Industry standard sFlow instrumentation is widely supported by switch vendors. Download sFlow-RT analytics software and combine real-time DDoS detection with business policies to automate mitigation actions. A number of DDoS mitigation examples are Continue reading
SignalFx
This article describes a proof of concept demonstrating how SignalFx's cloud service can be used to cost effectively monitor large scale cloud infrastructure by leveraging standard sFlow instrumentation. SignalFx offers a free 14 day trial, making it easy to evaluate solutions based on this demonstration.
The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Metrics are pushed from sFlow-RT to SignalFx using the REST API.
Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts, virtual machines and containers and local services. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.
SignalFx's service is priced based on the number of data points that they need to store and they estimate a cost of $15 per host Continue reading
Dell OS10 SDN router demo
In this video from Dell's Network Field Day 11 (#NFD11) presentation, Madhu Santhanam demonstrates an interesting use case for the new OS10 switch operating system that was introduced at the event.
Podcast with Nick Buraglio and Brent Salisbury
Non-Blocking #1: SFlow With Peter Phaal Of InMon And SFlow.Org is a discussion between Brent Salisbury (networkstatic.net), Nick Buraglio (forwardingplane.net), and Peter Phaal (blog.sflow.com).
Web sites and tools mentioned in the podcast:
- sFlow.org
- Devices that support sFlow
- Software to analyze sFlow
- sFlow.org mailing list
- sFlow structures
- blog.sflow.com (incorrectly referenced as blog.sflow.org in the podcast)
- Host sFlow
- sflowtool
The podcast touches on a number of topics that have been explored in greater detail on this blog. The topics are listed in roughly the order they are mentioned in the podcast:
- Widespread support for sFlow among switch vendors
- Disaggregated flow cache
- ULOG
- Push vs Pull
- sFlow vs SNMP for interface counters
- Broadcom ASIC table utilization metrics, DevOps, and SDN
- Broadcom BroadView Instrumentation
- Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX
- SDN and large flows
- Probes
- Packet headers
- Network virtualization Continue reading
Demystifying NFV Infrastructure Hotspots
Another paper, Demystifying NFV Infrastructure Hotspots by Ramki Krishnan, Anoop Ghanwani, and Michael Tien, demonstrates how industry standard sFlow instrumentation build into physical and virtual switches can provide the comprehensive real-time analytics needed to manage NFV deployments.
Environmental metrics with Cumulus Linux
The smonctl command can be used to dump sensor data as JSON formatted text:
cumulus@cumulus$ smonctl -jThe following Python script, smon_sflow.py, invokes the command, parses the output, and posts a set of custom sFlow metrics:
[
{
"pwm_path": "/sys/devices/soc.0/ffe03100.i2c/i2c-1/1-004d",
"all_ok": "1",
"driver_hwmon": [
"fan1"
],
"min": 2500,
"cpld_path": "/sys/devices/ffe05000.localbus/ffb00000.CPLD",
"state": "OK",
"prev_state": "OK",
"msg": null,
"input": 8998,
"type": "fan",
"pwm1": 121,
"description": "Fan1",
"max": 29000,
"start_time": 1450228330,
"var": 15,
"pwm1_enable": 0,
"prev_msg": null,
"log_time": 1450228330,
"present": "1",
"target": 0,
"name": "Fan1",
"fault": "0",
"pwm_hwmon": [
"pwm1"
],
"driver_path": "/sys/devices/soc.0/ffe03100.i2c/i2c-1/1-004d",
"div": "4",
"cpld_hwmon": [
"fan1"
]
},
...
#!/usr/bin/env python
import json
import socket
from subprocess import check_output
res = check_output(["/usr/sbin/smonctl","-j"])
smon = json.loads(res)
fan_maxpc = 0
fan_down = 0
fan_up = 0
psu_down = 0
psu_up = 0
temp_maxpc = 0
temp_up = 0
temp_down = 0
for s in smon:
type = s["type"]
if(type == Continue reading
Custom events
The JQuery script can be embedded in a web page to provide timing information:
$(window).load(function(){
var samplingRate = 10;
if(samplingRate !== 1 && Math.random() > (1/samplingRate)) return;
setTimeout(function(){
if(window.performance) {
var t = window.performance.timing;
var msg = {
sampling_rate : samplingRate,
t_url : {type:"string",value:window.location.href},
t_useragent : {type:"string",value:navigator.userAgent},
t_loadtime : {type:"int32",value:t.loadEventEnd-t.navigationStart},
t_connecttime : {type:"int32",value:t.responseEnd-t.requestStart}
};
$.ajax({
url:"/navtiming.php",
method:"PUT",
contentType:"application/json",
data:JSON.stringify(msg)
});
}
}, 0);
});The script supports random sampling. In this case a samplingRate of 10 means that, on average, 1-in-10 page hits will generate a measurement record. Measurement records are sent back to the server where the navtiming.php script acts as a gateway, augmenting the measurements and sending them as custom sFlow events.<?php
$rawInput = file_get_contents("php://input");
$rec = json_decode($rawInput);
$rec->datasource = "navtime";
$rec->t_ip = array("type" => "ip", "value" => $_SERVER['REMOTE_ADDR']);
$msg=array("rtflow"=>$rec);
$sock = fsockopen("udp://localhost",36343,$errno,$errstr);
if(! $sock) { Continue reading
Custom metrics with Cumulus Linux
Host sFlow version 1.28.3 adds support for Custom Metrics. This article demonstrates how the extensive set of standard sFlow measurements can be augmented using custom metrics.
Recent releases of Cumulus Linux simplify the task by making machine readable JSON a supported output in command line tools. For example, the cl-bgp tool can be used to dump BGP summary statistics:
cumulus@leaf1$ sudo cl-bgp summary show json
{ "router-id": "192.168.0.80", "as": 65080, "table-version": 5, "rib-count": 9, "rib-memory": 1080, "peer-count": 2, "peer-memory": 34240, "peer-group-count": 1, "peer-group-memory": 56, "peers": { "swp1": { "remote-as": 65082, "version": 4, "msgrcvd": 52082, "msgsent": 52084, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" }, "swp2": { "remote-as": 65083, "version": 4, "msgrcvd": 52082, "msgsent": 52083, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" } }, Continue reading
Using a proxy to feed metrics into Ganglia
The GitHub gmond-proxy project demonstrates how a simple proxy can be used to map metrics retrieved through a REST API into Ganglia's gmond TCP protocol.
The following commands install the proxy on the sFlow collector - an Ubuntu 14.04 system that is already runnig sFlow-RT:
wget https://raw.githubusercontent.com/sflow-rt/gmond-proxy/master/gmond_proxy.pyThe following commands install Ganglia's gmetad collector and web user interface on the Ganglia server - an Ubuntu 14.04 system:
sudo mv gmond_proxy.py /etc/init.d/
sudo chown root:root /etc/init.d/gmond_proxy.py
sudo chmod 755 /etc/init.d/gmond_proxy.py
sudo service gmond_proxy.py start
sudo update-rc.d gmond_proxy.py start
sudo apt-get install gmetadNext edit the /etc/ganglia/gmetad.conf file and configure the proxy as a data source:
sudo apt-get install ganglia-webfrontend
cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled
data_source "my cluster" sflow-rtRestart the Apache and gmetad daemons:
sudo service gmetad restart
sudo service apache2 Continue reading
Broadcom BroadView Instrumentation
The highlighted Buffer congestion state / statistics capability in the slide refers to the BroadView Buffer Statistics Tracking (BST) instrumentation. The Memory Management Unit (MMU) is on-chip logic that manages how the on-chip packet buffers are organized. BST is a feature that enables tracking the usage of these buffers. It includes snapshot views of peak utilization of the on-chip buffer memory across queues, ports, priority group, service pools and the entire chip.
DDoS Blackhole
The screen capture above shows a simulated DNS amplification attack. The Top Targets chart is a real-time view of external traffic to on-site IP addresses. The red line indicates the threshold that has been set at 10,000 packets per second and it is clear that traffic to address 192.168.151.4 exceeds the threshold. The Top Protocols chart below shows that the increase in traffic is predominantly DNS. The Controls chart shows that a control was added the instant the traffic crossed the threshold.
OVN service injection demonstration
Enabling extensibility in OVN, by Gal Sagie, Huawei and Liran Schour, IBM, Open vSwitch 2015 Fall Conference describes a method for composing actions from an external application with actions installed by the Open Network Virtualization (OVN) controller.
An API allows services to be attached to logical topology elements in the OVN logical topology, resulting in a table in the OVN logical flow table that is under the controller of the external service. Changes to the logical table are then automatically instantiated as concrete flows in the Open vSwitch instances responsible for handling the packets in the flow.
The demo presented involves detecting large "Elephant" flows using sFlow instrumentation embedded in Open vSwitch. Once a large flow is detected, logical flows are instantiated in the OVN controller to mark the packets. The concrete marking rules are inserted in the Open vSwitch packet processing pipelines handling the logical flow's packets. In the demo, the marked packets are then diverted by the physical network to a dedicated optical circuit.
There are a number of interesting traffic control use cases described on this blog that could leverage the capabilities of Open vSwitch using this approach:
Open vSwitch 2015 Fall Conference
Open vSwitch is an open source software virtual switch that is popular in cloud environments such as OpenStack. Open vSwitch is a standard Linux component that forms the basis of a number of commercial and open source solutions for network virtualization, tenant isolation, and network function virtualization (NFV) - implementing distributed virtual firewalls and routers.The recent Open vSwitch 2015 Fall Conference agenda included a wide variety speakers addressing a range of topics, including: Open Network Virtualization (OVN), containers, service chaining, and network function virtualization (NFV).
The video above is a recording of the following sFlow related talk from the conference:
New OVS instrumentation features aimed at real-time monitoring of virtual networks (Peter Phaal, InMon)This talk is a follow up to an Open vSwitch 2014 Fall Conference talk on the Continue reading
The talk will describe the recently added packet-sampling mechanism that returns the full list of OVS actions from the kernel. A demonstration will show how the OVS sFlow agent uses this mechanism to provide real-time tunnel visibility. The motivation for this visibility will be discussed, using examples such as end-to-end troubleshooting across physical and virtual networks, and tuning network packet paths by influencing workload placement in a VM/Container environment.
Network virtualization visibility demo
The diagram above shows the demonstration testbed. It consists of a leaf and spine network connecting two hosts, each of which is running a pair of Docker containers connected to Open vSwitch (OVS). The vSwitches are controlled by Open Virtual Network (OVN), which has been configured to create two logical switches, the first connecting the left most containers on each host and the second connecting the right most containers. The testbed is described in more detail in Open Virtual Network (OVN) and is built from free components and can easily be replicated.
The dashboard in the video illustrates the end to end visibility that is possible by combining standard sFlow instrumentation in the physical switches with sFlow instrumentation in Open vSwitch and Host sFlow agents on the servers.
The diagram on the left of the dashboard shows a logical map of the elements in the testbed. The top panel shows the two logical switches Continue reading
SC15 live real-time weathermap
From the SCinet web page, "SCinet brings to life a very high-capacity network that supports the revolutionary applications and experiments that are a hallmark of the SC conference. SCinet will link the convention center to research and commercial networks around the world. In doing so, SCinet serves as the platform for exhibitors to demonstrate the advanced computing resources of their home institutions and elsewhere by supporting a wide variety of bandwidth-driven applications including supercomputing and cloud computing."
The real-time weathermap leverages industry standard sFlow instrumentation built into network switch and router hardware to provide scaleable monitoring of the over 6 Terrabit/s aggregate link capacity comprising the SCinet network. Link colors are updated every second to reflect operational status and utilization of each link.
Clicking on a link in the map pops up a 1 second resolution strip chart showing the protocol mix carried by the link.
The SCinet real-time weathermap was constructed using open source components running on the sFlow-RT real-time analytics engine. Download sFlow-RT and see what Continue reading
sFlow Test
Many of the tests can be run while the switches are in production and are a useful way of verifying that a switch is configured and operating correctly.
The stress tests can be scaled to run without specialized equipment. For example, the recommended sampling rate for 10G links in production is 1-in-10,000. Driving a switch with 48x10G ports to 30% of total capacity would require a load generator capable of generating 288Gbit/s. However, dropping the sampling rate to 1-in-100 and generating a load of 2.88Gbit/s is an equivalent test of the sFlow agent's performance and can be achieved by two moderately powerful servers with 10G network adapters.
For example, using the test setup above, run an iperf server on Server2:
iperf -suThen run the following sequence of tests on Server1:
#!/bin/bash
RT="10.0.0. Continue reading
Active Route Manager
There is an interesting parallel with the Open vSwitch architecture, see Open vSwitch performance monitoring, which maintains a cache of active flows in the Linux kernel to accelerate forwarding. In the SDN routing case, active prefixes are pushed to the switch ASIC in order to bypass the slower software router.In this example, the software is being used in passive mode, estimating the cache hit / miss rates without offloading routes. The software has been configured to manage a cache of 10,000 prefixes. The first screen shot shows the cache warming up.
The first panel shows routes being learned from the route reflector: the upper chart shows the approximately 600,000 routes being learned from the BGP route reflector, and the lower chart shows the rate at which Continue reading
Fabric View
Fabric View solves the visibility challenge by using the industry standard sFlow instrumentation built into most data center switches. Fabric View represents the fabric as if it were a single large chassis switch, treating each leaf switch as a line card and the spine switches as the backplane. The result is an intuitive tool that is easily understood by anyone familiar with traditional networks.
Fabric View provides real-time, second-by-second visibility to traffic, identifying top talkers, protocols, tenants, tunneled traffic, Continue reading
Real-time analytics and control applications
The flow-graph application, shown above, generates a real-time graph of communication between hosts. The application uses a simple sFlow-RT script to track associations between hosts based on their communication patterns and plots the results using the vis.js dynamic, browser based visualization library. This example can be modified to track different types of relationship and extended to incorporate other popular data visualization libraries such as D3.js.
Each of the examples has a server-side component that uses sFlow-RT's script API to collect, analyze, and export measurements. An HTML5 client side user interface connects to the server and presents the data.
The sFlow-RT analytics engine is a highly scaleable platform for processing sFlow measurements Continue reading
Open Virtual Network (OVN)
Open Virtual Network (OVN) is an open source network virtualization solution built as part of the Open vSwitch (OVS) project. OVN provides layer 2/3 virtual networking and firewall services for connecting virtual machines and Linux containers.
OVN is built on the same architectural principles as VMware's commercial NSX and offers the same core network virtualization capability — providing a free alternative that is likely to see rapid adoption in open source orchestration systems, Mirantis: Why the Open Virtual Network (OVN) matters to OpenStack.
This article uses OVN as an example, describing a testbed which demonstrates how the standard sFlow instrumentation build into the physical and virtual switches provides the end-to-end visibility required to manage large scale network virtualization and deliver reliable services.
Open Virtual Network
The Northbound DB provides a way to describe the logical networks that are required. The database abstracts away implementation details which are handled by the ovn-northd and ovn-controllers and presents an easily consumable network virtualization service to orchestration tools like OpenStack.
The purple tables on the left describe a simple logical switch LS1 that has two logical ports LP1 and LP2 with MAC addresses AA and BB respectively. The green tables on the right show Continue reading