Microservices
 |
| Figure 1: Visibility and the software defined data center |
While I genuinely believe that the network will play an immensely strategic role in the microservices world, inspecting and storing billions of API calls on a daily basis will require significant computing and storage resources. In addition, deep packet inspection could be challenging at line rates; so, sampling, at the expense of full visibility, might be an alternative. Finally, network traffic analysis must be combined with service-level telemetry data (that we already collect today) in order to get a comprehensive and in-depth picture of the distributed application.Sampling isn't just an alternative, sampling is the key to making large scale microservice visibility a reality. Shrink ray describes how sampling acts as a scaling function, reducing the task of monitoring large scale microservice infrastructure from an intractable measurement and big data problem to a lightweight real-time data center wide visibility solution for monitoring, managing, Continue reading
Open vSwitch version 2.5 released
- sFlow agent now reports tunnel and MPLS structures.The sFlow Tunnel Structures specification enhances visibility into network virtualization by capturing encapsulation / decapsulation actions performed by tunnel end points. In many network virtualization implementations VXLAN, GRE, Geneve tunnels are terminate in Open vSwitch and so the new feature has broad application.
...
- Add experimental version of OVN. OVN, the Open Virtual Network, is a
system to support virtual network abstraction. OVN complements the
existing capabilities of OVS to add native support for virtual network
abstractions, such as virtual L2 and L3 overlays and security groups.
The second related feature is the inclusion of the Open Virtual Network (OVN), providing a simple method of building virtual networks for OpenStack and Docker.
The following articles provide additional background:
Linux bridge, macvlan, ipvlan, adapters
In addition to the new BPF capability, hsflowd has a couple of other ways to monitor traffic:
- iptables, add a statistic rule to the iptables firewall to add traffic monitoring
- Open vSwitch, has built-in sFlow instrumentation that can be configured by hsflowd.
The following commands build and install hsflowd on an Ubuntu 14.03 host:
sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libpcap-dev
sudo apt-get install git
git clone https://github. Continue reading
CloudFlare DDoS Mitigation Pipeline
The Usenix Enigma 2016 talk from Marek Majkowski describes CloudFlare's automated DDoS mitigation solution. CloudFlare provides reverse proxy services for millions of web sites and their customers are frequently targets of DDoS attacks. The talk is well worth watching in its entirety to learn about their experiences.
Elements of the CloudFlare solution are readily accessible to anyone interested in building DDoS mitigation solutions. Industry standard sFlow instrumentation is widely supported by switch vendors. Download sFlow-RT analytics software and combine real-time DDoS detection with business policies to automate mitigation actions. A number of DDoS mitigation examples are Continue reading
SignalFx
This article describes a proof of concept demonstrating how SignalFx's cloud service can be used to cost effectively monitor large scale cloud infrastructure by leveraging standard sFlow instrumentation. SignalFx offers a free 14 day trial, making it easy to evaluate solutions based on this demonstration.
The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Metrics are pushed from sFlow-RT to SignalFx using the REST API.
Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts, virtual machines and containers and local services. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.
SignalFx's service is priced based on the number of data points that they need to store and they estimate a cost of $15 per host Continue reading
Dell OS10 SDN router demo
In this video from Dell's Network Field Day 11 (#NFD11) presentation, Madhu Santhanam demonstrates an interesting use case for the new OS10 switch operating system that was introduced at the event.
Podcast with Nick Buraglio and Brent Salisbury
Non-Blocking #1: SFlow With Peter Phaal Of InMon And SFlow.Org is a discussion between Brent Salisbury (networkstatic.net), Nick Buraglio (forwardingplane.net), and Peter Phaal (blog.sflow.com).
Web sites and tools mentioned in the podcast:
- sFlow.org
- Devices that support sFlow
- Software to analyze sFlow
- sFlow.org mailing list
- sFlow structures
- blog.sflow.com (incorrectly referenced as blog.sflow.org in the podcast)
- Host sFlow
- sflowtool
The podcast touches on a number of topics that have been explored in greater detail on this blog. The topics are listed in roughly the order they are mentioned in the podcast:
- Widespread support for sFlow among switch vendors
- Disaggregated flow cache
- ULOG
- Push vs Pull
- sFlow vs SNMP for interface counters
- Broadcom ASIC table utilization metrics, DevOps, and SDN
- Broadcom BroadView Instrumentation
- Rapidly detecting large flows, sFlow vs. NetFlow/IPFIX
- SDN and large flows
- Probes
- Packet headers
- Network virtualization Continue reading
Demystifying NFV Infrastructure Hotspots
Another paper, Demystifying NFV Infrastructure Hotspots by Ramki Krishnan, Anoop Ghanwani, and Michael Tien, demonstrates how industry standard sFlow instrumentation build into physical and virtual switches can provide the comprehensive real-time analytics needed to manage NFV deployments.
Environmental metrics with Cumulus Linux
The smonctl command can be used to dump sensor data as JSON formatted text:
cumulus@cumulus$ smonctl -jThe following Python script, smon_sflow.py, invokes the command, parses the output, and posts a set of custom sFlow metrics:
[
{
"pwm_path": "/sys/devices/soc.0/ffe03100.i2c/i2c-1/1-004d",
"all_ok": "1",
"driver_hwmon": [
"fan1"
],
"min": 2500,
"cpld_path": "/sys/devices/ffe05000.localbus/ffb00000.CPLD",
"state": "OK",
"prev_state": "OK",
"msg": null,
"input": 8998,
"type": "fan",
"pwm1": 121,
"description": "Fan1",
"max": 29000,
"start_time": 1450228330,
"var": 15,
"pwm1_enable": 0,
"prev_msg": null,
"log_time": 1450228330,
"present": "1",
"target": 0,
"name": "Fan1",
"fault": "0",
"pwm_hwmon": [
"pwm1"
],
"driver_path": "/sys/devices/soc.0/ffe03100.i2c/i2c-1/1-004d",
"div": "4",
"cpld_hwmon": [
"fan1"
]
},
...
#!/usr/bin/env python
import json
import socket
from subprocess import check_output
res = check_output(["/usr/sbin/smonctl","-j"])
smon = json.loads(res)
fan_maxpc = 0
fan_down = 0
fan_up = 0
psu_down = 0
psu_up = 0
temp_maxpc = 0
temp_up = 0
temp_down = 0
for s in smon:
type = s["type"]
if(type == Continue reading
Custom events
The JQuery script can be embedded in a web page to provide timing information:
$(window).load(function(){
var samplingRate = 10;
if(samplingRate !== 1 && Math.random() > (1/samplingRate)) return;
setTimeout(function(){
if(window.performance) {
var t = window.performance.timing;
var msg = {
sampling_rate : samplingRate,
t_url : {type:"string",value:window.location.href},
t_useragent : {type:"string",value:navigator.userAgent},
t_loadtime : {type:"int32",value:t.loadEventEnd-t.navigationStart},
t_connecttime : {type:"int32",value:t.responseEnd-t.requestStart}
};
$.ajax({
url:"/navtiming.php",
method:"PUT",
contentType:"application/json",
data:JSON.stringify(msg)
});
}
}, 0);
});The script supports random sampling. In this case a samplingRate of 10 means that, on average, 1-in-10 page hits will generate a measurement record. Measurement records are sent back to the server where the navtiming.php script acts as a gateway, augmenting the measurements and sending them as custom sFlow events.<?php
$rawInput = file_get_contents("php://input");
$rec = json_decode($rawInput);
$rec->datasource = "navtime";
$rec->t_ip = array("type" => "ip", "value" => $_SERVER['REMOTE_ADDR']);
$msg=array("rtflow"=>$rec);
$sock = fsockopen("udp://localhost",36343,$errno,$errstr);
if(! $sock) { Continue reading
Custom metrics with Cumulus Linux
Host sFlow version 1.28.3 adds support for Custom Metrics. This article demonstrates how the extensive set of standard sFlow measurements can be augmented using custom metrics.
Recent releases of Cumulus Linux simplify the task by making machine readable JSON a supported output in command line tools. For example, the cl-bgp tool can be used to dump BGP summary statistics:
cumulus@leaf1$ sudo cl-bgp summary show json
{ "router-id": "192.168.0.80", "as": 65080, "table-version": 5, "rib-count": 9, "rib-memory": 1080, "peer-count": 2, "peer-memory": 34240, "peer-group-count": 1, "peer-group-memory": 56, "peers": { "swp1": { "remote-as": 65082, "version": 4, "msgrcvd": 52082, "msgsent": 52084, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" }, "swp2": { "remote-as": 65083, "version": 4, "msgrcvd": 52082, "msgsent": 52083, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" } }, Continue reading
Using a proxy to feed metrics into Ganglia
The GitHub gmond-proxy project demonstrates how a simple proxy can be used to map metrics retrieved through a REST API into Ganglia's gmond TCP protocol.
The following commands install the proxy on the sFlow collector - an Ubuntu 14.04 system that is already runnig sFlow-RT:
wget https://raw.githubusercontent.com/sflow-rt/gmond-proxy/master/gmond_proxy.pyThe following commands install Ganglia's gmetad collector and web user interface on the Ganglia server - an Ubuntu 14.04 system:
sudo mv gmond_proxy.py /etc/init.d/
sudo chown root:root /etc/init.d/gmond_proxy.py
sudo chmod 755 /etc/init.d/gmond_proxy.py
sudo service gmond_proxy.py start
sudo update-rc.d gmond_proxy.py start
sudo apt-get install gmetadNext edit the /etc/ganglia/gmetad.conf file and configure the proxy as a data source:
sudo apt-get install ganglia-webfrontend
cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled
data_source "my cluster" sflow-rtRestart the Apache and gmetad daemons:
sudo service gmetad restart
sudo service apache2 Continue reading
Broadcom BroadView Instrumentation
The highlighted Buffer congestion state / statistics capability in the slide refers to the BroadView Buffer Statistics Tracking (BST) instrumentation. The Memory Management Unit (MMU) is on-chip logic that manages how the on-chip packet buffers are organized. BST is a feature that enables tracking the usage of these buffers. It includes snapshot views of peak utilization of the on-chip buffer memory across queues, ports, priority group, service pools and the entire chip.
DDoS Blackhole
The screen capture above shows a simulated DNS amplification attack. The Top Targets chart is a real-time view of external traffic to on-site IP addresses. The red line indicates the threshold that has been set at 10,000 packets per second and it is clear that traffic to address 192.168.151.4 exceeds the threshold. The Top Protocols chart below shows that the increase in traffic is predominantly DNS. The Controls chart shows that a control was added the instant the traffic crossed the threshold.
OVN service injection demonstration
Enabling extensibility in OVN, by Gal Sagie, Huawei and Liran Schour, IBM, Open vSwitch 2015 Fall Conference describes a method for composing actions from an external application with actions installed by the Open Network Virtualization (OVN) controller.
An API allows services to be attached to logical topology elements in the OVN logical topology, resulting in a table in the OVN logical flow table that is under the controller of the external service. Changes to the logical table are then automatically instantiated as concrete flows in the Open vSwitch instances responsible for handling the packets in the flow.
The demo presented involves detecting large "Elephant" flows using sFlow instrumentation embedded in Open vSwitch. Once a large flow is detected, logical flows are instantiated in the OVN controller to mark the packets. The concrete marking rules are inserted in the Open vSwitch packet processing pipelines handling the logical flow's packets. In the demo, the marked packets are then diverted by the physical network to a dedicated optical circuit.
There are a number of interesting traffic control use cases described on this blog that could leverage the capabilities of Open vSwitch using this approach:
Open vSwitch 2015 Fall Conference
Open vSwitch is an open source software virtual switch that is popular in cloud environments such as OpenStack. Open vSwitch is a standard Linux component that forms the basis of a number of commercial and open source solutions for network virtualization, tenant isolation, and network function virtualization (NFV) - implementing distributed virtual firewalls and routers.The recent Open vSwitch 2015 Fall Conference agenda included a wide variety speakers addressing a range of topics, including: Open Network Virtualization (OVN), containers, service chaining, and network function virtualization (NFV).
The video above is a recording of the following sFlow related talk from the conference:
New OVS instrumentation features aimed at real-time monitoring of virtual networks (Peter Phaal, InMon)This talk is a follow up to an Open vSwitch 2014 Fall Conference talk on the Continue reading
The talk will describe the recently added packet-sampling mechanism that returns the full list of OVS actions from the kernel. A demonstration will show how the OVS sFlow agent uses this mechanism to provide real-time tunnel visibility. The motivation for this visibility will be discussed, using examples such as end-to-end troubleshooting across physical and virtual networks, and tuning network packet paths by influencing workload placement in a VM/Container environment.
Network virtualization visibility demo
The diagram above shows the demonstration testbed. It consists of a leaf and spine network connecting two hosts, each of which is running a pair of Docker containers connected to Open vSwitch (OVS). The vSwitches are controlled by Open Virtual Network (OVN), which has been configured to create two logical switches, the first connecting the left most containers on each host and the second connecting the right most containers. The testbed is described in more detail in Open Virtual Network (OVN) and is built from free components and can easily be replicated.
The dashboard in the video illustrates the end to end visibility that is possible by combining standard sFlow instrumentation in the physical switches with sFlow instrumentation in Open vSwitch and Host sFlow agents on the servers.
The diagram on the left of the dashboard shows a logical map of the elements in the testbed. The top panel shows the two logical switches Continue reading
SC15 live real-time weathermap
From the SCinet web page, "SCinet brings to life a very high-capacity network that supports the revolutionary applications and experiments that are a hallmark of the SC conference. SCinet will link the convention center to research and commercial networks around the world. In doing so, SCinet serves as the platform for exhibitors to demonstrate the advanced computing resources of their home institutions and elsewhere by supporting a wide variety of bandwidth-driven applications including supercomputing and cloud computing."
The real-time weathermap leverages industry standard sFlow instrumentation built into network switch and router hardware to provide scaleable monitoring of the over 6 Terrabit/s aggregate link capacity comprising the SCinet network. Link colors are updated every second to reflect operational status and utilization of each link.
Clicking on a link in the map pops up a 1 second resolution strip chart showing the protocol mix carried by the link.
The SCinet real-time weathermap was constructed using open source components running on the sFlow-RT real-time analytics engine. Download sFlow-RT and see what Continue reading
sFlow Test
Many of the tests can be run while the switches are in production and are a useful way of verifying that a switch is configured and operating correctly.
The stress tests can be scaled to run without specialized equipment. For example, the recommended sampling rate for 10G links in production is 1-in-10,000. Driving a switch with 48x10G ports to 30% of total capacity would require a load generator capable of generating 288Gbit/s. However, dropping the sampling rate to 1-in-100 and generating a load of 2.88Gbit/s is an equivalent test of the sFlow agent's performance and can be achieved by two moderately powerful servers with 10G network adapters.
For example, using the test setup above, run an iperf server on Server2:
iperf -suThen run the following sequence of tests on Server1:
#!/bin/bash
RT="10.0.0. Continue reading
Active Route Manager
There is an interesting parallel with the Open vSwitch architecture, see Open vSwitch performance monitoring, which maintains a cache of active flows in the Linux kernel to accelerate forwarding. In the SDN routing case, active prefixes are pushed to the switch ASIC in order to bypass the slower software router.In this example, the software is being used in passive mode, estimating the cache hit / miss rates without offloading routes. The software has been configured to manage a cache of 10,000 prefixes. The first screen shot shows the cache warming up.
The first panel shows routes being learned from the route reflector: the upper chart shows the approximately 600,000 routes being learned from the BGP route reflector, and the lower chart shows the rate at which Continue reading