Linux 4.11 kernel extends packet sampling support
Introduce psample, a general way for kernel modules to sample packets, without being tied to any specific subsystem. This netlink channel can be used by tc, iptables, etc. and allow to standardize packet sampling in the kernel commitThe psample netlink channel delivers sampled packet headers along with associated metadata from the Linux kernel to user space. The psample fields map directly into sFlow Version 5 sampled_header export structures:
| netlink psample | sFlow | Description |
|---|---|---|
| PSAMPLE_ATTR_IIFINDEX | input | Interface packet was received on. |
| PSAMPLE_ATTR_OIFINDEX | output | Interface packet was sent on. |
| PSAMPLE_ATTR_SAMPLE_GROUP | data source | The location within network device that generated packet sample. |
| PSAMPLE_ATTR_GROUP_SEQ | drops | Number of times that the sFlow agent detected that a packet marked to be sampled was dropped due to lack of resources. Agent calculates drops by tracking discontinuities in PSAMPLE_ATTR_GROUP_SEQ |
| PSAMPLE_ATTR_SAMPLE_RATE | sampling_rate | The Sampling Rate specifies the ratio of packets observed at the Data Source to the samples generated. For example a sampling rate of 100 specifies that, on Continue reading |
Arista eAPI
In the diagram, the sFlow-RT analytics engine receives streaming sFlow telemetry, provides real-time network-wide visibility, and automatically applies controls using eAPI to optimize forwarding, block denial of service attacks, or capture suspicious traffic.
Arista eAPI 101 describes the JSON RPC interface for programmatic control of Arista switches. The following eapi.js script shows how eAPI requests can be made using sFlow-RT's JavaScript API:
function runCmds(proto, agent, usr, pwd, cmds) {
var req = {
jsonrpc:'2.0',id:'sflowrt',method:'runCmds',
params:{version:1,cmds:cmds,format:'json'}
};
var url = (proto || 'http')+'://'+agent+'/command-api';
var resp = http(url,'post','application/json',JSON.stringify(req),usr,pwd);
if(!resp) throw "no response";
resp = JSON.parse(resp);
if(resp.error) throw resp.error.message;
return resp.result;
}The following test.js script demonstrates the eAPI functionality with a basic show request: include('eapi.js');
var result = runCmds('http','10.0.0.90','admin','arista',['show hostname']);
logInfo(JSON.stringify(result));Starting sFlow-RT: env "RTPROP=-Dscript.file=test.js" ./start.shRunning the script generates the following output:
2017-07-10T14:00:06-0700 Continue reading
Real-time DDoS mitigation using sFlow and BGP FlowSpec
The BGP FlowSpec extension (RFC 5575: Dissemination of Flow Specification Rules) provides a method of transmitting traffic filters that selectively block the attack traffic while allowing normal traffic to pass. BGP FlowSpec support has recently been added to sFlow-RT and this article demonstrates the new capability.
The following flowspec.js sFlow-RT script detects and blocks UDP-Based Amplification attacks:
var router = '10.0.0.141';
var id = '10.0.0.70';
var as = 65141;
var thresh = 1000;
var block_minutes = 1;
setFlow('udp_target',{keys:'ipdestination,udpsourceport',value:'frames'});
setThreshold('attack',{metric:'udp_target', value:thresh, byFlow:true});
bgpAddNeighbor(router,as,id,{flowspec:true});
var Continue reading
BGP FlowSpec on white box switch
Unfortunately, FlowSpec is currently only available on high end routing devices and so experimenting with the technology is expensive. Looking for an alternative, Cumulus Linux is an open Linux platform that allows users to install Linux packages and develop their own software.
This article describes a proof of concept implementation of basic FlowSpec functionality using ExaBGP installed on a free Cumulus VX virtual machine. The same solution can be run on inexpensive commodity white box hardware to deliver terabit traffic filtering in a production network.
First, install latest version of ExaBGP on the Cumulus Linux switch:
curl -L https://github.com/Exa-Networks/exabgp/archive/4.0.0.tar.gz | tar zxNow define the handler, acl.py, that will convert BGP FlowSpec updates into standard Linux netfilter/iptables entries used by Cumulus Linux to specify hardware ACLs (see Netfilter - ACLs):
#!/usr/bin/python
import json
import re
from os import listdir,remove
from os.path import isfile
from Continue reading
Remotely Triggered Black Hole (RTBH) Routing
The following configuration is installed on the ce-router:
router bgp 65140The ce-router peers with the upstream service provider router ( Continue reading
bgp router-id 0.0.0.140
neighbor 10.0.0.70 remote-as 65140
neighbor 10.0.0.70 port 1179
neighbor 172.16.141.2 remote-as 65141
!
address-family ipv4 unicast
neighbor 10.0.0.70 allowas-in
neighbor 10.0.0.70 route-map blackhole-in in
exit-address-family
!
ip community-list standard blackhole permit 65535:666
!
route-map blackhole-in permit 20
match community blackhole
match ip address prefix-len 32
set ip next-hop 192.0.2.1
Arista EOS telemetry
Linux as a Switch Operating System: Five Lessons Learned identifies benefits of using Linux as the basis for EOS. In this context, the Linux operating system made it easy to port the Host sFlow agent, use standard Linux package management (RPM Package Manager), and gather metrics using standard Linux APIs. A new eAPI module automatically synchronizes the Host sFlow daemon with the EOS sFlow configuration.
The following sflowtool output shows the additional metrics contributed by a Host sFlow agent installed on an Arista switch:
startDatagram =================================
datagramSourceIP 172.17.0.1
datagramSize 704
unixSecondsUTC 1490843418
datagramVersion 5
agentSubId 100000
agent 10.0.0.90
packetSequenceNo 714
sysUpTime 0
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 714
sourceId 2:1
counterBlock_tag 0:2001
counterBlock_tag 0:2010
udpInDatagrams 1459
udpNoPorts 16
udpInErrors 0
udpOutDatagrams 4765
udpRcvbufErrors 0
udpSndbufErrors 0
udpInCsumErrors 0
counterBlock_tag 0:2009
tcpRtoAlgorithm 1
tcpRtoMin 200
tcpRtoMax 120000
tcpMaxConn 4294967295
tcpActiveOpens 102
Continue reading
Nutanix
This article is based on a single hardware node running Nutanix Community Edition (CE), built following the instruction in Part I: How to setup a three-node NUC Nutanix CE cluster. If you don't have hardware readily available, the article, 6 Nested Virtualization Resources To Get You Started With Community Edition, describes how to run Nutanix CE as a virtual machine.The sFlow standard is widely supported by network equipment vendors, which combined with sFlow from each Nutanix appliance, delivers end to end visibility in the Nutanix cluster. The following screen captures from the free sFlowTrend tool are representative examples of the data available from the Nutanix appliance.
QUIC
An easy way to see if you have any QUIC traffic on your network is to use the standard sFlow instrumentation built into network switches. Configure the switches to send sFlow telemetry to an sFlow collector for visibility into network traffic.
For example, use Docker to run the sFlow-RT active-flows application to analyze the sFlow data stream:
docker run -p 6343:6343/udp -p 8008:8008 -d sflow/top-flowsAccess the web interface at http://localhost:8008/ and enter the following Flow Specification to monitor QUICK flows:
dns:ipsource,dns:ipdestination,quicpackettypeNote: Real-time domain name lookups describes how sFlow-RT incorporates DNS (Domain Name Service) requests in its real-time analytics pipeline so that traffic flows can be identified by domain name.
Telegraf, InfluxDB, Chronograf, and Kapacitor
The following telegraf.js script instructs sFlow-RT to periodically export host metrics to InfluxDB:
var influxdb = "http://10.0.0.56:8086/write?db=telegraf";
function sendToInfluxDB(msg) {
if(!msg || !msg.length) return;
var req = {
url:influxdb,
operation:'POST',
headers:{"Content-Type":"text/plain"},
body:msg.join('\n')
};
req.error = function(e) {
logWarning('InfluxDB POST failed, error=' + e);
}
try { httpAsync(req); }
catch(e) {
logWarning('bad request ' + req.url + ' ' + e);
}
}
var metric_names = [
Continue reading
Using Ganglia to monitor Linux services
Monitoring Linux services
This article describes how to use the open source Host sFlow agent to gather telemetry from daemons running under systemd.
Host sFlow systemd monitoring exports a standard set of metrics for each systemd service - the sFlow Host Structures extension defines metrics for Virtual Nodes (virtual machines, containers, etc.) that are used to export Xen, KVM, Docker, and Java resource usage. Exporting the standard metrics for systemd services provides interoperability with sFlow analyzers, allowing them to report on Linux services using existing virtual node monitoring capabilities.
While running daemons within containers helps systemd maintain control of the resources, it also provides a very useful abstraction for monitoring. For example, a single service (like the Apache web server) may consist of dozens of processes. Reporting on container level metrics abstracts away the per-process details and gives a view of the total resources consumed by the service. In addition, service metadata (like the service name) provides a useful way of identifying and grouping Continue reading
IPv6 Internet router using merchant silicon
The article describes a simple self contained solution that uses standard APIs and should be able to run on a variety of Linux based network operating systems, including: Cumulus Linux, Dell OS10, Arista EOS, and Cisco NX-OS.
The diagram shows the elements of the solution. Standard sFlow instrumentation embedded in the merchant silicon ASIC data plane in the white box switch provides real-time information on traffic flowing through the switch. The sFlow agent is configured to send the sFlow to an instance of sFlow-RT running on the switch. The Bird routing daemon is used to handle the BGP peering sessions and to install routes in the Linux kernel using the standard netlink interface. The network operating system in turn programs the switch ASIC with the kernel routes so that packets are forwarded by the switch hardware and not by the kernel software.
The key to this solution is Bird's multi-table capabilities. The full Internet Continue reading
Monitoring at Terabit speeds
From the SCinet web page, "The Fastest Network Connecting the Fastest Computers: SC16 will host the most powerful and advanced networks in the world – SCinet. Created each year for the conference, SCinet brings to life a very high-capacity network that supports the revolutionary applications and experiments that are a hallmark of the SC conference."
SC16 live real-time weathermaps provides additional demonstrations of high performance network monitoring.
SC16 live real-time weathermaps
From the SCinet web page, "The Fastest Network Connecting the Fastest Computers: SC16 will host the most powerful and advanced networks in the world – SCinet. Created each year for the conference, SCinet brings to life a very high-capacity network that supports the revolutionary applications and experiments that are a hallmark of the SC conference."
The real-time weathermap leverages industry standard sFlow instrumentation built into network switch and router hardware to provide scaleable monitoring of the SCinet network. Link colors are updated every second to reflect operational status and utilization of each link.
Connect to http://inmon.sc16.org/sflow-rt/app/OSiRIS-weather/html/ to see an animated diagram of the SC16 OSiRIS demonstration connecting SCinet with University of Michigan, Michigan State, Wayne Continue reading
Network performance monitoring
# tcpdump -ni eth0 tcpThe host TCP/IP stack continuously measured round trip time and estimates available bandwidth for each active connection as part of its normal operation. The tcpdump output shown above highlights timestamp information that is exchanged in TCP packets to provide the accurate round trip time measurements needed for reliable high speed data transfer.
11:29:28.949783 IP 10.0.0.162.ssh > 10.0.0.70.56174: Flags [P.], seq 1424968:1425312, ack 1081, win 218, options [nop,nop,TS val 2823262261 ecr 2337599335], length 344
11:29:28.950393 IP 10.0.0.70.56174 > 10.0.0.162.ssh: Flags [.], ack 1425312, win 4085, options [nop,nop,TS val 2337599335 ecr 2823262261], length 0
The open source Host sFlow agent already makes use of Berkeley Packet Filter (BPF) capability on Linux to efficiently sample packets and provide visibility into traffic flows. Adding support Continue reading
Real-time domain name lookups
First, use the dns.servers System Property is used to specify one or more DNS servers to handle the reverse lookup requests. For example, the following command uses Docker to run sFlow-RT with DNS lookups directed to server 10.0.0.1:
docker run -e "RTPROP=-Ddns.servers=10.0.0.1" \The following Python script dnspair.py uses the sFlow-RT REST API to define a flow and log the resulting flow records:
-p 8008:8008 -p 6343:6343/udp -d sflow/sflow-rt
#!/usr/bin/env pythonRunning the script generates the following output:
import requests
import json
flow = {'keys':'dns:ipsource,dns:ipdestination',
'value':'bytes','activeTimeout':10,'log':True}
requests.put('http://localhost:8008/flow/dnspair/json',data=json.dumps(flow))
flowurl = 'http://localhost:8008/flows/json?name=dnspair&maxFlows=10&timeout=60'
flowID = -1
while 1 == 1:
r = requests.get(flowurl + "&flowID=" + str(flowID))
if r.status_code != 200: break
flows = r.json()
if len(flows) == 0: continue
flowID = flows[0]["flowID"]
flows.reverse()
for f in flows:
print json.dumps(f,indent=1)
$ ./dnspair.py
{
"value": 233370.92322668363,
"end": 1476234478177,
"name": "dnspair",
"flowID": Continue reading
Collecting Docker Swarm service metrics
In this example, Docker Swarm is used to deploy a simple web service on a four node cluster:
docker service create --replicas 2 -p 80:80 --name apache httpd:2.4Next, the following script tests the agility of monitoring systems by constantly changing the number of replicas in the service:
#!/bin/bashThe above test is easy to set up and is a quick way to stress test monitoring systems and reveal accuracy and performance problems when they are confronted with container workloads.
while true
do
docker service scale apache=$(( ( RANDOM % 20 ) + 1 ))
sleep 30
done
Many approaches to gathering and recording metrics were developed for static environments and have a great deal of difficulty tracking rapidly changing container-based service pools without missing information, leaking resources, and slowing down. For example, each new container in Docker Swarm has unique name, e.g. apache.16.17w67u9157wlri7trd854x6q0. Monitoring solutions that record container names, or even worse, index data by container name, will suffer from bloated Continue reading
Docker 1.12 swarm mode elastic load balancing
Docker Built-In Orchestration Ready For Production: Docker 1.12 Goes GA describes the native swarm mode feature that integrates cluster management, virtual networking, and policy based deployment of services.
This article will demonstrate how real-time streaming telemetry can be used to construct an elastic load balancing solution that dynamically adjusts service capacity to match changing demand.
Getting started with swarm mode describes the steps to configure a swarm cluster. For example, following command issued on any of the Manager nodes deploys a web service on the cluster:
docker service create --replicas 2 -p 80:80 --name apache httpd:2.4And the following command raises the number of containers in the service pool from 2 to 4:
docker service scale apache=4Asynchronous Docker metrics describes how sFlow telemetry provides the real-time visibility required for elastic load balancing. The diagram shows how streaming telemetry allows the sFlow-RT controller to determine the load on the service pool so that it can use the Docker service API to automatically increase or decrease the size of the pool as demand changes. Elastic load balancing of the service pools ensures consistent service levels by adding additional resources if demand increases. In addition, efficiency is improved by releasing resources Continue reading
Asynchronous Docker metrics
This article describes how the latest version of the Host sFlow agent is able to track the performance of a rapidly changing population of Docker containers and export a real-time stream of standard sFlow metrics.
The diagram at the beginning of this article shows the sequence of messages, going from top to bottom, required to track a container. The Host sFlow agent first registers for container Continue reading
Triggered remote packet capture using filtered ERSPAN
Traditional hierarchical network designs were relatively straightforward to monitor using a packet broker since traffic flowed through a small number of core switches and so a small number of taps provided network wide visibility. The move to leaf and spine fabric architectures eliminates the performance bottleneck of core switches to deliver low latency and high bandwidth connectivity to data center applications. However, traditional packet brokers are less attractive since spreading traffic across many links with equal cost multi-path (ECMP) routing means that many more links need to be monitored.
This article will explore how the remote Selective Spanning capability in Cumulus Linux 3.0 combined with industry standard sFlow telemetry embedded in commodity switch hardware provides a cost effective alternative to traditional packet brokers.
Cumulus Linux uses iptables rules to specify packet capture sessions. For example, the following rule forwards packets with source IP 20.0.1.0 and destination IP 20.0.1.2 to a packet analyzer on host 20.0.2.2:
-A FORWARD --in-interface swp+ -s 20.0.0.2 -d 20. Continue reading