sFlow Archives - Page 10 of 14 - NetworkingNexus.net

Multi-tenant sFlow

This article discusses how real-time sFlow telemetry can be shared with network tenants to provide each tenant with a real-time view of their slice of the shared resources. The diagram shows a simple network with two tenants, Tenant A and Tenant B, each assigned their own subnet, 10.0.0.0/24 and 10.0.1.0/24 respectively.

One option would be to simply replicate the sFlow datagrams and send copies to both tenants. Forwarding using sflowtool describes how sflowtool can be used to replicate and forward sFlow and sFlow-RT can be configured to forward sFlow using its REST API:

curl -H "Content-Type:application/json" \
-X PUT --data '{"address":"10.0.0.1","port":6343}' \
http://127.0.0.1:8008/forwarding/TenantA/json

However, there are serious problems with this approach:

Private information about Tenant B's traffic is leaked to Tenant A.
Information from internal links within the network (i.e. links between s1, s2, s3 and s4) is leaked to Tenant A.
Duplicate data from each network hop is likely to cause Tenant A to over-estimate their traffic.

The sFlow-RT multi-tenant forwarding function addresses these challenges. The first task is to provide sFlow-RT with an accurate network topology specifying the internal links connecting the Continue reading

Network visibility with Docker

Microservices describes the critical role that network visibility provides as a common point of reference for monitoring, managing and securing the interactions between the numerous and diverse distributed service instances in a microservices deployment.

Industry standard sFlow is well placed to give network visibility into the Docker infrastructure used to support microservices. The sFlow standard is widely supported by data center switch vendors (Cisco, Arista, Juniper, Dell, HPE, Brocade, Cumulus, etc.) providing a cost effective and scaleable method of monitoring the physical network infrastructure. In addition, Linux bridge, macvlan, ipvlan, adapters described how sFlow is also an efficient means of leveraging instrumentation built into the Linux kernel to extend visibility into Docker host networking.

The following commands build the Host sFlow binary package from sources on an Ubuntu 14.04 system:

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libpcap-dev
sudo apt-get install wget
wget https://github.com/sflow/host-sflow/archive/v1.29.1.tar.gz
tar -xvzf v1.29.1.tar.gz
cd host-sflow-1.29.1
make DOCKER=yes PCAP=yes deb

This resulting hsflowd_1.29.1-1_amd64.deb package can be copied and installed on all the hosts in the Docker cluster using configuration management tools such as Puppet, Chef, Ansible, etc.

This Continue reading

Lasers!

Cool! You mean that I actually have frickin' switches with frickin' laser beams attached to their frickin' ports?

Dr. Evil is right, lasers are cool! The draft sFlow Optical Interface Structures specification exports metrics gathered from instrumentation built into Small Form-factor Pluggable (SFP) and Quad Small Form-factor Pluggable (QSFP) optics modules. This article provides some background on optical modules and discusses the value of including optical metrics in the sFlow telemetry stream exported from switches and hosts.

Pluggable optical modules are intelligent devices that do more than simply convert between optical and electrical signals. The functional diagram below shows the elements within a pluggable optical module.

The transmit and receive functions are shown in the upper half of the diagram. Incoming optical signals are received on a fiber and amplified as they are converted to electrical signals that can be handled by the switch. Transmit data drives the modulation of a laser diode which transmits the optical signal down a fiber.

The bottom half of the diagram shows the management components. Power, voltage and temperature sensors are monitored and the results are written into registers in an EEPROM that are accessible via a management interface.

The proposed sFlow extension standardizes Continue reading

Minimizing cost of visibility

Visibility allows orchestration systems (OpenDaylight, ONOS, OpenStack Heat, Kubernetes, Docker Storm, Apache Mesos, etc.) to adapt to changing demand by targeting resources where they are needed to increase efficiency, improve performance, and reduce costs. However, the overhead of monitoring must be low in order to realize the benefits.

An analogous observation that readers may be familiar with is the importance of minimizing costs when investing in order to maximize returns - see Vanguard Principle 3: Minimize cost

Suppose that a 100 server pool is being monitored and visibility will allow the orchestration system to realize a 10% improvement by better workload scheduling and placement - increasing the pool's capacity by 10% without the need to add an additional 10 servers and saving the associated CAPEX/OPEX costs.

The chart shows the impact that measurement overhead has in realizing the potential gains in this example. If the measurement overhead is 0%, then the 10% performance gain is fully realized. However, even a relatively modest 2% measurement overhead reduces the potential improvement to just under 8% (over a 20% drop in the potential gains). A 9% measurement overhead wipes out the potential efficiency gain and measurement overheads greater than 9% result in Continue reading

Docker network visibility demonstration

The 2 minute live demonstration shows how the open source Host sFlow agent can be used to efficiently monitor Docker networking in production environments. The demonstration shows real-time tracking of 30Gbit/s traffic flows using less than 1% of a single processor core.

Internet Exchange (IX) Metrics

IX Metrics has been released on GitHub, https://github.com/sflow-rt/ix-metrics. The application provides real-time monitoring of traffic between members in an Internet Exchange (IX).

Close monitoring of exchange traffic is critical to operations:

Ensure that there is sufficient capacity to accommodate new and existing members.
Ensure that all traffic sources are accounted for and that there are no unauthorized connections.
Ensure that only allowed traffic types are present.
Ensure that non-unicast traffic is strictly controlled.
Ensure that packet size policies are controlled to avoid loss due to MTU mismatches.

IX Metrics imports information about exchange members using the IX Member List JSON Schema. The member information is used to create traffic analytics and traffic is checked against the schema to identify errors, for example, if a member is using a MAC address that isn't listed.

The measurements from the exchange infrastructure are useful to members since it allows them to easily see how much traffic they are exchanging with other members through their peering relationships. This information is easy to collect using the exchange infrastructure, but much harder for members to determine independently.

The sFlow standard has long been a popular method of monitoring exchanges for a number of reasons:

sFlow Continue reading

Network and system analytics as a Docker microservice

Microservices describes why the industry standard sFlow instrumentation embedded within cloud infrastructure is uniquely able to provide visibility into microservice deployments.

The sFlow-RT analytics engine is well suited to deployment as a Docker microservice since the application is stateless and presents network and system analytics as a RESTful service.

The following steps demonstrate how to create a containerized deployment of sFlow-RT.

First, create a directory for the project and edit the Dockerfile:

mkdir sflow-rt
cd sflow-rt
vi Dockerfile

Add the following contents to Dockerfile:

FROM   centos:centos6
RUN    yum install -y java-1.7.0-openjdk
RUN    rpm -i http://www.inmon.com/products/sFlow-RT/sflow-rt-2.0-1072.noarch.rpm
EXPOSE 8008 6343/udp
CMD    /etc/init.d/sflow-rt start && tail -f /dev/null

Build the project:

docker build -t sflow-rt .

Run the service:

docker run -p 8008:8008 -p 6343:6343/udp -d sflow-rt

Access the API at http://docker_host:8008/ to verify that the service is running.

Now configure sFlow agents to send data to the docker_host on port 6343:

Switch configurations
Install Host sFlow agents to monitor hosts, hypervisors, Docker, etc.

The following articles provide examples of using the sFlow-RT REST API:

The diagram shows how new and existing cloud based or locally hosted orchestration, operations, and security tools can leverage sFlow-RT's analytics service to gain real-time visibility. The solution is extremely scaleable, a single sFlow-RT instance can monitor thousands of servers and the network devices connecting them.

Microservices

Figure 1: Visibility and the software defined data center

In the land of microservices, the network is the king(maker) by Sudip Chakrabarti, Lightspeed Venture Partners, makes the case that visibility into network traffic is the key to monitoring, managing and securing applications that are composed of large numbers of communicating services running in virtual machines or containers.

While I genuinely believe that the network will play an immensely strategic role in the microservices world, inspecting and storing billions of API calls on a daily basis will require significant computing and storage resources. In addition, deep packet inspection could be challenging at line rates; so, sampling, at the expense of full visibility, might be an alternative. Finally, network traffic analysis must be combined with service-level telemetry data (that we already collect today) in order to get a comprehensive and in-depth picture of the distributed application.

Sampling isn't just an alternative, sampling is the key to making large scale microservice visibility a reality. Shrink ray describes how sampling acts as a scaling function, reducing the task of monitoring large scale microservice infrastructure from an intractable measurement and big data problem to a lightweight real-time data center wide visibility solution for monitoring, managing, Continue reading

Open vSwitch version 2.5 released

The recent Open vSwitch version 2.5 release includes significant network virtualization enhancements:

   - sFlow agent now reports tunnel and MPLS structures.
   ...
   - Add experimental version of OVN.  OVN, the Open Virtual Network, is a
     system to support virtual network abstraction.  OVN complements the
     existing capabilities of OVS to add native support for virtual network
     abstractions, such as virtual L2 and L3 overlays and security groups.

The sFlow Tunnel Structures specification enhances visibility into network virtualization by capturing encapsulation / decapsulation actions performed by tunnel end points. In many network virtualization implementations VXLAN, GRE, Geneve tunnels are terminate in Open vSwitch and so the new feature has broad application.

The second related feature is the inclusion of the Open Virtual Network (OVN), providing a simple method of building virtual networks for OpenStack and Docker.

The following articles provide additional background:

Linux bridge, macvlan, ipvlan, adapters

The open source Host sFlow project added a feature to efficiently monitor traffic on Linux host network interfaces: network adapters, Linux bridge, macvlan, ipvlan, etc. Implementation of high performance sFlow traffic monitoring is made possible by the inclusion of random packet sampling support in the Berkeley Packet Filter (BPF) implementation in recent Linux kernels (3.19 or later).

In addition to the new BPF capability, hsflowd has a couple of other ways to monitor traffic:

iptables, add a statistic rule to the iptables firewall to add traffic monitoring
Open vSwitch, has built-in sFlow instrumentation that can be configured by hsflowd.

The BPF sampling mechanism is less complex to configure than iptables and can be used to monitor any Linux network device, including: network adapters (e.g. eth0) and the Linux bridge (e.g. docker0). Monitoring a network adapter also provides visibility into lightweight macvlan and ipvlan network virtualization technologies that are likely to become more prevalent in the Linux container ecosystem, see Using Docker with macvlan Interfaces.

The following commands build and install hsflowd on an Ubuntu 14.03 host:

sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install libpcap-dev
sudo apt-get install git
git clone https://github. Continue reading

CloudFlare DDoS Mitigation Pipeline

The Usenix Enigma 2016 talk from Marek Majkowski describes CloudFlare's automated DDoS mitigation solution. CloudFlare provides reverse proxy services for millions of web sites and their customers are frequently targets of DDoS attacks. The talk is well worth watching in its entirety to learn about their experiences.

Network switches stream standard sFlow data to CloudFlare's "Gatebot" Reactive Automation component, which analyzes the data to identify attack vectors. Berkeley Packet Filter (BPF) rules are constructed to target specific attacks and apply customer specific mitigation policies. The rules are automatically installed in iptables firewalls on the CloudFlare servers.

The chart shows that over a three month period CloudFlare's mitigation system handled between 30 and 300 attacks per day.

Attack volumes mitigated regularly hit 100 million packers per second and reach peaks of over 150 million packets per second. These large attacks can cause significant damage and automated mitigation is critical to reducing their impact.

Elements of the CloudFlare solution are readily accessible to anyone interested in building DDoS mitigation solutions. Industry standard sFlow instrumentation is widely supported by switch vendors. Download sFlow-RT analytics software and combine real-time DDoS detection with business policies to automate mitigation actions. A number of DDoS mitigation examples are Continue reading

SignalFx

SignalFx is an example of a cloud based analytics service. SignalFx provides a REST API for uploading metrics and a web portal that it simple to combine and trend data and build and share dashboards.

This article describes a proof of concept demonstrating how SignalFx's cloud service can be used to cost effectively monitor large scale cloud infrastructure by leveraging standard sFlow instrumentation. SignalFx offers a free 14 day trial, making it easy to evaluate solutions based on this demonstration.

The diagram shows the measurement pipeline. Standard sFlow measurements from hosts, hypervisors, virtual machines, containers, load balancers, web servers and network switches stream to the sFlow-RT real-time analytics engine. Metrics are pushed from sFlow-RT to SignalFx using the REST API.

Over 40 vendors implement the sFlow standard and compatible products are listed on sFlow.org. The open source Host sFlow agent exports standard sFlow metrics from hosts, virtual machines and containers and local services. For additional background, the Velocity conference talk provides an introduction to sFlow and case study from a large social networking site.

SignalFx's service is priced based on the number of data points that they need to store and they estimate a cost of $15 per host Continue reading

Dell OS10 SDN router demo

In this video from Dell's Network Field Day 11 (#NFD11) presentation, Madhu Santhanam demonstrates an interesting use case for the new OS10 switch operating system that was introduced at the event.

The core of OS10 is an unmodified Linux kernel with an application development environment for Control Plane Services (CPS). These APIs allow software running on the switch: native linux applications, third party applications, and native OS10 applications to run on the core OS10 operating system.

The FIB Optimization application consists of three components: an sFlow agent to provide network visibility, Quagga for BGP routing, and the Selective Route Push agent which provides a REST API for selectively populating the hardware routing tables in the switch ASIC. The FIB Optimization application allows an inexpensive data center switch to replace a much more expensive high capacity Internet router.

In this use case, the data center is connected to a single transit provider and multiple additional peer networks. Initially all traffic is sent via a default route to the transit provider. The full Internet routing table consists of nearly 600,000 prefixes - far too many to fit in the switch hardware forwarding tables which in typical low cost switches can only handle Continue reading

Podcast with Nick Buraglio and Brent Salisbury

"Have you seen sFlow options in your router configuration or flow collector? Are you looking for alternatives to SNMP or NetFlow? Have you been curious about the instrumentation of your new white box or virtual switch? Yes? Then you will probably enjoy learning more about sFlow!"

Non-Blocking #1: SFlow With Peter Phaal Of InMon And SFlow.Org is a discussion between Brent Salisbury (networkstatic.net), Nick Buraglio (forwardingplane.net), and Peter Phaal (blog.sflow.com).

Web sites and tools mentioned in the podcast:

sFlow.org
Devices that support sFlow
Software to analyze sFlow
sFlow.org mailing list
sFlow structures
blog.sflow.com (incorrectly referenced as blog.sflow.org in the podcast)
Host sFlow
sflowtool

The podcast touches on a number of topics that have been explored in greater detail on this blog. The topics are listed in roughly the order they are mentioned in the podcast:

Demystifying NFV Infrastructure Hotspots

Slides from the recent Dell NFV Summit 2015 are now available. Steve Wright's 7 Fallacies of NFV talk describes the importance of managing network resources in an NFV stack. The diagram above shows the complex network data paths that result from NFV as packets flow between virtual functions across physical and virtual switches.

The presentation describes how the Fallacies of Distributes Computing apply to NFV, highlighting the importance of effective management of network resources for effective NFV deployment.

Another paper, Demystifying NFV Infrastructure Hotspots by Ramki Krishnan, Anoop Ghanwani, and Michael Tien, demonstrates how industry standard sFlow instrumentation build into physical and virtual switches can provide the comprehensive real-time analytics needed to manage NFV deployments.

The vIMS (virtualized IP Multimedia Subsystem) is used as an example. The diagram below shows the functional elements of the logical architecture deployed on the hardware testbed shown above.

sFlow telemetry from the physical switches in the leaf and spine network, virtual switch instances, and hypervisors is streamed to an instance of the sFlow-RT analytics platform.

The dashboard application running on sFlow-RT demonstrates visibility into the traffic flows between virtual network functions.

The final set of charts in the dashboard shows the multi-media traffic flows running Continue reading

Environmental metrics with Cumulus Linux

Custom metrics with Cumulus Linux describes how to extend the set of metrics exported by the sFlow agent and used the export of BGP metrics as an example. This article demonstrates how environmental metrics (power supplies, temperatures, fan speeds etc.) can be exported.

The smonctl command can be used to dump sensor data as JSON formatted text:

cumulus@cumulus$ smonctl -j
[
    {
        "pwm_path": "/sys/devices/soc.0/ffe03100.i2c/i2c-1/1-004d", 
        "all_ok": "1", 
        "driver_hwmon": [
            "fan1"
        ], 
        "min": 2500, 
        "cpld_path": "/sys/devices/ffe05000.localbus/ffb00000.CPLD", 
        "state": "OK", 
        "prev_state": "OK", 
        "msg": null, 
        "input": 8998, 
        "type": "fan", 
        "pwm1": 121, 
        "description": "Fan1", 
        "max": 29000, 
        "start_time": 1450228330, 
        "var": 15, 
        "pwm1_enable": 0, 
        "prev_msg": null, 
        "log_time": 1450228330, 
        "present": "1", 
        "target": 0, 
        "name": "Fan1", 
        "fault": "0", 
        "pwm_hwmon": [
            "pwm1"
        ], 
        "driver_path": "/sys/devices/soc.0/ffe03100.i2c/i2c-1/1-004d", 
        "div": "4", 
        "cpld_hwmon": [
            "fan1"
        ]
    },
    ...

The following Python script, smon_sflow.py, invokes the command, parses the output, and posts a set of custom sFlow metrics:

#!/usr/bin/env python
import json
import socket
from subprocess import check_output

res = check_output(["/usr/sbin/smonctl","-j"])
smon = json.loads(res)
fan_maxpc = 0
fan_down = 0
fan_up = 0
psu_down = 0
psu_up = 0
temp_maxpc = 0
temp_up = 0
temp_down = 0
for s in smon:
  type = s["type"]
  if(type ==  Continue reading

Custom events

Measuring Page Load Speed with Navigation Timing describes the standard instrumentation built into web browsers. This article will use navigation timing as an example to demonstrate how custom sFlow events augment standard sFlow instrumentation embedded in network devices, load balancers, hosts and web servers.

The JQuery script can be embedded in a web page to provide timing information:

$(window).load(function(){
 var samplingRate = 10;
 if(samplingRate !== 1 && Math.random() > (1/samplingRate)) return;

 setTimeout(function(){
   if(window.performance) {
     var t = window.performance.timing;
     var msg = {
       sampling_rate : samplingRate,
       t_url         : {type:"string",value:window.location.href},
       t_useragent   : {type:"string",value:navigator.userAgent},
       t_loadtime    : {type:"int32",value:t.loadEventEnd-t.navigationStart},
       t_connecttime : {type:"int32",value:t.responseEnd-t.requestStart} 
     };
     $.ajax({
       url:"/navtiming.php",
       method:"PUT",
       contentType:"application/json",
       data:JSON.stringify(msg) 
     });
    }
  }, 0);
});

The script supports random sampling. In this case a samplingRate of 10 means that, on average, 1-in-10 page hits will generate a measurement record. Measurement records are sent back to the server where the navtiming.php script acts as a gateway, augmenting the measurements and sending them as custom sFlow events.

<?php
$rawInput = file_get_contents("php://input");
$rec = json_decode($rawInput);
$rec->datasource = "navtime";
$rec->t_ip = array("type" => "ip", "value" => $_SERVER['REMOTE_ADDR']);

$msg=array("rtflow"=>$rec);
$sock = fsockopen("udp://localhost",36343,$errno,$errstr);
if(! $sock) {  Continue reading

Custom metrics with Cumulus Linux

Cumulus Networks, sFlow and data center automation describes how Cumulus Linux is monitored using the open source Host sFlow agent that supports Linux, Windows, FreeBSD, Solaris, and AIX operating systems and KVM, Xen, XCP, XenServer, and Hyper-V hypervisors, delivering a standard set of performance metrics from switches, servers, hypervisors, virtual switches, and virtual machines.

Host sFlow version 1.28.3 adds support for Custom Metrics. This article demonstrates how the extensive set of standard sFlow measurements can be augmented using custom metrics.

Recent releases of Cumulus Linux simplify the task by making machine readable JSON a supported output in command line tools. For example, the cl-bgp tool can be used to dump BGP summary statistics:

cumulus@leaf1$ sudo cl-bgp summary show json
{ "router-id": "192.168.0.80", "as": 65080, "table-version": 5, "rib-count": 9, "rib-memory": 1080, "peer-count": 2, "peer-memory": 34240, "peer-group-count": 1, "peer-group-memory": 56, "peers": { "swp1": { "remote-as": 65082, "version": 4, "msgrcvd": 52082, "msgsent": 52084, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" }, "swp2": { "remote-as": 65083, "version": 4, "msgrcvd": 52082, "msgsent": 52083, "table-version": 0, "outq": 0, "inq": 0, "uptime": "05w1d04h", "prefix-received-count": 2, "prefix-advertised-count": 5, "state": "Established", "id-type": "interface" } },  Continue reading

Using a proxy to feed metrics into Ganglia

The GitHub gmond-proxy project demonstrates how a simple proxy can be used to map metrics retrieved through a REST API into Ganglia's gmond TCP protocol.

The diagram shows the elements of the Ganglia monitoring system. The Ganglia server contains runs the gmetad daemon that polls for data from gmond instances and stores time series data. Trend charts are presented through the web interface. The transparent gmond-proxy replaces a native gmond daemon and delivers metrics in response to gmetad's polling requests.

The following commands install the proxy on the sFlow collector - an Ubuntu 14.04 system that is already runnig sFlow-RT:

wget https://raw.githubusercontent.com/sflow-rt/gmond-proxy/master/gmond_proxy.py
sudo mv gmond_proxy.py /etc/init.d/
sudo chown root:root /etc/init.d/gmond_proxy.py
sudo chmod 755 /etc/init.d/gmond_proxy.py
sudo service gmond_proxy.py start
sudo update-rc.d gmond_proxy.py start

The following commands install Ganglia's gmetad collector and web user interface on the Ganglia server - an Ubuntu 14.04 system:

sudo apt-get install gmetad
sudo apt-get install ganglia-webfrontend
cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled

Next edit the /etc/ganglia/gmetad.conf file and configure the proxy as a data source:

data_source "my cluster" sflow-rt

Restart the Apache and gmetad daemons:

sudo service gmetad restart
sudo service apache2  Continue reading

Broadcom BroadView Instrumentation

The diagram above, from the BroadView™ 2.0 Instrumentation Ecosystem presentation, illustrates how instrumentation built into the network Data Plane (the Broadcom Trident/Tomahawk ASICs used in most data center switches) provides visibility to Software Defined Networking (SDN) controllers so that they can optimize network performance.

The sFlow measurement standard provides open, scaleable, multi-vendor, streaming telemetry that supports SDN applications. Broadcom has been augmenting the rich set of counter and flow measurements in the base sFlow standard with additional metrics. For example, Broadcom ASIC table utilization metrics, DevOps, and SDN describes metrics that were added to track ASIC table resource consumption.

The highlighted Buffer congestion state / statistics capability in the slide refers to the BroadView Buffer Statistics Tracking (BST) instrumentation. The Memory Management Unit (MMU) is on-chip logic that manages how the on-chip packet buffers are organized. BST is a feature that enables tracking the usage of these buffers. It includes snapshot views of peak utilization of the on-chip buffer memory across queues, ports, priority group, service pools and the entire chip.

The above chart from the Broadcom technical brief, Building an Open Source Data Center Monitoring Tool Using Broadcom BroadView™ Instrumentation Software, shows buffer utilization trended over an Continue reading

« Previous 1 … 8 9 10 11 12 … 14 Next »