The story of one latency spike

A customer reported an unusual problem with our CloudFlare CDN: our servers were responding to some HTTP requests slowly. Extremely slowly. 30 seconds slowly. This happened very rarely and wasn't easily reproducible. To make things worse all our usual monitoring hadn't caught the problem. At the application layer everything was fine: our NGINX servers were not reporting any long running requests.

Time to send in The Wolf.

He solves problems.

Following the evidence

First, we attempted to reproduce what the customer reported—long HTTP responses. Here is a chart of of test HTTP requests time measured against our CDN:

We ran thousands of HTTP queries against one server over a couple of hours. Almost all the requests finished in milliseconds, but, as you can clearly, see 5 requests out of thousands took as long as 1000ms to finish. When debugging network problems the delays of 1s, 30s are very characteristic. They may indicate packet loss since the SYN packets are usually retransmitted at times 1s, 3s, 7s, 15, 31s.

Blame the network

At first we thought the spikes in HTTP load times might indicate some sort of network problem. To be sure we ran ICMP pings against two IPs over many Continue reading

Home IoT security could come from a glowing rock next year

An Internet of Things security startup thinks it can reduce the complexity of a home full of connected devices to three colors: red, orange, and green. Those colors will glow from a wireless orb that looks like a smooth river rock and is small enough to fit in your hand. But it's what is behind this friendly bit of decor that will make the colors meaningful. The San Francisco startup, Dojo-Labs, makes a network security device that plugs into your home Internet gateway and talks to a cloud-based service. It's all managed through a smartphone app.  Dojo aims for nothing less than protecting a consumer's entire collection of home IoT gear against cyber attacks. It plans to do that by monitoring all devices around the clock for odd behaviors and then either alerting the user and fixing the problem (orange light) or telling the user there's something they need to do (red light). When everything's fine, it will show a green light. The orb is wireless, battery-powered and controlled by Dojo's client device via Bluetooth.To read this article in full or to leave a comment, please click here

Dell-EMC deal will ‘set back innovation,’ says Sun co-founder Vinod Khosla

While he thinks Dell buying EMC makes sense financially, Sun Microsystems co-founder Vinod Khosla has little faith in the merged company's ability to innovate."EMC and Dell merging is a really good financial move for Michael, but it will set back innovation and distract from innovation," said Khosla, now a prominent venture capitalist, in an onstage interview at the Structure conference in San Francisco on Wednesday.MORE: Hottest Black Friday Windows dealsAsked about the future for old-line technology companies like Cisco, IBM and Dell, Khosla was pessimistic. In his view, only about half of those tech titans will stick around in the future. What's more, he said, innovation from those companies has been seriously lacking.To read this article in full or to leave a comment, please click here

DMVPN Split Default Routing and Internet Access

One of the engineers listening to my DMVPN webinars sent me a follow-up question (yes, I always try to reply to them) asking how to implement direct Internet access from the spoke sites (aka local exit) in combination with split default routing you have to use in DMVPN Phase 2 or Phase 3 networks.

It’s really simple: either you have a design requirement that requires split default routing, or you don’t.

Read more ...

What did you Expect? Part 5, Basic Error Handling.

In the first four parts of What did you Expect, we covered the basics of getting started with automating interactions for network equipment.  In the first few posts it was important have a networking environment that  was 100% stable.  The last thing I needed when I was trying to learn to use python to automate network …

Hacking group that hit South Korea may be at it again with new target

A hacking group that crippled South Korean banks, government websites and news agencies in early 2013 may be active again, Palo Alto Networks said Wednesday.The firewall maker said it found strong similarities between malware used in a recent attack in Europe and that used in the South Korean attacks, referred to as Dark Seoul and Operation Troy.The organization in Europe that was attacked was likely a victim of spear-phishing, where an email with a malware attachment or a harmful link is sent to hand-picked employees.The malware had been wrapped into legitimate video player software that was hosted by an industrial control systems company, wrote Bryan Lee and Josh Grunzweig of Palo Alto in a blog post. The code appears to be the same as the malware used in the Dark Seoul attacks although without the destructive component that wipes hard drives.To read this article in full or to leave a comment, please click here

Damballa finds tools related to the malware that hit Sony

Security company Damaballa said it has found two utilities that are closely related to capabilities seen in the destructive malware that hit Sony Pictures Entertainment last year. The utilities were discovered as Damballa was investigating a new version of the "Destover" malware, which rendered thousands of computers unusable at Sony after attackers stole gigabytes of sensitive company information. One key question in the Sony breach is how the attackers were able to evade security systems. What Damaballa found are two utilities that help mask new files introduced to a system.  "Both utilities would be used during an attack to evade detection while moving laterally through a network to broaden the attack surface," wrote senior threat researchers Willis McDonald and Loucif Kharouni, in a blog post on Wednesday.To read this article in full or to leave a comment, please click here

CCIE Data Center v2.0 Blueprint Announced

Cisco has just announced CCIE Data Center Written and Lab Exam Content Updates.Important dates for the changes are:

  • Last day to test for the v1.0 written – July 22, 2016
  • First day to test for the v2.0 written – July 25, 2016
  • Last day to test for the v1.0 lab – July 22, 2016
  • First day to test for the v2.0 lab – July 25, 2016

Key hardware changes in the v2.0 blueprint are:

  • APIC Cluster
  • Nexus 9300
  • Nexus 7000 w/ F3 Module
  • Nexus 5600
  • Nexus 2300 Fabric Extender
  • UCS 4300 M-Series Servers

Key technical topic changes in the v2.0 blueprint are:

  • VXLAN
  • EVPN
  • LISP
  • Policy Driven Fabric (ACI)

More details to come!

Network virtualization visibility demo

New OVS instrumentation features aimed at real-time monitoring of virtual networks, Open vSwitch 2015 Fall Conference, included a demonstration of real-time visibility into the logical network overlays created by network virtualization, virtual switches, and the leaf and spine underlay carrying the tunneled traffic between hosts.

The diagram above shows the demonstration testbed. It consists of a leaf and spine network connecting two hosts, each of which is running a pair of Docker containers connected to Open vSwitch (OVS). The vSwitches are controlled by Open Virtual Network (OVN), which has been configured to create two logical switches, the first connecting the left most containers on each host and the second connecting the right most containers. The testbed is described in more detail in Open Virtual Network (OVN) and is built from free components and can easily be replicated.


The dashboard in the video illustrates the end to end visibility that is possible by combining standard sFlow instrumentation in the physical switches with sFlow instrumentation in Open vSwitch and Host sFlow agents on the servers.

The diagram on the left of the dashboard shows a logical map of the elements in the testbed. The top panel shows the two logical switches Continue reading

IDG Contributor Network: This company’s name isn’t a joke: Secret Double Octopus goes beyond encryption

The process of naming a startup is fraught with peril - founders need to find a name which is catchy, ideally short, and one for which the URL is still available. Seemingly throwing most of the rules (at least about brevity or sense) out the window, Secret Double Octopus, a new company just emerging from stealth, has at least ensured one thing - no one will forget its name.Beyond quirky names, however, this company is doing something interesting. Yet another cybersecurity company that originated in Israel, Secret Double Octopus (we'll call it SDO to avoid the risks of overuse injury from repeatedly typing the name) is all about securing networking traffic and authentication beyond the traditional approaches of PKI, SSL and VPN. SDO aims to help secure data in transit, whether it's between sites, between a website and the cloud, or within mobile or IoT use cases. SDO's approach employs secret sharing, thereby eliminating the need for cryptographic keys.To read this article in full or to leave a comment, please click here

TOWER 2.4 NOW AVAILABLE

We’re happy to announce the release of Ansible Tower 2.4. In this release, we’ve focused on some core improvements for our customers operating in spaces like government and security who have specific needs around authentication and tracking, but we expect these features will be useful to much of our general user base as well.

OAUTH, VIA GITHUB AND GOOGLE

No one wants to manage their users in multiple places, and many groups today use external providers for handling their identity and authentication. We’ve added support for pulling users and teams from either GitHub or Google Apps, using OAuth2. With this, you don’t need to add users directly to Tower - they can use the accounts they already have and are using in your organization.

ADDITIONAL ENTERPRISE AUTHENTICATION

Previously, for Enterprise users who have a standard corporate infrastructure Tower has included support for connecting to an LDAP or Active Directory server for user and team information. But not everyone exposes their LDAP for use with all internal services. With Tower 2.4, we’ve extended that enterprise authentication support to also include support for authenticating to a SAML 2.0 identity provider, and to authenticate against a RADIUS server. With this, Continue reading

1 … 3,251 3,252 3,253 3,254 3,255 … 3,852