Why network operations should care about AppDynamics

Earlier this year, Cisco surprised many industry watchers when it forked out a cool $3.7 billion to acquire AppDynamics, which was about 2x the valuation it had going into its IPO. Most people know Cisco as the de facto standard and market leader in networking. AppDynamics lives higher up the stack and provides a view into how applications are performing by collecting data from users, applications, databases and servers.One might surmise that Cisco will use AppDynamics to go after a different buyer, and that assumption is correct. AppDynamics paves the way for Cisco to have a meaningful discussion with lines of business, application developers and company leaders. However, thinking AppDynamics isn’t for Cisco’s current core customers, network engineers, is wrong. AppDynamics can provide an equal amount of value to that audience.To read this article in full or to leave a comment, please click here

Why network operations should care about AppDynamics

Earlier this year, Cisco surprised many industry watchers when it forked out a cool $3.7 billion to acquire AppDynamics, which was about 2x the valuation it had going into its IPO. Most people know Cisco as the de facto standard and market leader in networking. AppDynamics lives higher up the stack and provides a view into how applications are performing by collecting data from users, applications, databases and servers.One might surmise that Cisco will use AppDynamics to go after a different buyer, and that assumption is correct. AppDynamics paves the way for Cisco to have a meaningful discussion with lines of business, application developers and company leaders. However, thinking AppDynamics isn’t for Cisco’s current core customers, network engineers, is wrong. AppDynamics can provide an equal amount of value to that audience.To read this article in full or to leave a comment, please click here

Today, $50 off Sennheiser’s HD1 In-Ear Wireless Headphones – Deal Alert

The HD1 In-Ear Wireless headphones from industry veteran Sennheiser combines immaculate sound quality with take-anywhere mobility. Utilizing a sleek ergonomic design and crafted with high-quality materials, this premium headset exudes luxury. Exceptional wireless hi-fi sound is assured thanks to Bluetooth 4.1 with ACC and Qualcomm apt-X technologies. Featuring NFC for simple one touch pairing, an integrated microphone and ten-hour battery life, the HD1 In-Ear Wireless is a perfect companion to mobile devices. The typical list price of the HD1's is discounted 25% for today only, so you can get them for $149.95. See this deal now on Amazon.To read this article in full or to leave a comment, please click here

Why does one NGINX worker take all the load?

Why does one NGINX worker take all the load?

Scaling up TCP servers is usually straightforward. Most deployments start by using a single process setup. When the need arises more worker processes are added. This is a scalability model for many applications, including HTTP servers like Apache, NGINX or Lighttpd.

Why does one NGINX worker take all the load?CC BY-SA 2.0 image by Paul Townsend

Increasing the number of worker processes is a great way to overcome a single CPU core bottleneck, but opens a whole new set of problems.

There are generally three ways of designing a TCP server with regard to performance:

(a) Single listen socket, single worker process.

(b) Single listen socket, multiple worker processes.

(c) Multiple worker processes, each with separate listen socket.

Why does one NGINX worker take all the load?

(a) Single listen socket, single worker process This is the simplest model, where processing is limited to a single CPU. A single worker process is doing both accept() calls to receive the new connections and processing of the requests themselves. This model is the preferred Lighttpd setup.

Why does one NGINX worker take all the load?

(b) Single listen socket, multiple worker process The new connections sit in a single kernel data structure (the listen socket). Multiple worker processes are doing both the accept() calls and processing of the requests. This model enables some spreading of the inbound Continue reading

A Sample Makefile for Publishing Blog Articles

As some readers may already know, this site has been running on a static site generator since late 2014/early 2015, when I migrated from WordPress to Jekyll on GitHub Pages. I’ve since migrated again, this time to Hugo on S3/CloudFront. Along the way, I’ve taken an interest in using make and a Makefile to help automate certain tasks at the CLI. In this post, I’ll share how I’m using a Makefile to help with publishing blog articles.

If you’re not familiar with make or its use of a Makefile, have a look at this article I wrote on using a Makefile with Markdown documents, then come back here.

In general, the process for publishing a blog post using Hugo and S3/CloudFront basically looks like this:

  1. Write the blog post. (I haven’t found a tool to automate this yet!)
  2. Put the blog post into the right section of the Hugo directory tree. (In my setup, it’s in the content/post directory.)
  3. Build the static site using hugo.
  4. Upload the resulting HTML files to the appropriate S3 bucket.
  5. Invalidate the appropriate URLs (paths) in AWS CloudFront so that the CDN picks up the new files/pages.

Some of these steps Continue reading

Automated Testing & Intent Verification for Network Operations

The most important part of writing quality software is testing. Writing unit tests provide assurance the changes you’re making aren’t going to break anything in your software application. Sounds pretty great, right? Why is it that in networking operations we’re still mainly using ping, traceroute, and human verification for network validation and testing?

The Network is the Application

I’ve written in the past that deploying configurations faster, or more generally, configuration management, is just one small piece of what network automation is. A major component much less talked about is automated testing. Automated testing starts with data collection and quickly evolves to include verification. It’s quite a simple idea and one that we recommend as the best place to start with automation as it’s much more risk adverse to deploying configurations faster.

In our example, the network is the application, and unit tests need to be written to verify our application (as network operators) has valid configurations before each change is implemented, but also integrations tests are needed to ensure our application is operating as expected after each change.

DIY Testing

If you choose to go down the DIY path for network automation, which could involve using an open source Continue reading

Automated Testing & Intent Verification for Network Operations

The most important part of writing quality software is testing. Writing unit tests provide assurance the changes you’re making aren’t going to break anything in your software application. Sounds pretty great, right? Why is it that in networking operations we’re still mainly using ping, traceroute, and human verification for network validation and testing?

The Network is the Application

I’ve written in the past that deploying configurations faster, or more generally, configuration management, is just one small piece of what network automation is. A major component much less talked about is automated testing. Automated testing starts with data collection and quickly evolves to include verification. It’s quite a simple idea and one that we recommend as the best place to start with automation as it’s much more risk adverse to deploying configurations faster.

In our example, the network is the application, and unit tests need to be written to verify our application (as network operators) has valid configurations before each change is implemented, but also integrations tests are needed to ensure our application is operating as expected after each change.

DIY Testing

If you choose to go down the DIY path for network automation, which could involve using an open source Continue reading

How to save IOS configurations with Ansible

Slow save At the outset, a 1200 word article about saving configuration sounds strange. It would perhaps be perfectly normal if the topic was Vi and not Ansible, however there’s a reason for this and its simply speed and itempotency. Saving the configuration in the “wrong” way can take quite a lot of time and one reason for network automation is to accomplish tasks faster and constantly search for ways to improve your processes. This article assumes that you are running Ansible 2.4, but it should work in a similar way regardless. Continue reading

PNG-IX Network Security Workshop

The Internet Society (Aftab Siddiqui) and APNIC (Tashi Phuntsho) jointly conducted a Network Security Workshop in Port Moresby, Papua New Guinea (PNG) on 3-5 October 2017. This was arranged for current and potential members of the first neutral Internet Exchange Point (IX) in the country called PNG-IX, at the request of NICTA – the National Information and Communications Technology Authority – a government agency responsible for the regulation and licensing of Information Communication Technology (ICT) in Papua New Guinea. NICTA is also a key partner in establishing the Internet Exchange in PNG.

This first half of Day 1 (3 October) was dedicated to the PNG-IX awareness., such the role of an IX, how it works, why an IX has been established in PNG and why everyone should peer in order to achieve both short- and long-term benefits to the local Internet ecosystem. NICTA CEO Charles Punaha, NICTA Director Kila Gulo Vui, and APNIC Development Director Che-Hoo Cheng shared their views  

There were more than 40 participants in the Network Security workshop, with diverse backgrounds ranging from enterprise environments, state universities, financial institutions, telcos and ISPS. The training alumni completed lab work and learned about important security topics such as Continue reading

Back In The Future

There are several good nuggets that can be found in this Ted Talk. I particularly like the impetus on local communities and the idea of a Business Plan contest for high school students.

Innovation: Five Steps to Get Your Local Economy Back to the Future


Share your crazy ideas for building local economies and enabling one another by commenting below.

Disclaimer: This article includes the independent thoughts, opinions, commentary or technical detail of Paul Stewart. This may or may does not reflect the position of past, present or future employers.

Readers of this article may also enjoy:

  1. New Podcast for the Podcatcher – The Network Collective
  2. SSH is a BAD API
  3. OSX, Outlook 2011 and Evernote
  4. DNC – What does “dropped the firewall” even mean?
  5. Risky Business #349 <-- Wow, a LOT Happened in 2014

Yubikey for SSH after the Infineon disaster

Because of the Infineon Disaster of 2017 lots of TPM and Yubikey keys have to be regenerated.

I have previously blogged about how to create these keys inside the yubikey, so here’s just the short version of how to redo it by generating the key in software and importing it into the yubikey.

When it appears to stall, that’s when it’s waiting for a touch.

openssl genrsa -out key.pem 2048
openssl rsa -in key.pem -outform PEM -pubout -out public.pem
yubico-piv-tool -s 9a -a import-key  --touch-policy=always -i key.pem
yubico-piv-tool -a verify-pin -a selfsign-certificate -s 9a -S '/CN=my SSH key/' -i public.pem -o cert.pem
yubico-piv-tool -a import-certificate -s 9a -i cert.pem
rm key.pem public.pem cert.pem
ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -e

Delete all mentions of previous key. It’s good to have a disaster plan ahead of time if keys need to be replaced, but if you don’t have one:

  1. Inventory all bad keys. Make sure you have their fingerprints.
  2. Inventory all places this key could be installed.
  3. Generate new keys.
  4. Distribute new keys. (in this case, add to all relevant ~/.ssh/authorized_keys)
  5. Remove all old keys.
  6. Grep for the keys found in Continue reading

Yubikey for SSH after the Infineon disaster

Because of the Infineon Disaster of 2017 lots of TPM and Yubikey keys have to be regenerated.

I have previously blogged about how to create these keys inside the yubikey, so here’s just the short version of how to redo it by generating the key in software and importing it into the yubikey.

When it appears to stall, that’s when it’s waiting for a touch.

openssl genrsa -out key.pem 2048
openssl rsa -in key.pem -outform PEM -pubout -out public.pem
yubico-piv-tool -s 9a -a import-key  --touch-policy=always -i key.pem
yubico-piv-tool -a verify-pin -a selfsign-certificate -s 9a -S '/CN=my SSH key/' -i public.pem -o cert.pem
yubico-piv-tool -a import-certificate -s 9a -i cert.pem
rm key.pem public.pem cert.pem
ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -e

Delete all mentions of previous key. It’s good to have a disaster plan ahead of time if keys need to be replaced, but if you don’t have one:

  1. Inventory all bad keys. Make sure you have their fingerprints.
  2. Inventory all places this key could be installed.
  3. Generate new keys.
  4. Distribute new keys. (in this case, add to all relevant ~/.ssh/authorized_keys)
  5. Remove all old keys.
  6. Grep for the keys found in Continue reading

Today’s BGP leak in Brazil

Earlier today several people noticed network reachability problems for networks such as Twitter, Google and others. The root cause turned out to be another BGP mishap. Between 11:09 and 11:27 UTC traffic for many large CDN was rerouted through Brazil. Below an example for the Internet’s most famous prefix 8.8.8.0/24 (Google DNS)
At 2017-10-21 11:09:59 UTC, AS33362, an US based ISP saw the path towards Google’s 8.8.8.0/24 like this:

33362 6939 16735 263361 15169

This shows the US based network AS33362, would have sent traffic to Google via 6939 (HE) to 16735 (Algar Telecom, Brazil), to 263361 infovale telecom which would have tried to delivered it to Google. The successful delivery of packets would have been unlikely, typically due to congestion which would have been the result of the increase in attracted traffic or an ACL blocking the unexpected traffic.


Below an example from a network connected to the Toronto Internet Exchange, trying to reach 8.8.8.8 at the time Continue reading

NetDevOpEd: an engineer’s response to traditional solutions

I was out at Gartner Catalyst in London in September, speaking to IT professionals about their data center deployments. It was an enjoyable time engaging actively with other like-minded technical individuals that were interested in leveraging the boundaries of their technologies to drive greater business efficiencies and competitiveness.

The common theme across all the attendees I spoke to was the urge for containerization, flexibility of design and rapid deployment. These IT professionals were being tasked with reacting faster, and building more rapidly scalable environment. For their server and application needs, they all had turned to open solutions in Linux, leveraging operating systems such as Red Hat Enterprise Linux, Centos, Ubuntu, and orchestration tools such as Mesos and Docker Swarm to control Docker containers. The common point I saw was that all the compute infrastructure relied on open solutions that allowed for greater simplicity without sacrificing flexibility.

I would then ask these same IT professionals: “what do you use in for network infrastructure in these data centers?”
Universally, the response would come back: “Cisco” or “Arista” or “Juniper.”
I would push them: “Why?”
“Because it’s what we’ve always done.”
“It’s all we know.”
“No one ever Continue reading

1 … 1,886 1,887 1,888 1,889 1,890 … 3,871