Why does one NGINX worker take all the load?
Scaling up TCP servers is usually straightforward. Most deployments start by using a single process setup. When the need arises more worker processes are added. This is a scalability model for many applications, including HTTP servers like Apache, NGINX or Lighttpd.
CC BY-SA 2.0 image by Paul Townsend
Increasing the number of worker processes is a great way to overcome a single CPU core bottleneck, but opens a whole new set of problems.
There are generally three ways of designing a TCP server with regard to performance:
(a) Single listen socket, single worker process.
(b) Single listen socket, multiple worker processes.
(c) Multiple worker processes, each with separate listen socket.
(a) Single listen socket, single worker process This is the simplest model, where processing is limited to a single CPU. A single worker process is doing both accept() calls to receive the new connections and processing of the requests themselves. This model is the preferred Lighttpd setup.
(b) Single listen socket, multiple worker process The new connections sit in a single kernel data structure (the listen socket). Multiple worker processes are doing both the accept() calls and processing of the requests. This model enables some spreading of the inbound Continue reading
Troubleshooting: Why Ping When You Can CryPing?
In this video, learn about a free alternative tool to ping for network analysis.
Troubleshooting: Why Ping When You Can CryPing?
In this video, learn about a free alternative tool to ping for network analysis.
To BFD or not to BFD?
Omer asked a pretty common question about BFD on one of my blog posts (slightly reworded):
Would you still use BFD even if you have a direct router-to-router physical link without L2 transport in the middle to detect if there is some kind of software failure on the other side?
Sander Steffann quickly replied:
Read more ...How to save IOS configurations with Ansible
At the outset, a 1200 word article about saving configuration sounds strange. It would perhaps be perfectly normal if the topic was Vi and not Ansible, however there’s a reason for this and its simply speed and itempotency. Saving the configuration in the “wrong” way can take quite a lot of time and one reason for network automation is to accomplish tasks faster and constantly search for ways to improve your processes. This article assumes that you are running Ansible 2.4, but it should work in a similar way regardless.
Continue reading
Automated Testing & Intent Verification for Network Operations
The most important part of writing quality software is testing. Writing unit tests provide assurance the changes you’re making aren’t going to break anything in your software application. Sounds pretty great, right? Why is it that in networking operations we’re still mainly using ping, traceroute, and human verification for network validation and testing?
The Network is the Application
I’ve written in the past that deploying configurations faster, or more generally, configuration management, is just one small piece of what network automation is. A major component much less talked about is automated testing. Automated testing starts with data collection and quickly evolves to include verification. It’s quite a simple idea and one that we recommend as the best place to start with automation as it’s much more risk adverse to deploying configurations faster.
In our example, the network is the application, and unit tests need to be written to verify our application (as network operators) has valid configurations before each change is implemented, but also integrations tests are needed to ensure our application is operating as expected after each change.
DIY Testing
If you choose to go down the DIY path for network automation, which could involve using an open source Continue reading
Automated Testing & Intent Verification for Network Operations
The most important part of writing quality software is testing. Writing unit tests provide assurance the changes you’re making aren’t going to break anything in your software application. Sounds pretty great, right? Why is it that in networking operations we’re still mainly using ping, traceroute, and human verification for network validation and testing?
The Network is the Application
I’ve written in the past that deploying configurations faster, or more generally, configuration management, is just one small piece of what network automation is. A major component much less talked about is automated testing. Automated testing starts with data collection and quickly evolves to include verification. It’s quite a simple idea and one that we recommend as the best place to start with automation as it’s much more risk adverse to deploying configurations faster.
In our example, the network is the application, and unit tests need to be written to verify our application (as network operators) has valid configurations before each change is implemented, but also integrations tests are needed to ensure our application is operating as expected after each change.
DIY Testing
If you choose to go down the DIY path for network automation, which could involve using an open source Continue reading
PNG-IX Network Security Workshop
The Internet Society (Aftab Siddiqui) and APNIC (Tashi Phuntsho) jointly conducted a Network Security Workshop in Port Moresby, Papua New Guinea (PNG) on 3-5 October 2017. This was arranged for current and potential members of the first neutral Internet Exchange Point (IX) in the country called PNG-IX, at the request of NICTA – the National Information and Communications Technology Authority – a government agency responsible for the regulation and licensing of Information Communication Technology (ICT) in Papua New Guinea. NICTA is also a key partner in establishing the Internet Exchange in PNG.
This first half of Day 1 (3 October) was dedicated to the PNG-IX awareness., such the role of an IX, how it works, why an IX has been established in PNG and why everyone should peer in order to achieve both short- and long-term benefits to the local Internet ecosystem. NICTA CEO Charles Punaha, NICTA Director Kila Gulo Vui, and APNIC Development Director Che-Hoo Cheng shared their views
There were more than 40 participants in the Network Security workshop, with diverse backgrounds ranging from enterprise environments, state universities, financial institutions, telcos and ISPS. The training alumni completed lab work and learned about important security topics such as Continue reading
Back In The Future
There are several good nuggets that can be found in this Ted Talk. I particularly like the impetus on local communities and the idea of a Business Plan contest for high school students.
Innovation: Five Steps to Get Your Local Economy Back to the Future
Share your crazy ideas for building local economies and enabling one another by commenting below.
Disclaimer: This article includes the independent thoughts, opinions, commentary or technical detail of Paul Stewart. This may or may does not reflect the position of past, present or future employers.
Readers of this article may also enjoy:
Yubikey for SSH after the Infineon disaster
Because of the Infineon Disaster of 2017 lots of TPM and Yubikey keys have to be regenerated.
I have previously blogged about how to create these keys inside the yubikey, so here’s just the short version of how to redo it by generating the key in software and importing it into the yubikey.
When it appears to stall, that’s when it’s waiting for a touch.
openssl genrsa -out key.pem 2048
openssl rsa -in key.pem -outform PEM -pubout -out public.pem
yubico-piv-tool -s 9a -a import-key --touch-policy=always -i key.pem
yubico-piv-tool -a verify-pin -a selfsign-certificate -s 9a -S '/CN=my SSH key/' -i public.pem -o cert.pem
yubico-piv-tool -a import-certificate -s 9a -i cert.pem
rm key.pem public.pem cert.pem
ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -e
Delete all mentions of previous key. It’s good to have a disaster plan ahead of time if keys need to be replaced, but if you don’t have one:
- Inventory all bad keys. Make sure you have their fingerprints.
- Inventory all places this key could be installed.
- Generate new keys.
- Distribute new keys. (in this case, add to all relevant
~/.ssh/authorized_keys) - Remove all old keys.
- Grep for the keys found in Continue reading
Yubikey for SSH after the Infineon disaster
Because of the Infineon Disaster of 2017 lots of TPM and Yubikey keys have to be regenerated.
I have previously blogged about how to create these keys inside the yubikey, so here’s just the short version of how to redo it by generating the key in software and importing it into the yubikey.
When it appears to stall, that’s when it’s waiting for a touch.
openssl genrsa -out key.pem 2048
openssl rsa -in key.pem -outform PEM -pubout -out public.pem
yubico-piv-tool -s 9a -a import-key --touch-policy=always -i key.pem
yubico-piv-tool -a verify-pin -a selfsign-certificate -s 9a -S '/CN=my SSH key/' -i public.pem -o cert.pem
yubico-piv-tool -a import-certificate -s 9a -i cert.pem
rm key.pem public.pem cert.pem
ssh-keygen -D /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -e
Delete all mentions of previous key. It’s good to have a disaster plan ahead of time if keys need to be replaced, but if you don’t have one:
- Inventory all bad keys. Make sure you have their fingerprints.
- Inventory all places this key could be installed.
- Generate new keys.
- Distribute new keys. (in this case, add to all relevant
~/.ssh/authorized_keys) - Remove all old keys.
- Grep for the keys found in Continue reading
Using The Libvirt Provider With Vagrant
Install libvirt and the vagrant-libvirt provider for use with Vagrant.Using The Libvirt Provider With Vagrant
Libvirt is a set of software tools to manage virtual machine components such as network and storage. Libvirt provdies an abstraction layer hiding the differences across hypervisors such as KVM, Xen, VMWare ESXi, etc.. to avoid having to worry about the intricacies of each hypervisors...Today’s BGP leak in Brazil
Earlier today several people noticed network reachability problems for networks such as Twitter, Google and others. The root cause turned out to be another BGP mishap.Between 11:09 and 11:27 UTC traffic for many large CDN was rerouted through Brazil. Below an example for the Internet’s most famous prefix 8.8.8.0/24 (Google DNS)Some Google services seem to have been hijacked for roughly 15 minutes. Seen anything? @atoonk @bgpmon @bgpstream
— Fusl Neko Shy Dash (@OhNoItsFusl) October 21, 2017
MTR: https://t.co/RyCoE7zMld pic.twitter.com/DCT2JpKgsc
At 2017-10-21 11:09:59 UTC, AS33362, an US based ISP saw the path towards Google’s 8.8.8.0/24 like this:
33362 6939 16735 263361 15169
This shows the US based network AS33362, would have sent traffic to Google via 6939 (HE) to 16735 (Algar Telecom, Brazil), to 263361 infovale telecom which would have tried to delivered it to Google. The successful delivery of packets would have been unlikely, typically due to congestion which would have been the result of the increase in attracted traffic or an ACL blocking the unexpected traffic.
Below an example from a network connected to the Toronto Internet Exchange, trying to reach 8.8.8.8 at the time Continue reading
NetDevOpEd: an engineer’s response to traditional solutions
I was out at Gartner Catalyst in London in September, speaking to IT professionals about their data center deployments. It was an enjoyable time engaging actively with other like-minded technical individuals that were interested in leveraging the boundaries of their technologies to drive greater business efficiencies and competitiveness.
The common theme across all the attendees I spoke to was the urge for containerization, flexibility of design and rapid deployment. These IT professionals were being tasked with reacting faster, and building more rapidly scalable environment. For their server and application needs, they all had turned to open solutions in Linux, leveraging operating systems such as Red Hat Enterprise Linux, Centos, Ubuntu, and orchestration tools such as Mesos and Docker Swarm to control Docker containers. The common point I saw was that all the compute infrastructure relied on open solutions that allowed for greater simplicity without sacrificing flexibility.
I would then ask these same IT professionals: “what do you use in for network infrastructure in these data centers?”
Universally, the response would come back: “Cisco” or “Arista” or “Juniper.”
I would push them: “Why?”
“Because it’s what we’ve always done.”
“It’s all we know.”
“No one ever Continue reading