SYN packet handling in the wild
Here at Cloudflare, we have a lot of experience of operating servers on the wild Internet. But we are always improving our mastery of this black art. On this very blog we have touched on multiple dark corners of the Internet protocols: like understanding FIN-WAIT-2 or receive buffer tuning.
CC BY 2.0 image by Isaí Moreno
One subject hasn't had enough attention though - SYN floods. We use Linux and it turns out that SYN packet handling in Linux is truly complex. In this post we'll shine some light on this subject.
The tale of two queues
First we must understand that each bound socket, in the "LISTENING" TCP state has two separate queues:
- The SYN Queue
- The Accept Queue
In the literature these queues are often given other names such as "reqsk_queue", "ACK backlog", "listen backlog" or even "TCP backlog", but I'll stick to the names above to avoid confusion.
SYN Queue
The SYN Queue stores inbound SYN packets[1] (specifically: struct inet_request_sock). It's responsible for sending out SYN+ACK packets and retrying them on timeout. On Linux the number of retries is configured with:
$ sysctl net.ipv4.tcp_synack_retries
net.ipv4.tcp_synack_retries = 5
Meltdown and Its Networking Equivalents
One of my readers sent me this question:
Do you have any thoughts on this meltdown HPTI thing? How does a hardware issue/feature become a software vulnerability? Hasn't there always been an appropriate level of separation between kernel and user space?
There’s always been privilege-level separation between kernel and user space, but not the address space separation - kernel has been permanently mapped into the high-end addresses of user space (but not visible from the user-space code on systems that had decent virtual memory management hardware) since the days of OS/360, CP/M and VAX/VMS (RSX-11M was an exception since it ran on 16-bit CPU architecture and its designers wanted to support programs up to 64K byte in size).
Read more ...Salt From The Start To The Beginning
Salt quick start guide.Salt From The Start To The Beginning
As described by their website; Salt is "Event-driven automation for a software-defined world". You gotta love marketing :) Salt is a large project with many features including; configuration management, an event based reactor, cloud management and network automation. Salt can do ALOT but...NSX-T: OpenAPI and SDKs
Nowadays everything is about automation. Organizations are moving away from the traditional static infrastructure to full automation and here the need of NSX is significant. There are many use-cases for NSX, but the common in all of them is that they all need to be automated.
VMware is investing heavenly for different tools to ease the automation aspect of NSX but in order to take full advantage of it one need to understand what happens under the hood. It is also important if someone wants to build their own custom automation tool or CMP (Cloud Management Platform). Many existing solutions like Openstack, Kubernetes, vRO and so on automate NSX-T using different plugins. In fact, those plugins are sending REST API calls to NSX Manager in order to automate logical topology CRUD(Create, Read, Update, Delete) operations.
Based on our experience we decided that NSX-T APIs will be based on JSON format following OpenAPI standard. The use of Open APIs is to enable third party developers to build applications and services around NSX-T by standardising on how REST APIs are described. This means one can use standard tools like Swagger to read and use those APIs. Below is a quick example from my Mac on Continue reading
Worth Reading: Robust IPAM
Elisa Jasinska covered several IPAMs in her overview of open-source network automation tools, and we had Jeremy Stretch talking about NetBox in the Building Network Automation Solutions online course, but if you’re looking for a really robust easy-to-implement solution, check out this document from 1998 (deployment experience, including a large-scale one).
Deleting Checkpoint Logs
The directories that need to be emptied to delete all the logs on the Checkpoint managers.
Checkpoint Identity Awareness
The 3 main elements that run identity awareness under the hub are Active Directory Query (ADQ), PDP and PEP. They all intertwine in some way to allow the different blades of the Checkpoint to track and restrict access based on AD user and machine name. I tested these features as part of a POC and personally I would not consider them fit for purpose in a production environment. See the caveats at the end of the post for more details on this.
ParkMyCloud Automates Multi-Cloud Cost Control
The platform works with AWS, Azure, and Google, and plans to support Alibaba Cloud later this year.
NSX-T: Routing where you need it (Part 2, North-South Routing)
In the first part of this blog series, NSX-T: Routing where you need it (Part 1), I discussed how East-West (E-W) routing is completely distributed on NSX-T and how routing is done by the Distributed Router (DR) running as a kernel module in each hypervisor.
In this post, I will explain how North-South (N-S) routing is done in NSX-T and we will also look at the ECMP topologies. This N-S routing is provided by the centralized component of logical router, also known as Service Router. Before we get into the N-S routing or packet walk, let’s define Service Router.
Service Router (SR)
Whenever a service which cannot be distributed is enabled on a Logical Router, a Service Router (SR) is instantiated. There are some services today on NSX-T which are not distributed such as:
1) Connectivity to physical infrastructure
2) NAT
3) DHCP server
4) MetaData Proxy
5) Edge Firewall
6) Load Balancer
Let’s take a look at one of these services (connectivity to physical devices) and see why a centralized routing component makes sense for running this service. Connectivity to physical topology is intended to exchange routing information from NSX domain to external networks (DC, Campus or Continue reading
Domain Name System Security Extensions (DNSSEC) Cheat Sheet Released
Here is the Cheat Sheet for DNSSEC. Recently vulnerabilities in the DNS were discovered that allow an attacker to hijack this process of looking some one up or looking a site up on the Internet using their name. The purpose of the attack is to take control of the session to, for example, send the user to the hijacker's own deceptive web site for account and password collection.
These vulnerabilities have increased interest in introducing a technology called DNS Security Extensions (DNSSEC) to secure this part of the Internet's infrastructure.
If you found a bug or want new content to be added, email me!
New developments in the net neutrality debate
Democrats in the USA haven’t given up on net neutrality since the FCC voted to repeal the 2015 Open Internet Order in December.
Senator Markey, a Democrat from Massachusetts, has put forward a bill that would use the Congressional Review Act (CRA) to reverse the Federal Communications Commission’s (FCC) decision. The CRA allows lawmakers 60 legislative days after the FCC submits its regulations to Congress to take action.
That bill now has the support of more than 40 Senators, including Senator Susan Collins (R-Maine), its first Republican supporter.
Senator Markey needs 52 votes to get the bill passed in the Senate, which is unlikely given that Republicans remain in control of both chambers of Congress. Nevertheless, Democrats see value in forcing a vote on the bill ahead of the 2018 midterm elections. Polls indicate that 83 per cent of Americans support keeping the FCC’s net neutrality rules.
If the bill fails to move forward, Congress may pursue a legislative solution – a bill that would codify net neutrality rules into law, rather than a reversal of the FCC’s decision. In fact, the Republicans are possibly signaling an appetite to come to a legislative solution to this issue. Regardless of Continue reading