Moving Quicksilver into production
One of the great arts of software engineering is making updates and improvements to working systems without taking them offline. For some systems this can be rather easy, spin up a new web server or load balancer, redirect traffic and you’re done. For other systems, such as the core distributed data store which keeps millions of websites online, it’s a bit more of a challenge.
Quicksilver is the data store responsible for storing and distributing the billions of KV pairs used to configure the millions of sites and Internet services which use Cloudflare. In a previous post, we discussed why it was built and what it was replacing. Building it, however, was only a small part of the challenge. We needed to deploy it to production into a network which was designed to be fault tolerant and in which downtime was unacceptable.
We needed a way to deploy our new service seamlessly, and to roll back that deploy should something go wrong. Ultimately many, many, things did go wrong, and every bit of failure tolerance put into the system proved to be worth its weight in gold because none of this was visible to customers.
The Bridge
Our goal Continue reading
Adaptive Applications are the Printing Press for the Digital Era
Adaptive applications are an inescapable evolution driven by the need to scale business that increasingly relies on the ability to keep applications fast, secure, and available.Measuring Internet Resilience in Africa
For many in the African region, Internet interruptions or service degradations occur frequently, which results in a disjointed Internet experience. In order to help improve this experience, we need to track and measure various Internet characteristics through network telemetry. This data can help to identify infrastructure and traffic issues and can provide key information to help decision makers decide where infrastructure investment and policy change might need to be made.
However, as shown by a survey carried out by AFRINIC, the Regional Internet Registry (RIR) for Africa, in 2019, Internet measurement is not a common practice in the African region. This is largely due to a scarcity of deployed measurement tools, platforms and equipment, a lack of awareness in the subject, and the lack of relevant skills to carry out the measurement tasks. The shortage of measurement equipment in African countries makes it very challenging to accurately determine the problem areas that need to be addressed in order to improve Internet reliability and resilience in Africa.
Collaborative Measurements
Assessing the resilience of the Internet is a key component of the Internet Society’s work on Measuring the Internet. To help identify the causes of Internet interruptions and service degradations, we Continue reading
NFA v 20.11 has arrived, featuring IP Prefix Grouping, BGP Community filtering + more.
NFA v 20.11 has just been released. The new version offers IP Prefix Grouping capability, BGP Community Filtering, New Data export option,
The post NFA v 20.11 has arrived, featuring IP Prefix Grouping, BGP Community filtering + more. appeared first on Noction.
Growing Beyond Networking Skills
One of my subscribers trying to figure out how to improve his career choices sent me this question:
I am Sr. Network Engineer with 12+ Years’ experience. I was quit happy with my networking skills but will all the recent changes I’m confused. I am not able to understand what are the key skills I should learn as a network engineer to keep myself demandable.
Before reading the rest of this blog post, please read Cloud and the Three IT Geographies by Massimo Re Ferre.
Growing Beyond Networking Skills
One of my subscribers trying to figure out how to improve his career choices sent me this question:
I am Sr. Network Engineer with 12+ Years’ experience. I was quit happy with my networking skills but will all the recent changes I’m confused. I am not able to understand what are the key skills I should learn as a network engineer to keep myself demandable.
Before reading the rest of this blog post, please read Cloud and the Three IT Geographies by Massimo Re Ferre.
BPF: The future of configs
BPF has some wow-presentations, showing how it enables new performance measuring and tracing. Brendan Gregg has a whole bunch, for example. But I don’t think’s it’s very well explained just why BPF is such a big deal.
Most of the demos are essentially cool and useful looking tools, with an “oh by the way BPF made this happen”. Similar to how it’s common to see announcements about some software, where the very title of the announcement ends with “written in Go”. It gives a vibe of “so what?”.
If you’re interested in system tooling and configuration, and aren’t already aware of BPF, then this is for you.
I’m not an expert on BPF, but this will hopefully help someone else bootstrap faster.
bpftrace
bpftrace is really cool. Clearly it’s inspired by dtrace. But one should not mistake bpftrace for BPF. bpftrace is only yet another tool that uses BPF, albeit one that allows you to create trace points in a domain specific language.
This is not the full power of BPF. It’s not at all the big picture.
BPF and configs
Let’s take packet filtering as an example. Once upon a time in Linux
there was ipfwadm. I Continue reading
Pyenv Install Centos 8
Install and use pyenv on Centos 8.Data Mobility and Freedom Maximizes Its Business Value
Companies that fail to manage their unstructured file and object data often end up with unstructured data silos, leaving valuable intelligence on the table.IETF 109
The latest IETF meeting was held in mid-November. Here I’m going to pick just one presentation from each of a small collection of the week’s working group meetings and explore that topic in a little more detail.The History of EARN, RARE, and European Networks (part 1)
European networks from the mid-1980’s to the late 2000’s underwent a lot of change, bolstered by the rise and fall of America Online, the laying of a lot of subsea cables, and the creation of several organizations, including EARN and RARE, to bolster the spread and use of the Internet. Daniele Bovio joins Donald Sharp and Russ White on this episode of the History of Networking to give us a good overall perspective of this history.
You can find more information about the history of EARN at https://earn-history.net.
New LibSSH Connection Plugin for Ansible Network Replaces Paramiko, Adds FIPS Mode Enablement
As Red Hat Ansible Automation Platform expands its footprint with a growing customer base, security continues to be an important aspect of organizations’ overall strategy. Red Hat regularly reviews and enhances the foundational codebase to follow better security practices. As part of this effort, we are introducing FIPS 140-2 readiness enablement by means of a newly developed Ansible SSH connection plugin that uses the libssh library.
Ansible Network SSH Connection Basics
Since most network appliances don't support or have limited capability for the local execution of a third party software, the Ansible network modules are not copied to the remote host unlike linux hosts; instead, they run on the control node itself. Hence, Ansible network can’t use the typical Ansible SSH connection plugin that is used with linux host. Furthermore, due to this behavior, performance of the underlying SSH subsystem is critical. Not only is the new LibSSH connection plugin enabling FIPS readiness, but it was also designed to be more performant than the existing Paramiko SSH subsystem.
The top level network_cli connection plugin, provided by the ansible.netcommon Collection (specifically ansible.netcommon.network_cli), provides an SSH based connection to the network appliance. It in turn calls the Continue reading
Leading from the Front: How the Internet Society’s Training Efforts Are Helping to Upskill Africa’s Future Digital Champions
Tech skills are important for digital transformation in Africa. To realize this transformation, the Internet Society is supporting work-ready digital skills development with local talent.
By 2030, over 230 million jobs in Sub-Saharan Africa will require digital skills. With the fourth industrial revolution taking shape, we are helping to build a community of digital champions in Africa who will fill the skills gap on the continent.
African governments are already banking big on the contributions that digital technologies will provide to this transformation. For example, the African Union’s Digital Transformation Strategy (2020-2030) wants to create a “Digital Single Market” for the continent, while national governments are increasingly embracing digitalization in several sectors of their economies.
But the continent needs the right talent for its ambitious economic agenda to succeed. Though more learners are taking interest in STEM (science, technology, engineering, and mathematics) subjects, school curricula in Africa primarily focus on theoretical learning with scarce engagement with digital subjects. This is one of the main reasons a skills gap exists for jobs that require digital skills.
To play a role in positively changing these circumstances, the Internet Society developed the Introduction to Network Operations course, which equips novice and intermediate level Continue reading
Fast Failover: Hardware and Software Implementations
In previous blog posts in this series we discussed whether it makes sense to invest into fast failover network designs, the topologies you can use in such designs, and the fault detection techniques. I also hinted at different fast failover implementations; this blog post focuses on some of them.
Hardware-based failover changes the hardware forwarding tables after a hardware-detectable link failure, most likely loss-of-light or transceiver-reported link fault. Forwarding hardware cannot do extensive calculations; the alternate paths are thus usually pre-programmed (more details below).
Fast Failover: Hardware and Software Implementations
In previous blog posts in this series we discussed whether it makes sense to invest into fast failover network designs, the topologies you can use in such designs, and the fault detection techniques. I also hinted at different fast failover implementations; this blog post focuses on some of them.
Hardware-based failover changes the hardware forwarding tables after a hardware-detectable link failure, most likely loss-of-light or transceiver-reported link fault. Forwarding hardware cannot do extensive calculations; the alternate paths are thus usually pre-programmed (more details below).
Pyenv Install Centos 8
cuction Pyenv is a similar tool to rbenv, where you can easily install and use mulitple versions of python. This allows you to test across multiple versions of python as well as pin an app to a specific release. Software Versions Used Centos - 8.2.2004 Pyenv - 1.2.21 Pre-requisites ...Advanced Threat Intelligence Begins with Network Visibility
The current reality has pushed users, applications, and data to the edge of the network —where traditional perimeter security solutions have historically fallen short. Threat actors know this, of course, and have spent the past nine months targeting the weakest link in the security stack: the user.
Email and web browsing continue to be popular attack vectors. Security vendors have beefed up web and email security, but issues with legacy architectures are letting some attacks slip through. Information and context derived from advanced threat intelligence remain the most powerful weapons in a security team’s arsenal. Advanced technologies such as artificial intelligence and machine learning can help scan, detect, and warn at scale, but they’re not bulletproof. Increasingly sophisticated threat actors, powered by AI and ML, are finding ways to evade threat detection.
Security professionals interested in learning more about the current state of advanced threat inspection, threat intelligence, and the emerging technologies that power these capabilities should check out the following sessions:
The Promise and Peril of AI for Cybersecurity (ISNS2794)
Artificial intelligence and machine learning are powerful, indeed essential, components of security Continue reading