Automatic Remediation of Kubernetes Nodes
We use Kubernetes to run many of the diverse services that help us control Cloudflare’s edge. We have five geographically diverse clusters, with hundreds of nodes in our largest cluster. These clusters are self-managed on bare-metal machines which gives us a good amount of power and flexibility in the software and integrations with Kubernetes. However, it also means we don’t have a cloud provider to rely on for virtualizing or managing the nodes. This distinction becomes even more prominent when considering all the different reasons that nodes degrade. With self-managed bare-metal machines, the list of reasons that cause a node to become unhealthy include:
- Hardware failures
- Kernel-level software failures
- Kubernetes cluster-level software failures
- Degraded network communication
- Software updates are required
- Resource exhaustion1
Unhappy Nodes
We have plenty of examples of failures in the aforementioned categories, but one example has been particularly tedious to deal with. It starts with the following log line from the kernel:
unregister_netdevice: waiting for lo to become free. Usage count = 1
The issue is further observed with the number of network interfaces on the node owned by the Container Network Interface (CNI) plugin getting out of proportion with the number of running pods:
$ Continue readingHedge 91: Leslie Daigle and IP Addresses Acting Badly
What if you could connect a lot of devices to the Internet—without any kind of firewall or other protection—and observe attackers trying to find their way “in?” What might you learn from such an exercise? One thing you might learn is a lot of attacks seem to originate from within a relatively small group of IP addresses—IP addresses acing badly. Listen in as Leslie Daigle of Thinking Cat and the Techsequences podcast, Tom Ammon, and Russ White discuss just such an experiment and its results.
Day Two Cloud 106: Towards A More Open Cloud
On today's Day Two Cloud we discuss the notion of open cloud. The premise is about reducing or minimizing costs of migrating from a public cloud. In theory, open cloud lets organizations keep their options open to make changes and reduces lock-in. But is open cloud even feasible? Our guest is Chris Psaltis, co-founder and CEO of Mist.io, a startup building an open-source, multi-cloud management platform.
The post Day Two Cloud 106: Towards A More Open Cloud appeared first on Packet Pushers.
Day Two Cloud 106: Towards A More Open Cloud
On today's Day Two Cloud we discuss the notion of open cloud. The premise is about reducing or minimizing costs of migrating from a public cloud. In theory, open cloud lets organizations keep their options open to make changes and reduces lock-in. But is open cloud even feasible? Our guest is Chris Psaltis, co-founder and CEO of Mist.io, a startup building an open-source, multi-cloud management platform.5 Ways AIOps Can Simplify NetOps Teams’ Lives
AIOps has great potential to streamline workflows and increase productivity within IT and NetOps teams. This can improve business outcomes and allow for the reallocation of resources to other projects.Free Speech is More than Words
A couple of weeks ago, I joined Leslie Daigle and Alexa Reid on Techsequences to talk about free speech and the physical platform—does the right to free speech include the right to build and operate physical facilities like printing presses and web hosting? I argue it does. Listen in if you want to hear my argument, and how this relates to situations such as the “takedown” of Parler.
Data Center Fabric Monitoring & Visibility: Tools and Options
So far in our blog series on data center network trends, we have focused on options and best practices for...
The post Data Center Fabric Monitoring & Visibility: Tools and Options appeared first on Pluribus Networks.
IT Pro Salary Survey: What You Earned in 2020
Did you get a raise in 2020? The InformationWeek 2020 IT Salary Survey lets you know how you stacked up against your peers during a year that was like no other.NATs, PATs, and Network Hygiene
While reading a research paper on address spoofing from 2019, I ran into this on NAT (really PAT) failures—
The authors state 49% of the NATs they discovered in their investigation of spoofed addresses fail in one of these two ways. From what I remember way back when the first NAT/PAT device (the PIX) was deployed in the real world (I worked in TAC at the time), there was a lot of discussion about what a firewall should do with packets sourced from addresses not indicated anywhere.
If I have an access list including 192.168.1.0/24, and I get a packet sourced from 192.168.2.24, Continue reading
Heavy Networking 588: Exploring The Hidden Realms Of Subsea Cables With Telstra (Sponsored)
Subsea communications cables are an essential part of the global Internet. On today's Heavy Networking, sponsored by Telstra, we dive into the realms of undersea cables to learn how they are laid, signalling methods, POPs and landing stations, how they can be damaged (and repaired), and more. Our Telstra guests are Andy Lumsden, Head of Network Engineering and Operations; and Jeff McHardy, General Manager, Network Development and Commercial Management.
The post Heavy Networking 588: Exploring The Hidden Realms Of Subsea Cables With Telstra (Sponsored) appeared first on Packet Pushers.
Heavy Networking 588: Exploring The Hidden Realms Of Subsea Cables With Telstra (Sponsored)
Subsea communications cables are an essential part of the global Internet. On today's Heavy Networking, sponsored by Telstra, we dive into the realms of undersea cables to learn how they are laid, signalling methods, POPs and landing stations, how they can be damaged (and repaired), and more. Our Telstra guests are Andy Lumsden, Head of Network Engineering and Operations; and Jeff McHardy, General Manager, Network Development and Commercial Management.The Data Driven Enterprise
The rise of cloud migration for enterprises with mission critical applications is redefining the data center. The reality for any enterprise: a systematic approach balancing workloads in the cloud and premises while securing data. Data and applications must be managed as critical assets in the 21st century.
The Data Driven Enterprise
The rise of cloud migration for enterprises with mission critical applications is redefining the data center. The reality for any enterprise: a systematic approach balancing workloads in the cloud and premises while securing data. Data and applications must be managed as critical assets in the 21st century.
Tech Bytes: An IT Punch List For Reopening The Office (Sponsored)
If IT thinks it can take the office network out of mothballs and expect everything to be fine when employees show up, think again. On today's Tech Bytes podcast, sponsored by Netskope, guest Hansang Bae shares a punch list of tasks IT should complete before end users get back to the office.
The post Tech Bytes: An IT Punch List For Reopening The Office (Sponsored) appeared first on Packet Pushers.
Tech Bytes: An IT Punch List For Reopening The Office (Sponsored)
If IT thinks it can take the office network out of mothballs and expect everything to be fine when employees show up, think again. On today's Tech Bytes podcast, sponsored by Netskope, guest Hansang Bae shares a punch list of tasks IT should complete before end users get back to the office.Demo: Cloud Networking with Overlapping CIDR, L7 Firewalls, Segmentation, and Flow visibility
I created a live demo showing some cool capabilities of the Aviatrix Cloud Networking Platform. In this demo I play the role of a SaaS provider that onboards new customers via VPN, and needs to meet the following requirements:
-
Easily onboard new customers even if their IP addressing overlaps with the SaaS provider.
-
Provide secure segmentation and isolation between customers.
-
Easily insert next gen firewalls between the customers and the SaaS for deep packet inspection and threat analysis.
-
Have complete flow level visibility of customer network traffic, and operation tools to diagnose and troubleshoot problems.
-
Provide end-to-end encryption to secure sensitive data in flight.
-
And be able to meet all of these requirements using any cloud provider.
In the demo I show how easy it is to meet requirements like this using Aviatrix. And best of all, no matter which cloud provider(s) you’re using, the solution and architecture is exactly the same. This SaaS provider can use the services and global footprint of any or all cloud providers, and do it with consistent repeatable architecture.
You can leave comments on this post here: where I posted this on LinkedIN.
Is there a particular scenario you want to see in a Continue reading