Cloudflare Incident on January 24th, 2023
Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products including aspects of our Workers platform, our Zero Trust solution, and control plane functions in our content delivery network (CDN).
Cloudflare provides a service token functionality to allow automated services to authenticate to other services. Customers can use service tokens to secure the interaction between an application running in a data center and a resource in a public cloud provider, for example. As part of the release, we intended to introduce a feature that showed administrators the time that a token was last used, giving users the ability to safely clean up unused tokens. The change inadvertently overwrote other metadata about the service tokens and rendered the tokens of impacted accounts invalid for the duration of the incident.
The reason a single release caused so much damage is because Cloudflare runs on Cloudflare. Service tokens impact the ability for accounts to authenticate, and two of the impacted accounts power multiple Cloudflare services. When these accounts’ service tokens were overwritten, the services that run on these accounts began to experience Continue reading
The Root Zone of the DNS Revisited
The DNS is a remarkably simple system. You send it queries and you get back answers. However, the DNS is simple in the same way that Chess or Go are simple. They are all constrained environments governed by a small set of rigid rules, but they all possess astonishing complexity.VisionFive 2 quickstart
RISC-V small computer
For a long time I’ve wanted something Raspberry-pi-like but with RISC-V. And finally there is one, and a defensible price! Especially with the Raspberry Pi 4 shortage this seemed like a good idea.
This post is my first impressions and setup steps.
It’s just like when I was a kid!
When I was in my late teens I was playing with different architectures, mostly using discarded university computers. It was fun to have such different types of computers. Back then it was SPARC (And UltraSparc), Alpha, and x86. Maybe access to some HPPA. I even had a MIPS (SGI Indigo 2).
Nowadays instead of SPARC, Alpha, and x86 it’s ARM, RISC-V, and x64.
Luckily they can be smaller nowadays. Before I left home my room had more towers of computers than it had furniture. In my first flat I had a full size rack!
Write SD card
pv starfive-jh7110-VF2_515_v2.5.0-69-minimal-desktop.img \
| sudo dd of=/dev/sda
Repartition SD card
We need to repartition, because the boot partition is way too small. It only fits one kernel/initrd, which became a problem I ran into.
Unfortunately gparted doesn’t seem to work on disk images. It Continue reading
Kubernetes Security And Networking 2: Getting Started With Role-Based Access Control (RBAC) – Video
Role-Based Access Control, or RBAC, lets you set permissions around who can access a system and at what level. RBAC is basic, but essential, security function. This video looks at RBAC for Kubernetes from two perspectives: in native Kubernetes and in platforms such as Azure Active Directory. Host Michael Levan brings his background in system […]
The post Kubernetes Security And Networking 2: Getting Started With Role-Based Access Control (RBAC) – Video appeared first on Packet Pushers.
Is a Custom Mobile Device the Right Fit for Your Workforce?
Standardizing on a locked-down device for everyone with one form factor greatly simplifies both the user experience and tech support.Azure Networking Fundamentals: Network Security Group (NSG)
Comment: Here is a part of the introduction section of the second chapter of my Azure Networking Fundamentals book. I will also publish other chapters' introduction sections soon so you can see if the book is for you. The book is available at Leanpub and Amazon (links on the right pane).
This chapter introduces three NSG scenarios. The first example explains the NSG-NIC association. In this section, we create a VM that acts as a Bastion host*). Instead of using the Azure Bastion service, we deploy a custom-made vm-Bastion to snet-dmz and allow SSH connection from the external network. The second example describes the NSG-Subnet association. In this section, we launch vm-Front-1 in the front-end subnet. Then we deploy an NSG that allows SSH connection from the Bastion host IP address. The last part of the chapter introduces an Application Security Group (ASG), which we are using to form a logical VM group. We can then use the ASG as a destination in the security rule in NSG. There are two ASGs in figure 2-1. We can create a logical group of VMs by associating them with the same Application Security Group (ASG). The ASG can then be used Continue reading
Accelerating cloud-native development brings opportunities and challenges for enterprises
By 2025, Gartner estimates that over 95% of new digital workloads will be deployed on cloud-native platforms, up from 30% in 2021. This momentum of these workloads and solutions presents a significant opportunity for companies that can meet the challenges of the burgeoning industry.
As digitalization continues pushing applications and services to the cloud, many companies discover that traditional security, compliance, and observability approaches do not transfer directly to cloud-native architectures. This is the primary takeaway from Tigera’s recent The State of Cloud-Native Security report. As 75% of companies surveyed are focusing on cloud-native application development, it is imperative that leaders understand the differences, challenges, and opportunities of cloud-native environments to ensure they reap the efficiency, flexibility, and speed that these architectures offer.
Containers: Rethinking security
The flexibility container workloads provide makes the traditional ‘castle and moat’ approach to security obsolete. Cloud-native architectures do not have a single vulnerable entry point but many potential attack vectors because of the increased attack surface. Sixty-seven percent of companies named security as the top challenge regarding the speed of deployment cycles. Further, 69% of companies identified container-level firewall capabilities, such as intrusion detection and prevention, web application firewall, protection from “Denial of Service” Continue reading
Intelligent, automatic restarts for unhealthy Kafka consumers
At Cloudflare, we take steps to ensure we are resilient against failure at all levels of our infrastructure. This includes Kafka, which we use for critical workflows such as sending time-sensitive emails and alerts.
We learned a lot about keeping our applications that leverage Kafka healthy, so they can always be operational. Application health checks are notoriously hard to implement: What determines an application as healthy? How can we keep services operational at all times?
These can be implemented in many ways. We’ll talk about an approach that allows us to considerably reduce incidents with unhealthy applications while requiring less manual intervention.
Kafka at Cloudflare
Cloudflare is a big adopter of Kafka. We use Kafka as a way to decouple services due to its asynchronous nature and reliability. It allows different teams to work effectively without creating dependencies on one another. You can also read more about how other teams at Cloudflare use Kafka in this post.
Kafka is used to send and receive messages. Messages represent some kind of event like a credit card payment or details of a new user created in your platform. These messages can be represented in multiple ways: JSON, Protobuf, Avro and so on.
Will DPUs Change the Network?
It’s easy to get excited about what seems to be a new technology and conclude that it will forever change the way we do things. For example, I’ve seen claims that SmartNICs (also known as Data Processing Units – DPU) will forever change the network.
TL&DR: Of course they won’t.
Before we start discussing the details, it’s worth remembering what a DPU is: it’s another server with its own CPU, memory, and network interface card (NIC) that happens to have PCI hardware that emulates the host interface cards. It might also have dedicated FPGA or ASICs.
Network Break 414: 230 Juniper Vulnerabilities, Should Cisco Patch An EOL Router, T-Mobile Takes Weeks To Spot Breach
On today's Network Break podcast we cover a raft of Juniper vulnerabilities, whether Cisco should patch serious vulnerabilities in end-of-life products, a big T-Mobile breach, Avaya dealing with significant debt, sweeping rounds of layoffs, and more IT news.
The post Network Break 414: 230 Juniper Vulnerabilities, Should Cisco Patch An EOL Router, T-Mobile Takes Weeks To Spot Breach appeared first on Packet Pushers.