Holistic web protection: industry recognition for a prolific 2020
I love building products that solve real problems for our customers. These days I don’t get to do so as much directly with our Engineering teams. Instead, about half my time is spent with customers listening to and learning from their security challenges, while the other half of my time is spent with other Cloudflare Product Managers (PMs) helping them solve these customer challenges as simply and elegantly as possible. While I miss the deeply technical engineering discussions, I am proud to have the opportunity to look back every year on all that we’ve shipped across our application security teams.
Taking the time to reflect on what we’ve delivered also helps to reinforce my belief in the Cloudflare approach to shipping product: release early, stay close to customers for feedback, and iterate quickly to deliver incremental value. To borrow a term from the investment world, this approach brings the benefits of compounded returns to our customers: we put new products that solve real-world problems into their hands as quickly as possible, and then reinvest the proceeds of our shared learnings immediately back into the product.
It is these sustained investments that allow us to release a flurry of small improvements Continue reading
Soar: Simulation for Observability, reliAbility, and secuRity
Serving more than approximately 25 million Internet properties is not an easy thing, and neither is serving 20 million requests per second on average. At Cloudflare, we achieve this by running a homogeneous edge environment: almost every Cloudflare server runs all Cloudflare products.
As we offer more and more products and enjoy the benefit of horizontal scalability, our edge stack continues to grow in complexity. Originally, we only operated at the application layer with our CDN service and DoS protection. Then we launched transport layer products, such as Spectrum and Argo. Now we have further expanded our footprint into the IP layer and physical link with Magic Transit. They all run on every machine we have. The work of our engineers enables our products to evolve at a fast pace, and to serve our customers better.
However, such software complexity presents a sheer challenge to operation: the more changes you make, the more likely it is that something is going to break. Continue reading
Automation Win: Chatops-Based Security
It’s amazing how quickly you can deploy new functionality once you have a solid foundation in place. In his latest blog post Adrian Giacometti described how he implemented a security solution that allows network operators to block source IP addresses (identified by security tools) across dozens of firewalls using a bot listening to a Slack channel.
Would you be surprised if I told you we covered similar topics in our automation course? ?
Building Backdoors and Fixing Malfeasance
You might have seen the recent news this week that there is an exploitable backdoor in Zyxel hardware that has been discovered and is being exploited. The backdoor admin account with the clever name ‘zyfwp’ is not something that has been present in the devices forever. The account was put in during firmware version 4.60, which was released in Q4 2020.
Zyxel is rushing to patch the devices and remove the backdoor account. Users are being advised to disable remote administration until the accounts can be deactivated and proven to be removed. However, the bigger question in my mind relates to the addition of the user account in the first place. Why would you knowingly install a backdoor?
Hello, Joshua
Backdoors are nothing new in the computer world. I’d argue the most famous backdoor account in the history of computer hacking belongs to Joshua, the dormant login for the War Operations Programmed Response (WOPR) computer system in the 1983 movie Wargames. Joshua was an old login for the creator to access the system outside of the military chain of command. When the developer was removed from the project the account was forgotten about until a kid discovered it and Continue reading
Considerations for Host-based Firewalls (Part 2)
This is a guest blog post by Matthias Luft, Principal Platform Security Engineer @ Salesforce, and a regular ipSpace.net guest speaker.
A couple of months ago I had the pleasure to publish my first guest post here and, as to be expected from ipspace.net, it triggered some great discussion.
With this input and some open thoughts from the last post, I want to dive into a few more topics.
Application Engineering vs. (?) Network Engineering
One trigger for the initial post was the question whether host-based firewalls (HBFs), potentially combined with solutions to learn rulesets based on flows, are intrinsically better than central firewalls. While we discussed the mileage around that already, comments and questions emphasized how often we have to handle a “software engineering vs. network engineering” mentality – which should not involve any blame in either direction as this mindset is usually enforced by organizational structures.
For whatever it is worth, I can only stress the point that a strong collaboration between software and network engineering will resolve way more issues than any technology. I award myself a “Thanks, Captain Obvious” here, but I still want to make the point to try Continue reading
Playing with Python Ransomware
I recently came across the Python script pokus.py which implements a ransomware function. The code is written for demonstration purposes only, so it completely lacks obfuscation techniques. The goal is to identify the vulnerability that is intentionally implemented in the code so we can restore the attached file - private.gif.rsw (the file is compressed by […]Continue reading...
Integrating Cloudflare Gateway and Access
We’re excited to announce that you can now set up your Access policies to require that all user traffic to your application is filtered by Cloudflare Gateway. This ensures that all of the traffic to your self-hosted and SaaS applications is secured and centrally logged. You can also use this integration to build rules that determine which users can connect to certain parts of your SaaS applications, even if the application does not support those rules on its own.
Stop threats from returning to your applications and data
We built Cloudflare Access as an internal project to replace our own VPN. Unlike a traditional private network, Access follows a Zero Trust model. Cloudflare’s edge checks every request to protected resources for identity and other signals like device posture (i.e., information about a user’s machine, like Operating system version, if antivirus is running, etc.).
By deploying Cloudflare Access, our security and IT teams could build granular rules for each application and log every request and event. Cloudflare’s network accelerated how users connected. We launched Access as a product for our customers in 2018 to share those improvements with teams of any size.
Over the last two years, we Continue reading
New Vulnerability Exposes Kubernetes to Man-in-the-Middle Attacks: Here’s How to Mitigate
What is CVE-2020-8554?
A few weeks ago a solution engineer discovered a critical flaw in Kubernetes architecture and design, and announced that a “security issue was discovered with Kubernetes affecting multi-tenant clusters. If a potential attacker can already create or edit services and pods, then they may be able to intercept traffic from other pods (or nodes) in the cluster.” If a hostile user can create a ClusterIP service and set the spec.externalIP field, they can intercept traffic to that IP. In addition, if a user can patch the status of a LoadBalancer service, which is a privileged operation, they can also intercept traffic by exploiting the vulnerability.
Who is Affected?
All Kubernetes versions including the latest release v1.20 are vulnerable to this attack, with the most significant impact being to multi-tenant clusters. Multi-tenant clusters that grant tenants the ability to create and update services and pods are most vulnerable. Since this is a major design flaw with no fix in sight, it becomes imperative to understand and mitigate this CVE.
Technical Overview
The man-in-the-middle (MITM) attack starts with step 1 (shown in the diagram, below). A workload sends a connection request to legitimate IP 4.4. Continue reading
Beat – An Acoustics Inspired DDoS Attack
On the week of Black Friday, Cloudflare automatically detected and mitigated a unique ACK DDoS attack, which we’ve codenamed “Beat”, that targeted a Magic Transit customer. Usually, when attacks make headlines, it’s because of their size. However, in this case, it’s not the size that is unique but the method that appears to have been borrowed from the world of acoustics.
Acoustic inspired attack
As can be seen in the graph below, the attack’s packet rate follows a wave-shaped pattern for over 8 hours. It seems as though the attacker was inspired by an acoustics concept called beat. In acoustics, a beat is a term that is used to describe an interference of two different wave frequencies. It is the superposition of the two waves. When the two waves are nearly 180 degrees out of phase, they create the beating phenomenon. When the two waves merge they amplify the sound and when they are out of sync they cancel one another, creating the beating effect.
Acedemo.org has a nice tool where you can create your own beat wave. As you can see in the screenshot below, the two waves in blue and red are out Continue reading
Encrypting your WAF Payloads with Hybrid Public Key Encryption (HPKE)
The Cloudflare Web Application Firewall (WAF) blocks more than 72B malicious requests per day from reaching our customers’ applications. Typically, our users can easily confirm these requests were not legitimate by checking the URL, the query parameters, or other metadata that Cloudflare provides as part of the security event log in the dashboard.
Sometimes investigating a WAF event requires a bit more research and a trial and error approach, as the WAF may have matched against a field that is not logged by default.
Not logging all parts of a request is intentional: HTTP headers and payloads often contain sensitive data, including personally identifiable information, which we consider a toxic asset. Request headers may contain cookies and POST payloads may contain username and password pairs submitted during a login attempt among other sensitive data.
We recognize that providing clear visibility in any security event is a core feature of a firewall, as this allows users to better fine tune their rules. To accomplish this, while ensuring end-user privacy, we built encrypted WAF matched payload logging. This feature will log only the specific component of the request the WAF has deemed malicious — and it is encrypted using a customer-provided key Continue reading
The deal with DMCA 1201 reform
There are two fights in Congress now against the DMCA, the "Digital Millennium Copyright Act". One is over Section 512 covering "takedowns" on the web. The other is over Section 1201 covering "reverse engineering", which weakens cybersecurity.
Even before digital computers, since the 1880s, an important principle of cybersecurity has been openness and transparency ("Kerckhoff's Principle"). Only through making details public can security flaws be found, discussed, and fixed. This includes reverse-engineering to search for flaws.
Cybersecurity experts have long struggled against the ignorant who hold the naive belief we should instead coverup information, so that evildoers cannot find and exploit flaws. Surely, they believe, given just anybody access to critical details of our security weakens it. The ignorant have little faith in technology, that it can be made secure. They have more faith in government's ability to control information.
Technologists believe this information coverup hinders well-meaning people and protects the incompetent from embarrassment. When you hide information about how something works, you prevent people on your own side from discovering and fixing flaws. It also means that you can't hold those accountable for their security, since it's impossible to notice security flaws until after they've been exploited. At the Continue reading
Improving DNS Privacy with Oblivious DoH in 1.1.1.1
Today we are announcing support for a new proposed DNS standard — co-authored by engineers from Cloudflare, Apple, and Fastly — that separates IP addresses from queries, so that no single entity can see both at the same time. Even better, we’ve made source code available, so anyone can try out ODoH, or run their own ODoH service!
But first, a bit of context. The Domain Name System (DNS) is the foundation of a human-usable Internet. It maps usable domain names, such as cloudflare.com, to IP addresses and other information needed to connect to that domain. A quick primer about the importance and issues with DNS can be read in a previous blog post. For this post, it’s enough to know that, in the initial design and still dominant usage of DNS, queries are sent in cleartext. This means anyone on the network path between your device and the DNS resolver can see both the query that contains the hostname (or website) you want, as well as the IP address that identifies your device.
To safeguard DNS from onlookers and third parties, the IETF standardized DNS encryption with DNS over HTTPS (DoH) and DNS over TLS (DoT). Both protocols Continue reading
OPAQUE: The Best Passwords Never Leave your Device
Passwords are a problem. They are a problem for reasons that are familiar to most readers. For us at Cloudflare, the problem lies much deeper and broader. Most readers will immediately acknowledge that passwords are hard to remember and manage, especially as password requirements grow increasingly complex. Luckily there are great software packages and browser add-ons to help manage passwords. Unfortunately, the greater underlying problem is beyond the reaches of software to solve.
The fundamental password problem is simple to explain, but hard to solve: A password that leaves your possession is guaranteed to sacrifice security, no matter its complexity or how hard it may be to guess. Passwords are insecure by their very existence.
You might say, “but passwords are always stored in encrypted format!” That would be great. More accurately, they are likely stored as a salted hash, as explained below. Even worse is that there is no way to verify the way that passwords are stored, and so we can assume that on some servers passwords are stored in cleartext. The truth is that even responsibly stored passwords can be leaked and broken, albeit (and thankfully) with enormous effort. An increasingly pressing problem stems from the Continue reading
Current Work in BGP Security
I’ve been chasing BGP security since before the publication of the soBGP drafts, way back in the early 2000’s (that’s almost 20 years for those who are math challenged). The most recent news largely centers on the RPKI, which is used to ensure the AS originating an advertisements is authorized to do so (or rather “owns” the resource or prefix). If you are not “up” on what the RPKI does, or how it works, you might find this old blog post useful—its actually the tenth post in a ten post series on the topic of BGP security.
Recent news in this space largely centers around the ongoing deployment of the RPKI. According to Wired, Google and Facebook have both recently adopted MANRS, and are adopting RPKI. While it might not seem like autonomous systems along the edge adopting BGP security best practices and the RPKI system can make much of a difference, but the “heavy hitters” among the content providers can play a pivotal role here by refusing to accept routes that appear to be hijacked. This not only helps these providers and their customers directly—a point the Wired article makes—this also helps the ‘net in a larger way Continue reading
Fun Times: Another Broken Linux ALG
Dealing with protocols that embed network-layer addresses into application-layer messages (like FTP or SIP) is great fun, more so if the said protocol traverses a NAT device that has to find the IP addresses embedded in application messages while translating the addresses in IP headers. For whatever reason, the content rewriting functionality is called application-level gateway (ALG).
Even when we’re faced with a monstrosity like FTP or SIP that should have been killed with napalm a microsecond after it was created, there’s a proper way of doing things and a fast way of doing things. You could implement a protocol-level proxy that would intercept control-plane sessions… or you could implement a hack that tries to snoop TCP payload without tracking TCP session state.
Not surprisingly, the fast way of doing things usually results in a wonderful attack surface, more so if the attacker is smart enough to construct HTTP requests that look like SIP messages. Enjoy ;)
VMware is Not New to Enterprise Security
By: Keith Luck
None of us can stop thinking about how 2020 has changed the way we go about our daily tasks. Going to school, going to the store, going out to eat — going anywhere at all. But now, for the first time, we are not even going to work! Everyone has been pushed to work from home. This change has a wide-ranging set of variables that need to be addressed, from the business limits on resources for connectivity to the employee’s limits on remote resources of space, privacy, and uninterrupted concentration.
The overnight reliance on remote, personal, shared services for connectivity from the worker to the corporation has forever put an end to the idea of a security perimeter. Zero Trust Architecture (ZTA) has moved from being an academic discussion to a persistent customer requests for solutions. This shift is furthered by the timely release of the US National Institute of Standards and Technology’s NIST Special Publication 800-207 ZTA Guide. At the same time, we now see numerous security industry vendors claiming their products will provide Zero Trust.
Naturally, many VMware customers want Continue reading