Using Google Cloud Platform to Analyze Cloudflare Logs
We’re excited to announce that we now offer deep insights into your domain’s web traffic, working with Google Cloud Platform (GCP). While Cloudflare Enterprise customers always have had access to their logs, they previously had to rely on their own tools to process them, adding extra complexity and cost.
Cloudflare logs provide real time insight into traffic, malicious activity, attack incidents, and infrastructure health checks. The output is used to help customers adjust their settings, manage costs and resources, and plan for expansion.
Working with Google, we created an end-to-end solution that allows customers to retrieve Cloudflare access logs, store and process data in a simple way. GCP components such as Google Storage, Cloud Function, BigQuery and Data Studio come together to make this possible.
One of the biggest challenges of data analysis is to store and process large volume of data within a short time period while avoiding high costs. GCP Storage and BigQuery easily address these challenges.
Cloudflare customers can decide if they wish to obtain and process data from Cloudflare access logs on demand or on a regular basis. The full solution is described in this Knowledge Base article. Initial setup takes no more than 30 minutes Continue reading
Spotify’s Cloudflare App is open source: fork it for your next project
Earlier this year, Cloudflare Apps was launched so app developers may leverage our global network of 6 million+ websites, applications, and APIs. I’d like to take a moment to highlight Spotify, which was a launch partner for Cloudflare Apps, especially since they have elected to open source the code to their Cloudflare App.
About Spotify
Spotify is the leading digital service for streaming music, serving more than 140 million listeners.
What does the Spotify app do?
Recently, Spotify launched a Cloudflare App to instantly and easily embed the Spotify player onto your website without having to copy / paste anything.
Who should install the Spotify app?
A musician who runs a site for their band - they can now play samples of new tracks on their tour calendar page and psych up their fans.
A game creator who wants to share their game's soundtrack with their fans.
An activewear company which wants to deliver popular running playlists to its customers.
Web properties that install the Spotify app have the ability to increase user engagement.
Add Spotify widgets to your web pages and let your users play tracks and follow Spotify profiles. Add a Spotify Play Button Continue reading
How to Monkey-Patch the Linux Kernel
I have a weird setup. I type in Dvorak. But, when I hold ctrl or alt, my keyboard reverts to Qwerty.
You see, the classic text-editing hotkeys, ctrl+Z, ctrl+X, ctrl+C, and ctrl+V are all located optimally for a Qwerty layout: next to the control key, easy to reach with your left hand while mousing with your right. In Dvorak, unfortunately, these hotkeys are scattered around mostly on the right half of the keyboard, making them much less convenient. Using Dvorak for typing but Qwerty for hotkeys turns out to be a nice compromise.
But, the only way I could find to make this work on Linux / X was to write a program that uses X "grabs" to intercept key events and rewrite them. That was mostly fine, until recently, when my machine, unannounced, updated to Wayland. Remarkably, I didn't even notice at first! But at some point, I realized my hotkeys weren't working right. You see, Wayland, unlike X, actually has some sensible security rules, and as a result, random programs can't just man-in-the-middle all keyboard events anymore. Which broke my setup.
Yes, that's right, I'm that guy:
Source: xkcd 1172
So what was I to do? I began Continue reading
Why does one NGINX worker take all the load?
Scaling up TCP servers is usually straightforward. Most deployments start by using a single process setup. When the need arises more worker processes are added. This is a scalability model for many applications, including HTTP servers like Apache, NGINX or Lighttpd.
CC BY-SA 2.0 image by Paul Townsend
Increasing the number of worker processes is a great way to overcome a single CPU core bottleneck, but opens a whole new set of problems.
There are generally three ways of designing a TCP server with regard to performance:
(a) Single listen socket, single worker process.
(b) Single listen socket, multiple worker processes.
(c) Multiple worker processes, each with separate listen socket.
(a) Single listen socket, single worker process This is the simplest model, where processing is limited to a single CPU. A single worker process is doing both accept() calls to receive the new connections and processing of the requests themselves. This model is the preferred Lighttpd setup.
(b) Single listen socket, multiple worker process The new connections sit in a single kernel data structure (the listen socket). Multiple worker processes are doing both the accept() calls and processing of the requests. This model enables some spreading of the inbound Continue reading
Performing & Preventing SSL Stripping: A Plain-English Primer
Over the past few days we learnt about a new attack that posed a serious weakness in the encryption protocol used to secure all modern Wi-Fi networks. The KRACK Attack effectively allows interception of traffic on wireless networks secured by the WPA2 protocol. Whilst it is possible to backward patch implementations to mitigate this vulnerability, security updates are rarely installed universally.
Prior to this vulnerability, there were no shortage of wireless networks that were vulnerable to interception attacks. Some wireless networks continue to use a dated security protocol (called WEP) that is demonstrably "totally insecure" 1; other wireless networks, such as those in coffee shops and airports, remain completely open and do not authenticate users. Once an attacker gains access to a network, they can act as a Man-in-the-Middle to intercept connections over the network (using tactics known as ARP Cache Poisoning and DNS Hijacking). And yes, these interception tactics can easily be deployed against wired networks where someone gains access to an ethernet port.
With all this known, it is beyond doubt that it is simply not secure to blindly trust the medium that connects your users to the internet. HTTPS was created to allow HTTP traffic to Continue reading
Helping to make LuaJIT faster
This is a guest post by Laurence Tratt, who is a programmer and Reader in Software Development in the Department of Informatics at King's College London where he leads the Software Development Team. He is also an EPSRC Fellow.
Programming language Virtual Machines (VMs) are familiar beasts: we use them to run apps on our phone, code inside our browsers, and programs on our servers. Traditional VMs are useful and widely used: nearly every working programmer is familiar with one or more of the “standard” Lua, Python, or Ruby VMs. However, such VMs are simplistic, containing only an interpreter (a simple implementation of a language). These often can’t run our programs as fast as we need; and, even when they can, they often waste huge amounts of server CPU time. We sometimes forget that servers consume a large, and growing, chunk of the world’s electricity output: slow language implementations are, quite literally, changing the world, and not in a good way.
More advanced VMs come with Just-In-Time (JIT) compilers (well known examples include LuaJIT, HotSpot (aka “the JVM”), PyPy, and V8). Such VMs observe a program’s run-time behaviour and use that to compile frequently executed parts of the program Continue reading
A Celebration of Learning at Grace Hopper
Photo by Cloudflare Staff
Over the course of my career, I’ve been to many conferences, interacted with thousands of candidates, and attended countless keynotes, roundtables, and sessions. I can say without a doubt, that the Grace Hopper Celebration, stood out from the rest. And I think my team would agree.
During the three day event, we screened more than 50 candidates, conducted 24 onsite interviews, and had more than 600 people visit our booth. Not bad for a booth near the back competing with an AirBnB booth that had a literal house on top of it.
Before the conference, we were expecting about 200 visitors to our booth, so the turnout clearly exceeded our expectations. More importantly, we couldn’t have predicted the breadth of talent we would interact with at the conference. That’s not to say that I was surprised; Grace Hopper attracts women from all over the world, including students, seasoned professionals, hackers, engineers, and business leaders. This year was the biggest yet, with more than 12,000 attendees from across all tech sectors, backgrounds, and interests. So I certainly wasn’t surprised to meet all of these women, but I was definitely inspired.
Photo by Cloudflare Staff
My team Continue reading
Cloudflare London Meetup Recap
Cloudflare helps make over 6 million websites faster and more secure. In doing so, Cloudflare has a vast and diverse community of users throughout the world. Whether discussing Cloudflare on social media, browsing our community forums or following Pull Requests on our open-source projects; there is no shortage of lively discussions amongst Cloudflare users. Occasionally, however, it is important to move these discussions out from cyberspace and take time to connect in person.
A little while ago, we did exactly this and ran a meetup in the Cloudflare London office. Ivan Rustic from Hardenize was our guest speaker, he demonstrated how Hardenize developed a Cloudflare App to help build a culture of security. I presented two other talks which included a primer on how the Cloudflare network is architected and wrapped up with a discussion on how you can build and monetise your very own Cloudflare App.
Since we presented this meet-up, I've received a few requests to share the videos of all the talks. You can find all three of the talks from our last London office meet-up in this blog post.
How Cloudflare Works
App Highlight: Hardenize by Ivan Ristić
Introduction to Building with Cloudflare Apps
Learn More...
A Fast, Secure Migration to Google Cloud Platform using Cloudflare
OnAir Video Presentation
Abstract
Looking to host your website, application, or API in the cloud, or migrate to a new cloud provider while keeping your data secure? In this webinar, Trey Guinn, Head of Solutions Engineering at Cloudflare, will discuss how companies should approach security, during and after migration. We'll highlight the migration story of LUSH, one of the largest global e-Commerce cosmetic retailers, and how they took the right steps to migrate from their previous cloud provider to Google Cloud Platform, in less than 3 weeks. Trey will be performing a live demo on setting up Cloudflare load balancing across cloud providers, as well as
Speakers
Asad Baheri
Security & Networking Partner Manager
Google Cloud Platform
Trey Guinn
Head of Solutions Engineering
Cloudflare
Webinar Transcription and Load Balancing Demo
Asad Baheri
Today we're going to talk about LUSH's migration to Google Cloud and how Cloudflare, one of our top security and performance partners, can help you with your own cloud migration. Throughout our presentation, we'll be talking about security best practices, how CDNs and the CDN Interconnect program works, and we're also going to also give you a demo of Cloudflare's load balancing to start your migration.
One of Continue reading
Dobar dan, Hrvatska! Announcing Cloudflare’s Zagreb Data Center
Fire the Gric Cannon! Hot on the heels of several birthday week product announcements, we continue to expand our global network.
Cloudflare is excited to announce the launch of our newest data center in Zagreb, Croatia, furthering the breadth of our network to 118 cities across 58 countries. Our Europe network alone now spans 33 cities across 25 countries (with at least ten new cities being planned). [For trivia fans: Our list of data centers beginning with the letter Z now spans four cities, with the others being Zhengzhou, Zhuzhou and Zurich].
Zagreb: A Thousand Years
CC BY 2.0 image by Mario Fajt, sobrecroacia.com
With a rich history going back almost a thousand years, Zagreb is sometimes called the City of Museums. Visitors can join the Saturday spica to Dolac market to try out the traditional paprenjak biscuit, hop on the shortest cable car in the world, explore Maksimir Park and more in this charming European city.
Local Interconnection
Croatia is home to over 3 million Internet users, with Internet penetration approaching 75%, which is high but still significantly lower than the European average of ~85%. Our newest deployment improves the security and Continue reading
WHOIS going to be at the Grace Hopper Celebration?
Ubuntu us are doing the round trip! It’s time to live - WAN you arrive at GHC, come meet us and say HELO (we love GNU faces, we’ll be very api to meet you). When you’re exhausted like IPv4, git over to the Cloudflare corner to reboot –– we’ll have chargers and Wi-Fi (it’s not a SYN to REST). R booth can be your ESC. Then Thursday morning we’re hosting a breakfast bash with Zendesk –– it will be quite the Assembly, you should definitely Go, compile a bowl of serial, drink a bit of CIDR or a cup of tee.
I’m also speaking at 1:30PM on Wednesday in OCCC W414 hashing out encryption and updates for IoT –– DES should be a fun session.
ACK! I did NAT tell you how to find us. Check for sum women in capes a few hops away from the booths with the lava LAMP stack. I'm the one with cURLs.
In D air! Excited to LANd. C you soon.
Go Hack Nights at Cloudflare
At Cloudflare we're extensively using the Go programming language to build a better Internet. Go is a free and open source programming language created by Google in 2007 and open sourced in 2009. Earlier this year, Go made news when it entered the list of top 10 programming languages on the TIOBE Index.
Recently we launched an internal monthly Go Hack Night at our San Francisco office, open to anyone who works at Cloudflare regardless of their department or position. Anyone from newbie programmers to our most experienced Go engineers are encouraged to attend, and experienced engineers are asked to throw on a mentor badge and help guide colleagues with installing and learning Go.
We had over 30 attendees at our inaugural Go Hack Night, and our survey reveals some great stats:
- 26% of attendees were completely new to programming
- 61% of attendees were experienced in other languages but new to Go
- Every attendee said they learned something!
We actively encourage an inclusive learning culture and we're super excited to make the Go programming language more accessible to our entire company.
If you're interested in working with Go and helping to build a better Internet, we're hiring!
P.S. if Continue reading
A New Cybersecurity Strategy for Europe
October is European Cybersecurity Month, an annual advocacy campaign to raise awareness of cyber risks among citizens and businesses, and to share best practices in cybersecurity. This year’s campaign was launched at an event in Estonia, a country which both holds the current Presidency seat of the European Council and is well known as being highly cyber aware and digitally savvy.
It is fitting, therefore, that it is under Estonia’s Presidency that the European Commission announced a number of initiatives last month aimed at stepping up the European Union’s cybersecurity capacity and response to cyber attacks, while laying the foundations for increased cyber awareness and better cyber hygiene overall.
This EU’s Cybersecurity Strategy is a welcome initiative, as we already know that the overall cyber threat level is rising. At Cloudflare, we deal with a new type of DDoS attack every 3 minutes, and it has been that way for the last 6 months. This year alone, we've seen a DDoS attack that peaked at 300 Mpps and another at 480 Gbps. Furthermore, as DDoS mitigation companies like Cloudflare have become adept at handling 'traditional' DDoS attacks, the attackers have also adapted and increasingly try out new techniques.
A holistic Continue reading
Code Everywhere: Why We Built Cloudflare Workers
It all comes down to the speed of light. It always does. The speed of light limits the latency possible between someone using the Internet and the application they are accessing. It doesn’t matter if they are walking down the street hailing a car using a ride-sharing app, sitting in an office accessing a SaaS application on the web, or if their wearable device is reporting health information over WiFi. The speed of light is everywhere.
When you can’t fight the speed of light you only have one possible solution: move closer to where the end users are. In simplistic terms, that’s what Cloudflare has done by building its network of 117 data centers around the world. We’ve cut the latency between users and servers by moving closer.
The Missing Link
But to date all we’ve moved closer are things like SSL handshakes, WAF processing of requests and caching of content. All those things help make Internet applications faster and safer, but there’s a huge missing component... code.
The code that makes Internet applications work is still sequestered in servers and cloud services around the world. And there are only a limited number of such locations even for large cloud Continue reading
Introducing Cloudflare Workers: Run Javascript Service Workers at the Edge
TL;DR: You'll soon be able to deploy Javascript to Cloudflare's edge, written against an API similar to Service Workers.
Try writing a Worker in the playground »
Introduction
Every technology, when sufficiently complicated, becomes programmable.
You see this everywhere, but as a lifelong gamer, my personal favorite example is probably graphics cards. In the '90s, graphics hardware generally provided a fixed set of functionality. The OpenGL standard specified that the geometry pipeline would project points from 3D space onto your viewport, then the raster pipeline would draw triangles between them, with gradient shading and perhaps a texture applied. You could only use one texture at a time. There was only one lighting algorithm, which more or less made every surface look like plastic. If you wanted to do anything else, you often had to give up the hardware entirely and drop back to software.
Of course, new algorithms and techniques were being developed all the time. So, hardware vendors would add the best ideas to their hardware as "extensions". OpenGL ended up with hundreds of vendor-specific extensions to support things like multi-texturing, bump maps, reflections, dynamic shadows, and more.
Then, in 2001, everything changed. The first GPU with a programmable Continue reading
Cloudflare Apps Platform Update: September Edition
This is the September edition of our blog series showcasing the latest platform improvements in developer analytics, user feedback, release notes, and more!
Since launch, we’ve received hundreds of feature requests from developers and users alike. Feedback has been the source of some our most popular features. This month’s post is celebration of the innovation achieved when great ideas are shared.
Let’s dive in!
? Developer Analytics
Continuing with the theme of feedback, App developers can now track their apps’ popularity and growth:
The usage charts help identify which changes have a positive impact on your app.
If you’ve created a paid app you can also track its financial performance:
? User feedback
Charts and graphs are great for tracking trends, but what do your users actually think of your app? Wonder no longer; users can now leave comments when adding and removing apps from their site. Each comment includes sentiment tags and an optional message from the user.
? Page Selectors
Cloudflare users have always been able to select which routes their apps are active, though apps this was too course Continue reading
Introducing Cloudflare Warp: Hide Behind The Edge
I work at a company whose job it is to be attacked. As I’m writing this, an automatic mitigation is fighting two ongoing DDoS attacks. Any machine that’s publicly routable on the internet today can be a vector for attack, and that’s a problem.
Today we want to turn the tables and give you a new way of exposing services to the internet without having them be directly, publicly routable. Meet Cloudflare Warp.
CC BY-SA 2.0 image by Christian Ortiz
Playing Hide and Seek with Bots and Hackers
Cloudflare internally runs about 4,000 containers that make up about 1.5K services and applications. Some of these containers need to network with other local containers, and others need to accept connections over the wire.
Every devops engineer knows that bad things happen to good machines, and so our platform operations team tries to hide servers altogether from the internet. There are several ways to do this:
- Rotate IP addresses
- Deploy proxies
- Create firewall rules
- Configure IP tables
- Limit connections by client certificate
- Cross connect with an upstream provider
- Configure a GRE tunnel
- Authentication mechanisms like OAuth or OIDC
These can be complicated or time consuming, yet none of them are Continue reading
Down the Rabbit Hole: The Making of Cloudflare Warp
In the real world, tunnels are often carved out from the mass of something bigger - a hill, the ground, but also man-made structures.
CC BY-SA 2.0 image by Matt Brown
In an abstract sense Cloudflare Warp is similar; its connection strategy punches a hole through firewalls and NAT, and provides easy and secure passage for HTTP traffic to your origin. But the technical reality is a bit more interesting than this strained metaphor invoked by the name of similar predecessor technologies like GRE tunnels.
Relics
Generic Routing Encapsulation or GRE is a well-supported standard, commonly used to join two networks together over the public Internet, and by some CDNs to shield an origin from DDoS attacks. It forms the basis of the legacy VPN protocol PPTP.
Establishing a GRE tunnel requires configuring both ends of the tunnel to accept the other end’s packets and deciding which IP ranges should be routed through the tunnel. With this in place, an IP packet destined for any address in the configured range will be encapsulated within a GRE packet. The GRE packet is delivered directly to the other end of the tunnel, which removes the encapsulation and forwards the original Continue reading
Introducing Cloudflare Stream: Fixing the Streaming Video Market
Cloudflare turns seven years old today. We launched on September 27, 2010.
It was only a few days after our launch that we got our first request to support video streaming. Yet, until today, we'd avoided it.
Why? Simply put: the video streaming market is screwed up. While there's a lot of money spent on video, there are only really about 1,000 customers that do any meaningful level of streaming.
This is in large part because it's technically far too complicated. If you want to move beyond just uploading your videos to a consumer service like YouTube, then you have to use at least three different services. You need someone to encode your video into a streamable format, you need someone else to act as the content delivery network delivering the bytes, and you need someone else still to provide the player code that runs on the client device. Further, since video encoding standards keep evolving and vary across generations of devices, it becomes challenging to ensure a consistently high quality experience for all visitors.
And if that sounds like a technical mess, the business side is even worse. Encoding companies charge based on CPU usage, which is driven by Continue reading
How Cloudflare Streams
Every day, close to 1 billion people watch video through Cloudflare. That’s 100 petabytes of video every month. Last year, video made up 73% of all internet traffic, which is why earlier today we announced Cloudflare Stream, an end to end video solution designed to bring instant video to any device and network connection. Here’s how Stream works to optimize video upload, encoding and delivery.
Uploads
Using Cloudflare Stream starts with a call to the Stream API to /upload a video.
HTTP does not by default provide a reliable upload mechanism for large files, which can make it tricky to upload large media content like high quality video. If there’s any latency in the network connection, a timeout can cancel the entire upload and require the client to start over.
We use an open source upload protocol called Tus which allows for resumable uploads where the upload previously failed. Tus does this by splitting the download into manageable chunks and tracking completed chunks on the server.
Encodes
Video files are containers that hold the video file, the audio track, and some metadata. The video file is compressed by a codec. The codec is a compression algorithm for encoding video Continue reading