Yerevan, Armenia: Cloudflare Data Center #103
In the coming days, Cloudflare will be announcing a series of new data centers across five continents. We begin with Yerevan, the capital and largest city of Armenia, the mountainous country in the South Caucasus. This deployment is our 37th data center in Asia, and 103rd data center globally.
History
Yerevan, one of the oldest continuously inhabited cities in the world, has a rich history going back all the way to 782 BC. Famous for its cognac, lavash flatbread, and beautiful medieval churches, Armenia is also home to more chess grandmasters per capita than most countries!
6 Million Websites Faster
Latency (ms) decreases 6x for UCOM Internet user in Yerevan to Cloudflare. Source: Cedexis
The newest Cloudflare deployment will make 6 million Internet properties faster and more secure, as we serve traffic to Yerevan and adjoining countries.
If the Cloudflare datacenter closest to the Equator (to date) was Singapore, the next deployment brings us even closer. Which one do you think it is?
The Cloudflare network today
－ The Cloudflare Team
Introducing Zero Round Trip Time Resumption (0-RTT)
Cloudflare’s mission is to help build a faster and more secure Internet. Over the last several years, the Internet Engineering Task Force (IETF) has been working on a new version of TLS, the protocol that powers the secure web. Last September, Cloudflare was the first service provider to enable people to use this new version of the protocol, TLS 1.3, improving security and performance for millions of customers.
Today we are introducing another performance-enhancing feature: zero round trip time resumption, abbreviated as 0-RTT. About 60% of the connections we see are from people who are visiting a site for the first time or revisiting after an extended period of time. TLS 1.3 speeds up these connections significantly. The remaining 40% of connections are from visitors who have recently visited a site and are resuming a previous connection. For these resumed connections, standard TLS 1.3 is safer but no faster than any previous version of TLS. 0-RTT changes this. It dramatically speeds up resumed connections, leading to a faster and smoother web experience for web sites that you visit regularly. This speed boost is especially noticeable on mobile networks.
We’re happy to announce that 0-RTT is Continue reading
An AMP validator you can cURL
Cloudflare has been a long time supporter of AMP, an open-source markup language 1.5 billion web pages are using to accelerate their mobile web performance. Cloudflare runs Ampersand, the only alternative to Google’s AMP cache, and earlier this year we launched Accelerated Mobile Links, a way for sites on Cloudflare to open external links on their site in AMP format, as well as Firebolt, leveraging AMP to speed up ad performance.
One of the biggest challenges developers face in converting their web pages to AMP is testing their AMP pages for valid AMP syntax before deploying. It's not enough to make the templates work at dev time, you also need to validate individual pages before they’re published. Imagine, for example, a publishing company where content creators who are unfamiliar with AMP are modifying pages. Because the AMP markup language is so strict, one person adding an interactive element to a page can all of a sudden break the AMP formatting and stop the page from validating.
We wanted to make it as easy as possible to move webpages and sites to AMP so we built an AMP linter API for developers to check that their Continue reading
Cloudflare at Google NEXT 2017
The Cloudflare team is headed to Google NEXT 2017 from March 8th - 10th at Moscone Center in San Francisco, CA. We’re excited to meet with customers, partners, and new friends.
Come learn about Cloudflare’s recent partnership with Google Cloud Platform (CGP) through their CDN Interconnect Program. Cloudflare offers performance and security to over 25,000 Google Cloud Platform customers. The CDN Interconnect program allows Cloudflare’s servers to establish high-speed interconnections with Google Cloud Platform at various locations around the world, accelerating dynamic content while reducing bandwidth and egress billing costs.
We’ll be at booth C7 discussing the benefits of Cloudflare, our partnership with Google Cloud Platform, and handing out Cloudflare SWAG. In addition, our Co-Founder, Michelle Zatlyn, will be presenting “What is Google Cloud Platform’s CDN Interconnect Program?
Google Cloud Platform’s CDN Interconnect program allows select CDN providers to establish direct interconnect links with Google’s edge network at various locations. Customers egressing network traffic from Google Cloud Platform through one of these links will benefit from the direct connectivity to the CDN providers and will Continue reading
Quantifying the Impact of “Cloudbleed”
Last Thursday we released details on a bug in Cloudflare's parser impacting our customers. It was an extremely serious bug that caused data flowing through Cloudflare's network to be leaked onto the Internet. We fully patched the bug within hours of being notified. However, given the scale of Cloudflare, the impact was potentially massive.
The bug has been dubbed “Cloudbleed.” Because of its potential impact, the bug has been written about extensively and generated a lot of uncertainty. The burden of that uncertainty has been felt by our partners, customers, our customers’ customers. The question we’ve been asked the most often is: what risk does Cloudbleed pose to me?
We've spent the last twelve days using log data on the actual requests we’ve seen across our network to get a better grip on what the impact was and, in turn, provide an estimate of the risk to our customers. This post outlines our initial findings.
The summary is that, while the bug was very bad and had the potential to be much worse, based on our analysis so far: 1) we have found no evidence based on our logs that the bug was maliciously exploited before it was patched; Continue reading
Incident report on memory leak caused by Cloudflare parser bug
Last Friday, Tavis Ormandy from Google’s Project Zero contacted Cloudflare to report a security problem with our edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare.
It turned out that in some unusual circumstances, which I’ll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.
For the avoidance of doubt, Cloudflare customer SSL private keys were not leaked. Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug.
We quickly identified the problem and turned off three minor Cloudflare features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) that were all using the same HTML parser chain that was causing the leakage. At that point it was no longer possible for memory to be returned in an HTTP response.
Because of the seriousness of such a bug, a cross-functional team from software engineering, infosec and operations formed in San Francisco and London to fully understand Continue reading
LuaJIT Hacking: Getting next() out of the NYI list
At Cloudflare we’re heavy users of LuaJIT and in the past have sponsored many improvements to its performance.
LuaJIT is a powerful piece of software, maybe the highest performing JIT in the industry. But it’s not always easy to get the most out of it, and sometimes a small change in one part of your code can negatively impact other, already optimized, parts.
One of the first pieces of advice anyone receives when writing Lua code to run quickly using LuaJIT is “avoid the NYIs”: the language or library features that can’t be compiled because they’re NYI (not yet implemented). And that means they run in the interpreter.
CC BY-SA 2.0 image by Dwayne Bent
Another very attractive feature of LuaJIT is the FFI library, which allows Lua code to directly interface with C code and memory structures. The JIT compiler weaves these memory operations in line with the generated machine language, making it much more efficient than using the traditional Lua C API.
Unfortunately, if for any reason the Lua code using the FFI library has to run under the interpreter, it takes a very heavy performance hit. As it happens, under the interpreter the FFI is usually Continue reading
You can now use Google Authenticator and any TOTP app for Two-Factor Authentication
Since the very beginning, Cloudflare has offered two-factor authentication with Authy, and starting today we are expanding your options to keep your account safe with Google Authenticator and any Time-based One Time Password (TOTP) app of your choice.
If you want to get started right away, visit your account settings. Setting up Two-Factor with Google Authenticator or with any TOTP app is easy - just use the app to scan the barcode you see in the Cloudflare dashboard, enter the code the app returns, and you’re good to go.
Importance of Two-Factor Authentication
Often when you hear that an account was ‘hacked’, it really means that the password was stolen.
If the media stopped saying 'hacking' and instead said 'figured out their password', people would take password security more seriously.
— Khalil Sehnaoui (@sehnaoui) January 5, 2017
Two-Factor authentication is sometimes thought of as something that should be used to protect important accounts, but the best practice is to always enable it when it is available. Without a second factor, any mishap involving your password can lead to a compromise. Journalist Mat Honan’s high profile compromise in 2012 is a great example of the importance of two-factor authentication. When Continue reading
Discovering Great Talent with Path Forward
In the fall of 2016, I was just beginning my job search. I’d been lucky to lead HR at a number of great cutting-edge technology start-ups, and I was looking for my next adventure. I wanted to find a company that wasn’t just a great business--I wanted one that was also making a positive impact on the world, and one that had a mission I felt passionately about.
During my two decades running HR/People organizations, I’ve spent a lot of time working with--and talking to--parents in the workplace. I’ve been motivated to do so for a few reasons. According to the US census, mothers are the fastest-growing segment of the US workforce. Companies struggle to retain talented workers after they’ve become parents, especially mothers. It’s been reported that 43 percent of highly qualified women with children leave their careers. Millennials (who make up the majority of the US workforce) are reporting that they want to be more engaged parents and are placing a high value on companies that allow them to parent and still get promoted. Ultimately, I’ve come to believe that the skills you acquire while parenting are extremely relevant and valuable to the workforce.
So when Path Continue reading
NCC Group’s Cryptography Services audits our Go TLS 1.3 stack
The Cloudflare TLS 1.3 beta is run by a Go implementation of the protocol based on the Go standard library, crypto/tls. Starting from that excellent Go codebase allowed us to quickly start experimenting, to be the first wide server deployment of the protocol, and to effectively track the changes to the specification draft.
Of course, the security of a TLS implementation is critical, so we engaged NCC Group's Cryptography Services to perform an audit at the end of 2016.
You can find the codebase on the Cloudflare GitHub. It's a drop-in replacement for crypto/tls and comes with a go wrapper to patch the standard library as needed.
The code is developed in the open but is currently targeted only at internal use: the repository is frequently rebased and the API is not guaranteed to be stable or fully documented. You can take a sneak peek at the API here.
The final goal is to upstream the patches to the Go project so that all users of the Go standard library benefit from it. You can follow the process here.
Below we republish the article about the audit first appeared on the NCC Group's blog.
NCC Group's Cryptography Services Complete Continue reading
Want to see your DNS analytics? We have a Grafana plugin for that
Curious where your DNS traffic is coming from, how much DNS traffic is on your domain, and what records people are querying for that don’t exist? We now have a Grafana plugin for you.
Grafana is an open source data visualization tool that you can use to integrate data from many sources into one cohesive dashboard, and even use it to set up alerts. We’re big Grafana fans here - we use Grafana internally for our ops metrics dashboards.
In the Cloudflare Grafana plugin, you can see the response code breakdown of your DNS traffic. During a random prefix flood, a common type of DNS DDoS attack where an attacker queries random subdomains to bypass DNS caches and overwhelm the origin nameservers, you will see the number of NXDOMAIN responses increase dramatically. It is also common during normal traffic to have a small amount of negative answers due to typos or clients searching for missing records.
You can also see the breakdown of queries by data center and by query type to understand where your traffic is coming from and what your domains are being queried for. This is very useful to identify localized issues, and to see how your Continue reading
Cloudflare Crypto Meetup #5: February 28, 2017
Come join us on Cloudflare HQ in San Francisco on Tuesday, Febrary 28, 2017 for another cryptography meetup. We again had a great time at the last one, we decided to host another. It's becoming a pattern.
We’ll start the evening at 6:00p.m. with time for networking, followed up with short talks by leading experts starting at 6:30p.m. Pizza and beer are provided! RSVP here.
Here are the confirmed speakers:
Deirdre Connolly
Deirdre is a senior software engineer at Brightcove, where she is trying to secure old and new web applications. Her interests include applied cryptography, secure defaults, elliptic curves and their isogenies.
Post-quantum cryptography
Post-quantum cryptography is an active field of research in developing new cryptosystems that will be resistant to attack by future quantum computers. Recently a somewhat obscure area, isogeny-based cryptography, has been getting more attention, including impressive speed and compression optimizations and robust security analyses, bringing it into regular discussion alongside other post-quantum candidates. This talk will cover isogeny-based crypto, specifically these recents results regarding supersingular isogeny diffie-hellman, which is a possible replacement for the ephemeral key exchanges in use today.
Maya Kaczorowski
Maya Kaczorowski is a Product Manager at Google in Security Continue reading
DDoS Ransom: An Offer You Can Refuse
Cloudflare has covered DDoS ransom groups several times in the past. First, we reported on the copycat group claiming to be the Armada Collective and then not too long afterwards, we covered the "new" Lizard Squad. While in both cases the groups made threats that were ultimately empty, these types of security events can send teams scrambling to determine the correct response. Teams in this situation can choose from three types of responses: pay the ransom and enable these groups to continue their operations, not pay and hope for the best, or prepare an action plan to get protected.
Breaking the Ransom Cycle
We can’t stress enough that you should never pay the ransom. We fully understand that in the moment when your website is being attacked it might seem like a reasonable solution, but by paying the ransom, you only perpetuate the DDoS ransom group’s activities and entice other would be ransomers to start making similar threats. In fact, we have seen reports of victim organizations receiving multiple subsequent threats after they have paid the ransom. It would seem these groups are sharing lists of organizations that pay, and those organizations are more likely to be targeted again in Continue reading
NANOG – the art of running a network and discussing common operational issues
The North American Network Operators Group (NANOG) is the loci of modern Internet innovation and the day-to-day cumulative network-operational knowledge of thousands and thousands of network engineers. NANOG itself is a non-profit membership organization; but you don’t need to be a member in order to attend the conference or join the mailing list. That said, if you can become a member, then you’re helping a good cause.
The next NANOG conference starts in a few days (February 6-8 2017) in Washington, DC. Nearly 900 network professionals are converging on the city to discuss a variety of network-related issues, both big and small; but all related to running and improving the global Internet. For this upcoming meeting, Cloudflare has three network professionals in attendance. Two from the San Francisco office and one from the London office.
With the conference starting next week, it seemed a great opportunity to introduce readers of the blog as to why a NANOG conference is so worth attending.
Tutorials
While it seems obvious how to do some network tasks (you unpack the spiffy new wireless router from its box, you set up its security and plug it in); alas the global Internet is somewhat more complex. Continue reading
Protecting everyone from WordPress Content Injection
Today a severe vulnerability was announced by the WordPress Security Team that allows unauthenticated users to change content on a site using unpatched (below version 4.7.2) WordPress.
CC BY-SA 2.0 image by Nicola Sap De Mitri
The problem was found by the team at Sucuri and reported to WordPress. The WordPress team worked with WAF vendors, including Cloudflare, to roll out protection before the patch became available.
Earlier this week we rolled out two rules to protect against exploitation of this issue (both types mentioned in the Sucuri blog post). We have been monitoring the situation and have not observed any attempts to exploit this vulnerability before it was announced publicly.
Customers on a paid plan will find two rules in WAF, WP0025A and WP0025B, that protect unpatched WordPress sites from this vulnerability. If the Cloudflare WordPress ruleset is enabled then these rules are automatically turned on and blocking.
Protecting Everyone
As we have in the past with other serious and critical vulnerabilities like Shellshock and previous issues with JetPack, we have enabled these two rules for our free customers as well.
Free customers who want full protection for their WordPress sites can upgrade to a Continue reading
TLS 1.3 explained by the Cloudflare Crypto Team at 33c3
Nick Sullivan and I gave a talk about TLS 1.3 at 33c3, the latest Chaos Communication Congress. The congress, attended by more that 13,000 hackers in Hamburg, has been one of the hallmark events of the security community for more than 30 years.
You can watch the recording below, or download it in multiple formats and languages on the CCC website.
The talk introduces TLS 1.3 and explains how it works in technical detail, why it is faster and more secure, and touches on its history and current status.
The slide deck is also online.
This was an expanded and updated version of the internal talk previously transcribed on this blog.
TLS 1.3 hits Chrome and Firefox Stable
In related news, TLS 1.3 is reaching a percentage of Chrome and Firefox users this week, so websites with the Cloudflare TLS 1.3 beta enabled will load faster and more securely for all those new users.
You can enable the TLS 1.3 beta from the Crypto section of your control panel.
Firebolt: the fastest, safest ads on the web
Cloudflare’s mission is to help build a better Internet. That means a faster, more secure, open Internet world-wide. We have millions of customers using our services like free SSL, an advanced WAF, the latest compression and the most up to date security to ensure that their web sites, mobile apps and APIs are secure and fast.
One vital area of web technology has lagged behind in terms of speed and security: online ads. And consumers have been turning to ad blocking technology to secure and speed up their own web browsing.
Today, Cloudflare is introducing a new product to make web ads secure, fast and safe. That product is Firebolt.
Firebolt
With Firebolt, ad networks can instantly speed up and secure their ads, resulting in happy consumers and better conversion rates.
Firebolt delivers:
Lightning fast ad delivery
Cloudflare's global network of 102 data centers in 50 countries, combined with routing and performance technologies, makes the delivery of online ads to any device up to five times faster.
Free, simple SSL
Adding SSL to ad serving has been challenging for some ad networks. Cloudflare has years of experience providing free, one click SSL for our customers. Firebolt ads are Continue reading
Introducing Accelerated Mobile Links: Making the Mobile Web App-Quick
In 2017, we've predicted that more than half of the traffic to Cloudflare's network will come from mobile devices. Even if they are formatted to be displayed on a small screen, the mobile web is built on traditional web protocols and technologies that were designed for desktop CPUs, network connections, and displays. As a result, browsing the mobile web feels sluggish compared with using native mobile apps.
In October 2015, the team at Google announced Accelerated Mobile Pages (AMP), a new, open technology to make the mobile web as fast as native apps. Since then, a large number of publishers have adopted AMP. Today, 600 million pages across 700,000 different domains are available in the AMP format.
The majority of traffic to this AMP content comes from people running searches on Google.com. If a visitor finds content through some source other than a Google search, even if the content can be served from AMP, it typically won't be. As a result, the mobile web continues to be slower than it needs to be.
Making the Mobile Web App-Quick
Cloudflare's Accelerated Mobile Links helps solve this problem, making content, regardless of how it's discovered, app-quick. Once enabled, Accelerated Mobile Continue reading
Cloudflare’s Transparency Report for Second Half 2016 and an Additional Disclosure for 2013
Cloudflare is publishing today its seventh transparency report, covering the second half of 2016. For the first time, we are able to present information on a previously undisclosed National Security Letter (NSL) Cloudflare received in the 2013 reporting period.
Wikipedia provides the most succinct description of an NSL:
An NSL is an administrative subpoena issued by the United States federal government to gather information for national security purposes. NSLs do not require prior approval from a judge.… NSLs typically contain a nondisclosure requirement, frequently called a "gag order", preventing the recipient of an NSL from disclosing that the FBI had requested the information. https://en.wikipedia.org/wiki/National_security_letter
Shortly before the New Year, the FBI sent us the following letter about that NSL.
The letter withdrew the nondisclosure provisions (the “gag order”) contained in NSL-12-358696, which had constrained Cloudflare since the NSL was served in February 2013. At that time, Cloudflare objected to the NSL. The Electronic Frontier Foundation agreed to take our case, and with their assistance, we brought a lawsuit under seal to protect its customers' rights.
Early in the litigation, the FBI rescinded the NSL in July 2013 and withdrew the request for information. So no customer Continue reading
Token Authentication for Cached Private Content and APIs
While working to make the Internet a better place, we also want to make it easier for our customers to have control of their content and APIs, and who has access to them. Using Cloudflare’s Token Authentication features, customers can implement access control via URL tokens or HTTP request headers without having to build complex back-end systems.
Cloudflare will check these tokens at the edge before any request is relayed to an origin or served from cache. If the token is not valid the request is blocked. Since Cloudflare handles all the token validation, the origin server does not need to have complex authentication logic. In addition, a malicious user who attempts to forge tokens will be blocked from ever reaching the origin.
Leveraging our edge network of over 100 data centers, customers can use token authentication to perform access control checks on content and APIs, as well as allowing Cloudflare to cache private content and only serve it to users with a valid token tied specifically to that cached asset.
Performing access control on the edge has many benefits. Brute force attempts and other attacks on private assets don't ever reach Continue reading