Alex Krivit

Author Archives: Alex Krivit

Crawler Hints Update: Cloudflare Supports IndexNow and Announces General Availability

Crawler Hints Update: Cloudflare Supports IndexNow and Announces General Availability
Crawler Hints Update: Cloudflare Supports IndexNow and Announces General Availability

In the midst of the hottest summer on record, Cloudflare held its first ever Impact Week. We announced a variety of products and initiatives that aim to make the Internet and our planet a better place, with a focus on environmental, social, and governance projects. Today, we’re excited to share an update on Crawler Hints, an initiative announced during Impact Week. Crawler Hints is a service that improves the operating efficiency of the approximately 45% of Internet traffic that comes from web crawlers and bots.

Crawler Hints achieves this efficiency improvement by ensuring that crawlers get information about what they’ve crawled previously and if it makes sense to crawl a website again.

Today we are excited to announce two updates for Crawler Hints:

  1. The first: Crawler Hints now supports IndexNow, a new protocol that allows websites to notify search engines whenever content on their website content is created, updated, or deleted. By collaborating with Microsoft and Yandex, Cloudflare can help improve the efficiency of their search engine infrastructure, customer origin servers, and the Internet at large.
  2. The second: Crawler Hints is now generally available to all Cloudflare customers for free. Customers can benefit from these more Continue reading

Early Hints: How Cloudflare Can Improve Website Load Times by 30%

Early Hints: How Cloudflare Can Improve Website Load Times by 30%
Early Hints: How Cloudflare Can Improve Website Load Times by 30%

Want to know a secret about Internet performance? Browsers spend an inordinate amount of time twiddling their thumbs waiting to be told what to do. This waiting impacts page load performance. Today, we’re excited to announce support for Early Hints, which dramatically improves browser page load performance and reduces thumb-twiddling time.

In initial tests using Early Hints, we have observed more than 30% improvement to page load time for browsers visiting a website for the first time.

Early Hints is available in beta today — Cloudflare customers can request access to Early Hints in the dashboard’s Speed tab. It’s free for all customers because we think the web should be fast!

Early Hints in a nutshell

Browsers need instructions for what to render and what resources need to be fetched to complete “painting” a given web page. These instructions come from a server response. But the servers sending these responses often need time to compile these resources — this is known as “server think time.” While the servers are busy during this time… browsers sit idle and wait.

Early Hints takes advantage of “server think time” to asynchronously send instructions to the browser to begin loading resources while the Continue reading

Introducing Smart Edge Revalidation

Introducing Smart Edge Revalidation
Introducing Smart Edge Revalidation

Today we’re excited to announce Smart Edge Revalidation. It was designed to ensure that compute resources are synchronized efficiently between our edge and a browser. Right now, as many as 30% of objects cached on Cloudflare’s edge do not have the HTTP response headers required for revalidation. This can result in unnecessary origin calls. Smart Edge Revalidation fixes this: it does the work to ensure that these headers are present, even when an origin doesn’t send them to us. The advantage of this? There’s less wasted bandwidth and compute for objects that do not need to be redownloaded. And there are faster browser page loads for users.

So What Is Revalidation?

Introducing Smart Edge Revalidation

Revalidation is one part of a longer story about efficiently serving objects that live on an origin server from an intermediary cache. Visitors to a website want it to be fast. One foundational way to make sure that a website is fast for visitors is to serve objects from cache. In this way, requests and responses do not need to transit unnecessary parts of the Internet back to an origin and, instead, can be served from a data center that is closer to the visitor. As such, website operators Continue reading

CDN-Cache-Control: Precision Control for your CDN(s)

CDN-Cache-Control: Precision Control for your CDN(s)
CDN-Cache-Control: Precision Control for your CDN(s)

Today we are thrilled to announce our support of a new set of HTTP response headers that provide surgical control over our CDN’s caching decisions. CDN-Cache-Control allows customers to directly control how our CDN behaves without affecting the behavior of downstream or upstream caches.

You might be thinking that this sounds a lot like the Cache-Control header we all know and love. And it’s very similar! CDN-Cache-Control has exactly the same directives as the Cache-Control header. The problem CDN-Cache-Control sets out to solve is that with Cache-Control, some directives are targeted at specific classes of caches (like s-maxage for shared caches), while other directives are not targeted at controlling any specific classes of intermediary caches (think stale-while-revalidate). As these non-specific directives are returned to downstream caches, they’re often not applied uniformly. This problem is amplified as the number of intermediary caches grows between an origin and the client.

For example, a website may deploy a caching layer on the origin server itself, there might be a cache on the origin’s network, the site might use one or more CDNs to cache content distributed throughout the Internet, and the visitor’s browser might cache content as well. As the response returns Continue reading

Introducing: Smarter Tiered Cache Topology Generation

Introducing: Smarter Tiered Cache Topology Generation
Introducing: Smarter Tiered Cache Topology Generation

Caching is a magic trick. Instead of a customer’s origin responding to every request, Cloudflare’s 200+ data centers around the world respond with content that is cached geographically close to visitors. This dramatically improves the load performance for web pages while decreasing the bandwidth costs by having Cloudflare respond to a request with cached content.

However, if content is not in cache, Cloudflare data centers must contact the origin server to receive the content. This isn’t as fast as delivering content from cache. It also places load on an origin server, and is more costly compared to serving directly from cache. These issues can be amplified depending on the geographic distribution of a website’s visitors, the number of data centers contacting the origin, and the available origin resources for responding to requests.

To decrease the number of times our network of data centers communicate with an origin, we organize data centers into tiers so that only upper-tier data centers can request content from an origin and then they spread content to lower tiers. This means content that loads faster for visitors, is cheaper to serve, and reduces origin resource consumption.

Today, I’m thrilled to announce a fundamental improvement to Argo Continue reading

Cloudflare’s Always Online and the Internet Archive Team Up to Fight Origin Errors

Cloudflare’s Always Online and the Internet Archive Team Up to Fight Origin Errors
Cloudflare’s Always Online and the Internet Archive Team Up to Fight Origin Errors

Every day, all across the Internet, something bad but entirely normal happens: thousands of origin servers go down, resulting in connection errors and frustrated users. Cloudflare’s users collectively spend over four and a half years each day waiting for unreachable origin servers to respond with error messages. But visitors don’t want to see error pages, they want to see content!

Today is exciting for all those who want the Internet to be stronger, more resilient, and have important redundancies: Cloudflare is pleased to announce a partnership with the Internet Archive to bring new functionality to our Always Online service.

Always Online serves as insurance for our customers’ websites. Should a customer’s origin go offline, timeout, or otherwise break, Always Online is there to step in and serve archived copies of webpages to visitors. The Internet Archive is a nonprofit organization that runs the Wayback Machine, a service which saves snapshots of billions of websites across the Internet. By partnering with the Internet Archive, Cloudflare is able to seamlessly deliver responses for unreachable websites from the Internet Archive, while the Internet Archive can continue their mission of archiving the web to provide access to all knowledge.

Enabling Always Online in the Continue reading

The Project Jengo Saga: How Cloudflare Stood up to a Patent Troll – and Won!

The Project Jengo Saga: How Cloudflare Stood up to a Patent Troll – and Won!
The Project Jengo Saga: How Cloudflare Stood up to a Patent Troll – and Won!

Remember 2016? Pokemon Go was all the rage, we lost Prince, and there were surprising election results in both the UK and US. Back in 2016, Blackbird Technologies was notorious in the world of patent litigation. It was a boutique law firm that was one of the top ten most active patent trolls, filing lawsuits against more than 50 different defendants in a single year.

In October 2016, Blackbird was looking to acquire additional patents for their portfolio when they found an incredibly broad software patent with the ambiguous title, “PROVIDING AN INTERNET THIRD PARTY DATA CHANNEL.” They acquired this patent from its owner for $1 plus “other good and valuable consideration.” A little later, in March 2017, Blackbird decided to assert that patent against Cloudflare.

As we have explained previously, patent trolls benefit from a problematic incentive structure that allows them to take vague or abstract patents that they have no intention of developing and assert them as broadly as possible. Instead, these trolls collect licensing fees or settlements from companies who are otherwise trying to start a business, produce useful products, and create good jobs. Companies facing such claims usually convince themselves that settlements Continue reading

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Today marks the one year anniversary of Project Jengo, a crowdsourced search for prior art that Cloudflare created and funded in response to the actions of Blackbird Technologies, a notorious patent troll. Blackbird has filed more than one hundred lawsuits asserting dormant patents without engaging in any innovative or commercial activities of its own. In homage to the typical anniversary cliché, we are taking this opportunity to reflect on the last year and confirm that we’re still going strong.

Project Jengo arose from a sense of immense frustration over the way that patent trolls purchase over-broad patents and use aggressive litigation tactics to elicit painful settlements from companies. These trolls know that the system is slanted in their favor, and we wanted to change that. Patent lawsuits take years to reach trial and cost an inordinate sum to defend. Knowing this, trolls just sit back and wait for companies to settle. Instead of perpetuating this cycle, Cloudflare decided to bring the community together and fight back.

After Blackbird filed a lawsuit against Cloudflare alleging infringement of a vague and overly-broad patent (‘335 Patent), we launched Project Jengo, which offered a reward to people who submitted prior art that could Continue reading