Use Snowflake with R2 to extend your global data lake
R2 is the ideal object storage platform to build data lakes. It’s infinitely scalable, highly durable (eleven 9's of annual durability), and has no egress fees. Zero egress fees mean zero vendor lock-in. You are free to use the tools you want to get the maximum value from your data.
Today we’re excited to announce our partnership with Snowflake so that you can use Snowflake to query data stored in your R2 data lake and load data from R2 into Snowflake. Organizations use Snowflake's Data Cloud to unite siloed data, discover, and securely share data, and execute diverse analytic workloads across multiple clouds.
One challenge of loading data into Snowflake database tables and querying external data lakes is the cost of data transfer. If your data is coming from a different cloud or even different region within the same cloud, this typically means you are paying an additional tax for each byte going into Snowflake. Pairing R2 and Snowflake lets you focus on getting valuable insights from your data, without having to worry about egress fees piling up.
Getting started
Sign up for R2 and create an API token
If you haven’t already, you’ll need to sign up for R2 Continue reading
Announcing connect() — a new API for creating TCP sockets from Cloudflare Workers
Today, we are excited to announce a new API in Cloudflare Workers for creating outbound TCP sockets, making it possible to connect directly to any TCP-based service from Workers.
Standard protocols including SSH, MQTT, SMTP, FTP, and IRC are all built on top of TCP. Most importantly, nearly all applications need to connect to databases, and most databases speak TCP. And while Cloudflare D1 works seamlessly on Workers, and some hosted database providers allow connections over HTTP or WebSockets, the vast majority of databases, both relational (SQL) and document-oriented (NoSQL), require clients to connect by opening a direct TCP “socket”, an ongoing two-way connection that is used to send queries and receive data. Now, Workers provides an API for this, the first of many steps to come in allowing you to use any database or infrastructure you choose when building full-stack applications on Workers.
Database drivers, the client code used to connect to databases and execute queries, are already using this new API. pg, the most widely used JavaScript database driver for PostgreSQL, works on Cloudflare Workers today, with more database drivers to come.
The TCP Socket API is available today to everyone. Get started by reading the TCP Continue reading
Zero Trust Security for AI
A collection of tools from Cloudflare One to help your teams use AI services safely
Cloudflare One gives teams of any size the ability to safely use the best tools on the Internet without management headaches or performance challenges. We’re excited to announce Cloudflare One for AI, a new collection of features that help your team build with the latest AI services while still maintaining a Zero Trust security posture.
Large Language Models, Larger Security Challenges
A Large Language Model (LLM), like OpenAI’s GPT or Google’s Bard, consists of a neural network trained against a set of data to predict and generate text based on a prompt. Users can ask questions, solicit feedback, and lean on the service to create output from poetry to Cloudflare Workers applications.
The tools also bear an uncanny resemblance to a real human. As in some real-life personal conversations, oversharing can become a serious problem with these AI services. This risk multiplies due to the types of use cases where LLM models thrive. These tools can help developers solve difficult coding challenges or information workers create succinct reports from a mess of notes. While helpful, every input fed into a prompt becomes a piece of Continue reading
A raft of free Cloudflare services for AI startups
Over the past couple of years, we have piloted a program for early stage startups with free access to a selection of developer products that are high leverage for them. Last year, we launched version 2 of the startup program, which dramatically expanded the basket of products included.
While upgrading startups to the startup plan, I often get inquiries from startups that are fully bootstrapped and not affiliated with any accelerator program. Many of them, especially AI startups, are very promising and would benefit highly from the startup plan.
Typically, they also apply for the Workers Launchpad program, for whom semi-finalists can get upgraded to the startup plan as a benefit. But many of those startups would benefit from getting upgraded right away rather than wait for the review process for each cohort.
Starting today, AI startups no longer need an accelerator affiliation or an employee referral in order to qualify for the Startup Program.
How to get on the startup plan as an AI startup
Here’s what I need you to do if you are a founder of a bootstrapped AI startup. Create a Cloudflare account if you don’t have one, add a domain, and update Continue reading
Introducing Cursor: the Cloudflare AI Assistant
Today we’re excited to be launching Cursor – our experimental AI assistant trained to answer questions about Cloudflare’s Developer Platform. This is just the first step in our journey to help developers build in the fastest way possible using AI, so we wanted to take the opportunity to share our vision for a generative developer experience.
Whenever a new, disruptive technology comes along, it’s not instantly clear what the native way to interact with that technology will be.
However, if you’ve played around with Large Language Models (LLMs) such as ChatGPT, it’s easy to get the feeling that this is something that’s going to change the way we work. The question is: how? While this technology already feels super powerful, today, we’re still in the relatively early days of it.
While Developer Week is all about meeting developers where they are, this is one of the things that’s going to change just that — where developers are, and how they build code. We’re already seeing the beginnings of how the way developers write code is changing, and adapting to them. We wanted to share with you how we’re thinking about it, what’s on the horizon, and some of the large Continue reading
Introducing Constellation, bringing AI to the Cloudflare stack
This post is also available in 简体中文, 日本語, Deutsch and Español.
The Cloudflare Workers' ecosystem now features products and features ranging from compute, hosting, storage, databases, streaming, networking, security, and much more. Over time, we've been trying to inspire others to switch from traditional software architectures, proving and documenting how it's possible to build complex applications that scale globally on top of our stack.
Today, we're excited to welcome Constellation to the Cloudflare stack, enabling developers to run pre-trained machine learning models and inference tasks on Cloudflare's network.
One more building block in our Supercloud
Machine learning and AI have been hot topics lately, but the reality is that we have been using these technologies in our daily lives for years now, even if we do not realize it. Our mobile phones, computers, cars, and home assistants, to name a few examples, all have AI. It's everywhere.
But it isn't a commodity to developers yet, though. They often need to understand the mathematics behind it, the software and tools are dispersed and complex, and the hardware or cloud services to run the frameworks and data are expensive.
Today we're introducing another feature to our stack, allowing everyone to Continue reading
Query Cloudflare Radar and our docs using ChatGPT plugins
When OpenAI launched ChatGPT plugins in alpha we knew that it opened the door for new possibilities for both Cloudflare users and developers building on Cloudflare. After the launch, our team quickly went to work seeing what we could build, and today we’re very excited to share with you two new Cloudflare ChatGPT plugins – the Cloudflare Radar plugin and the Cloudflare Docs plugin.
The Cloudflare Radar plugin allows you to talk to ChatGPT about real-time Internet patterns powered by Cloudflare Radar.
The Cloudflare Docs plugin allows developers to use ChatGPT to help them write and build Cloudflare applications with the most up-to-date information from our documentation. It also serves as an open source example of how to build a ChatGPT plugin with Cloudflare Workers.
Let’s do a deeper dive into how each of these plugins work and how we built them.
Cloudflare Radar ChatGPT plugin
When ChatGPT introduced plugins, one of their use cases was retrieving real-time data from third-party applications and their APIs and letting users ask relevant questions using natural language.
Cloudflare Radar has lots of data about how people use the Internet, a well-documented public API, an OpenAPI specification, and it’s entirely built on Continue reading
Batteries included: how AI will transform the who and how of programming
The 1947 paper titled “Preparation of Problems for EDVAC-Type Machines” talks about the idea and usefulness of a “subroutine”. At the time there were only a tiny number of computers worldwide and subroutines were a novel idea, and it was clear that these subroutines were going to make programmers more productive: “Many operations which are thus excluded from the built-in set are still of sufficiently frequent occurrence to make undesirable the repetition of their coding in detail.”
Looking back it seems amazing that subroutines had to be invented, but at the time programmers wrote literally everything they needed to complete a task. That made programming slow, error-prone and restricted who could be a programmer to a relatively small group of people.
Luckily, things changed.
You can look at the history of computer programming as improvements in programmer productivity and widening the scope of who is a programmer. Think of syntax highlighting, high-level languages, IDEs, libraries and frameworks, APIs, Visual Basic, code completion, refactoring tools, spreadsheets, and so on.
And here we are with things changing again.
The new programmers
The recent arrival of LLMs capable of assisting programmers in writing, debugging and modifying code is yet Continue reading
Welcome to Developer Week 2023
This post is also available in French, Spanish, German, Japanese, Chinese.
It is an incredibly exciting time to be a developer.
The frameworks, libraries and developer tools we depend on keep leveling up in ways that allow us to build more efficiently. On top of that, we’re using AI-powered tools like ChatGPT and GitHub Copilot to ship code quicker than many of us ever could have imagined. This all means we’re spending less time on boilerplate code and setup, and more time writing the code that makes our applications unique.
It’s not only a time when we’re equipped with the tools to be successful in new ways, but we're also finding inspiration in what’s happening around us. It feels like every day there’s an advancement with AI that changes the boundaries of what we can build. Across meetups, conferences, chat rooms, and every other place we gather as developers, we’re pushing each other to expand our ideas of what is possible.
With so much excitement permeating through the global developer community, we couldn’t imagine a better time to be kicking off Developer Week here at Cloudflare.
A focus on developer experience
A big part of any Continue reading
Cloudflare’s view of Internet disruptions in Pakistan
On Tuesday, May 9, Imran Khan, former Prime Minister of Pakistan was arrested on corruption charges. Following the arrest, violent protests erupted in several cities, leading the government of Pakistan to order the shutdown of mobile Internet services, as well as the blocking of several social media platforms. Below, we examine the impact of these shutdowns at a national and local level, as seen through Cloudflare traffic data. In addition, we illustrate how Pakistanis appear to be turning to Cloudflare’s 1.1.1.1 resolver in an attempt to maintain access to the open Internet.
General traffic trends
Since Tuesday, May 9, peak traffic levels aggregated at a country level (as measured by HTTP request volume) have been declining, down nearly 30% during the first several days of the mobile Internet shutdowns. The lowest traffic levels (nadirs of the graph) have also declined, dropping by as much as one-third as well. In the sections below, we drill down into this traffic loss, looking at outages at a network level, and the impact of those outages at an administrative unit and city level.
The mobile network shutdowns have also impacted the profile of traffic that Cloudflare sees from Pakistan. In Continue reading
Magic in minutes: how to build a ChatGPT plugin with Cloudflare Workers
Today, we're open-sourcing our ChatGPT Plugin Quickstart repository for Cloudflare Workers, designed to help you build awesome and versatile plugins for ChatGPT with ease. If you don’t already know, ChatGPT is a conversational AI model from OpenAI which has an uncanny ability to take chat input and generate human-like text responses.
With the recent addition of ChatGPT plugins, developers can create custom extensions and integrations to make ChatGPT even more powerful. Developers can now provide custom flows for ChatGPT to integrate into its conversational workflow – for instance, the ability to look up products when asking questions about shopping, or retrieving information from an API in order to have up-to-date data when working through a problem.
That's why we're super excited to contribute to the growth of ChatGPT plugins with our new Quickstart template. Our goal is to make it possible to build and deploy a new ChatGPT plugin to production in minutes, so developers can focus on creating incredible conversational experiences tailored to their specific needs.
How it works
Our Quickstart is designed to work seamlessly with Cloudflare Workers. Under the hood, it uses our command-line tool wrangler to create a new project and deploy it to Workers.
How Pingora keeps count
A while ago we shared how we replaced NGINX with our in-house proxy, Pingora. We promised to share more technical details as well as our open sourcing plan. This blog post will be the first of a series that shares both the code libraries that power Pingora and the ideas behind them.
Today, we take a look at one of Pingora’s libraries: pingora-limits.
pingora-limits provides the functionality to count inflight events and estimate the rate of events over time. These functions are commonly used to protect infrastructure and services from being overwhelmed by certain types of malicious or misbehaving requests.
For example, when an origin server becomes slow or unresponsive, requests will accumulate on our servers, which adds pressure on both our servers and our customers’ servers. With this library, we are able to identify which origins have issues, so that action can be taken without affecting other traffic.
The problem can be abstracted in a very simple way. The input is a (never ending) stream of different types of events. At any point, the system should be able to tell the number of appearances (or the rate) of a certain type of event.
In a simple example, colors are Continue reading
How the coronation of King Charles III affected Internet traffic
When major events in a country happen Internet traffic patterns are often impacted, depending on the type of event. But what about the coronation of a king or queen? There’s no similar precedent, with a worldwide impact, in the Internet age, except maybe the coronation of the king of Thailand, in 2019. The last time it happened in the United Kingdom was 70 years ago (June 2, 1953), with Queen Elizabeth II; it was the first British coronation to be fully televised. Neither the Internet nor ARPANET were around at the time.
Imagine a grand royal event (if you saw the broadcast or the news, there’s no need), filled with pomp and pageantry, that's so captivating it impacts Internet traffic. That's what happened during the coronation of Charles III and Camilla, the newly crowned king and queen of the United Kingdom and other Commonwealth realms. As the coronation ceremony unfolded, on Saturday morning, May 6, 2023, there were clear spikes and dips in traffic, each coinciding with key moments of the ceremony.
Then came Sunday, and with it, the Coronation Big Lunch event. As the nation sat down to enjoy a communal meal throughout the country, Internet traffic took a Continue reading
Introducing Object Lifecycle Management for Cloudflare R2
Last year, R2 made its debut, providing developers with object storage while eliminating the burden of egress fees. (For many, egress costs account for over half of their object storage bills!) Since R2’s launch, tens of thousands of developers have chosen it to store data for many different types of applications.
But for some applications, data stored in R2 doesn’t need to be retained forever. Over time, as this data grows, it can unnecessarily lead to higher storage costs. Today, we’re excited to announce that Object Lifecycle Management for R2 is generally available, allowing you to effectively manage object expiration, all from the R2 dashboard or via our API.
Object Lifecycle Management
Object lifecycles give you the ability to define rules (up to 1,000) that determine how long objects uploaded to your bucket are kept. For example, by implementing an object lifecycle rule that deletes objects after 30 days, you could automatically delete outdated logs or temporary files. You can also define rules to abort unfinished multipart uploads that are sitting around and contributing to storage costs.
Getting started with object lifecycles in R2
Cloudflare dashboard
- From the Cloudflare dashboard, select R2.
- Select your R2 bucket.
- Navigate to Continue reading
The European Network Usage Fees proposal is about much more than a fight between Big Tech and Big European telcos
There’s an important debate happening in Europe that could affect the future of the Internet. The European Commission is considering new rules for how networks connect to each other on the Internet. It’s considering proposals that – no hyperbole – will slow the Internet for consumers and are dangerous for the Internet.
The large incumbent telcos are complaining loudly to anyone who wants to listen that they aren’t being adequately compensated for the capital investments they’re making. These telcos are a set of previously regulated monopolies who still constitute the largest telcos by revenue in Europe in today's competitive market. They say traffic volumes, largely due to video streaming, are growing rapidly, implying they need to make capital investments to keep up. And they call for new charges on big US tech companies: a “fair share” contribution that those networks should make to European Internet infrastructure investment.
In response to this campaign, in February the European Commission released a set of recommended actions and proposals “aimed to make Gigabit connectivity available to all citizens and businesses across the EU by 2030.” The Commission goes on to say that “Reliable, fast and secure connectivity is a must for everybody and Continue reading
Cloudflare is faster than Netskope and Zscaler across LATAM
This post is also available in Español and Português.
Last CIO Week, we showed you how our network stacks up against competitors across several countries. We demonstrated with our tests that Cloudflare Access is 38% faster than ZScaler (ZPA) worldwide.
Today we wanted to focus on LATAM and show how our network performed against Zscaler and Netskope in Argentina, Brazil, Chile, Colombia, Costa Rica, Ecuador, Mexico, Peru, Uruguay and Venezuela.
With 47 data centers across Latin America and Caribbean, Cloudflare has the largest number of SASE Points of Presence across all vendors, meaning we can offer our Zero Trust services closer to the end user and reduce unwanted latency.
We’ve run a series of tests comparing our Zero Trust Network Access product against Zscaler and Netskope’s comparable products.
For each of these tests, we used 95th percentile Time to First Byte and Response tests, which measure the time it takes for a user to make a request, and get the start of the response (Time to First Byte), and the end of the response (Response). These tests were designed with the goal of trying to measure performance from an end-user perspective.
In this blog we’re going to talk about Continue reading
How we built Network Analytics v2
Network Analytics v2 is a fundamental redesign of the backend systems that provide real-time visibility into network layer traffic patterns for Magic Transit and Spectrum customers. In this blog post, we'll dive into the technical details behind this redesign and discuss some of the more interesting aspects of the new system.
To protect Cloudflare and our customers against Distributed Denial of Service (DDoS) attacks, we operate a sophisticated in-house DDoS detection and mitigation system called dosd. It takes samples of incoming packets, analyzes them for attacks, and then deploys mitigation rules to our global network which drop any packets matching specific attack fingerprints. For example, a simple network layer mitigation rule might say “drop UDP/53 packets containing responses to DNS ANY queries”.
In order to give our Magic Transit and Spectrum customers insight into the mitigation rules that we apply to their traffic, we introduced a new reporting system called "Network Analytics" back in 2020. Network Analytics is a data pipeline that analyzes raw packet samples from the Cloudflare global network. At a high level, the analysis process involves trying to match each packet sample against the list of mitigation rules that dosd has deployed, so that it can Continue reading