Sudan’s exam-related Internet shutdowns

To prevent cheating in exams many countries restrict or even shut down Internet access during critical exam hours. I wrote two weeks ago about Syria having planned Internet shutdowns during June, for exams.

Sudan is doing the same thing and has had four shutdowns so far. Here's the Internet traffic pattern for Sudan over the last seven days. I've circled the shutdowns on Saturday, Sunday, Monday and Tuesday (today, June 22, 2021).

Cloudflare Radar allows anyone to track Internet traffic patterns around the world, and it has country-specific pages. The chart for the last seven days (shown above) came from the dedicated page for Sudan.

The Internet outages start at 0600 UTC (0800 local time) and end three hours later at 0900 UTC (1100 local time). This corresponds to the timings announced by the Sudanese Ministry of Education.

Further shutdowns are likely in Sudan on June 24, 26, 27, 29 and 30 (thanks to Twitter user _adonese for his assistance). Looking deeper into the data, the largest drop in use is for mobile Internet access in Sudan (the message above talks about mobile Internet use being restricted) while some non-mobile access appears to continue.

That can be seen by looking Continue reading

Syria’s exam-related Internet shutdowns

To prevent cheating in exams many countries restrict or even shut down Internet access during critical exam hours. For most of June Syria is having planned Internet shutdowns during critical exam periods. The exam schedule is as follows:

I’m grateful to a Twitter user for the translation from the original Arabic and collating the data.

Cloudflare Radar allows anyone to track Internet traffic patterns around the world, and it has country-specific pages. The chart for the last seven days of Internet use in Syria as seen by Cloudflare shows two drops to almost zero corresponding to the first two exams on the schedule.

The Internet outage starts at around 0100 UTC (0400 local time) and ends about four and a half hours later at 0530 UTC (0830 UTC). This covers the period before the exams start apparently to prevent any figuring out the answers.

If you want to follow the other outages for the remaining seven exams you can see live data on the Cloudflare Radar Syria page.

SSHing to my Raspberry Pi 400 from a browser, with Cloudflare Tunnel and Auditable Terminal

A few weeks ago I received a Raspberry Pi 400 as a gift. I didn’t have time to do anything beyond plug it in and verify that it works. It’s great that the Pi 400 comes with everything you need except for a screen: there’s the computer itself, mouse, HDMI cable and power adapter.

The Pi 400 has been sitting gathering dust when Cloudflare launched Auditable Terminal giving me the perfect excuse to get out the Pi 400 and hook it up.

Auditable Terminal gives you a fully featured SSH client in your browser. You authenticate using Cloudflare Access and can log into a computer from anywhere just using the browser and get a terminal. And using Cloudflare Tunnel you can securely connect a computer to Cloudflare without punching holes in a firewall. And you end up with a consistent terminal experience across devices: 256 colours, Unicode support and the same fonts everywhere.

This is ideal for my use case: set up the Pi 400 on my home network, use Cloudflare Tunnel to connect it to the Cloudflare network, use Auditable Terminal to connect to the Pi 400 via Cloudflare and the tunnel using nothing more than a browser.

Here’s Continue reading

Bringing AI to the edge with NVIDIA GPUs

Cloudflare has long used machine learning for bot detection, identifying anomalies, customer support and business intelligence.  And internally we have a cluster of GPUs used for model training and inference.

For even longer we’ve been running code “at the edge” in more than 200 cities worldwide. Initially, that was code that we wrote and any customization was done through our UI or API. About seven years ago we started deploying custom code, written in Lua, for our enterprise customers.

But it’s quite obvious that using a language that isn’t widely understood, and going through an account executive to get code written, isn’t a viable solution and so four years ago we announced Cloudflare Workers. Workers allows anyone, on any plan, to write code that gets deployed to our edge network. And they can do it in the language they choose.

After launching Workers we added storage through Workers KV as programs need algorithms plus data. And we’ve continued to add to the Workers platform with Workers Unbound, Durable Objects, Jurisdictional Restrictions and more.

But many of today’s applications need access to the latest machine learning and deep learning methods. Those applications need three things: to scale easily, Continue reading

Developers, Developers, Developers: Welcome to Developer Week 2021

Runtimes, serverless, edge compute, containers, virtual machines, functions, pods, virtualenv. All names for things developers need to go from writing code to running code. It’s a painful reality that for most developers going from code they’ve written to code that actually runs can be hard.

Excruciatingly, software development is made hard by dependencies on modules, by scaling, by security, by cost, by availability, by deployment, by builds, and on and on. All the ugly reality of crystallizing thoughts into lines of code that actually run, successfully, somewhere, more than once, non-stop, and at scale.

And so… Welcome to Developer Week 2021!

Like we have done in previous Innovation Weeks (such as Security Week or Privacy Week), we will be making many (about 20) announcements of products and features to make developers’ lives easier. And by easy I mean removing the obstacles that stop you, dear developer, from writing code and deploying it so it scales to Internet size.

And Cloudflare Workers, our platform for software developers who want to deploy Internet-facing applications that start instantly and scale Internetly, has been around since 2017 (or to put it in perspective, since iPhone 8) and helping developers code and deploy in seconds Continue reading

About the March 8 & 9, 2021 Verkada camera hack

Cloudflare uses a vendor called Verkada for cameras in our offices in San Francisco, Austin, New York, London and Singapore. These cameras are used at the entrances, exits and main thoroughfares of our offices and have been part of maintaining the security of offices that have been closed for almost a year.

Yesterday, we were notified of a breach of Verkada that allowed a hacker to access Verkada’s internal support tools to manage those cameras remotely, as well as access them through a remote root shell. As soon as we were notified of the breach, we proceeded to shut down the cameras in all our office locations to prevent further access.

To be clear: this hack affected the cameras and nothing else. No customer data was accessed, no production systems, no databases, no encryption keys, nothing. Some press reports indicate that we use a facial recognition feature available in Verkada. This is not true. We do not.

Our internal systems follow the same Zero Trust model that we provide to our customers, and as such our corporate office networks are not implicitly trusted by our other locations or data centers. From a security point of view connecting from one of Continue reading

Creating serendipity with Python

We've been experimenting with breaking up employees into random groups (of size 4) and setting up video hangouts between them. We're doing this to replace the serendipitous meetings that sometimes occur around coffee machines, in lunch lines or while waiting for the printer. And also, we just want people to get to know each other.

Which lead to me writing some code. The core of which is divide n elements into groups of at least size g minimizing the size of each group. So, suppose an office has 15 employees in it then it would be divided into three groups of sizes 5, 5, 5; if an office had 16 employees it would be 4, 4, 4, 4; if it had 17 employees it would be 4, 4, 4, 5 and so on.

I initially wrote the following code (in Python):

    groups = [g] * (n//g)

    for e in range(0, n % g):
        groups[e % len(groups)] += 1

The first line creates n//g (// is integer division) entries of size g (for example, if g == 4 and n == 17 then groups == [4, 4, 4, 4]). The for loop deals with the 'left over' parts that Continue reading

Who won Super Bowl LV? A look at Internet traffic during the game

The obvious answer is the Tampa Bay Buccaneers but the less obvious answer comes from asking “which Super Bowl advertiser got the biggest Internet bump?”. This blog aims to answer that question.

Before, during, and after the game a crack team of three people who work on Cloudflare Radar looked at real time statistics for traffic to advertisers’ websites, social media in the US, US food delivery services, and websites covering (American) football. Luckily, one of us (Kari) is (a) American and (b) a fan of football. Unluckily, one of us (Kari) is a fan of the Kansas City Chiefs.

Cloudflare Radar uses a variety of sources to provide aggregate information about Internet traffic and attack trends. In this blog post we use DNS name resolution data to estimate traffic to websites. We can’t see who visited the websites mentioned below, or what anyone did on the websites, but DNS can give us an estimate of the interest generated by the commercials. This analysis only looked at the top-level names in each domain (so example.com and www.example.com and not any other subdomains).

The Big Picture

To get the ball rolling here’s a look at traffic to Continue reading

Cloudflare Radar’s 2020 Year In Review

Throughout 2020, we tracked changing Internet trends as the SARS-Cov-2 pandemic forced us all to change the way we were living, working, exercising and learning. In early April, we created a dedicated website https://builtforthis.net/ that showed some of the ways in which Internet use had changed, suddenly, because of the crisis.

On that website, we showed how traffic patterns had changed; for example, where people accessed the Internet from, how usage had jumped up dramatically, and how Internet attacks continued unabated and ultimately increased.

Today we are launching a dedicated Year In Review page with interactive maps and charts you can use to explore what changed on the Internet in 2020. Year In Review is part of Cloudflare Radar. We launched Radar in September 2020 to give anyone access to Internet use and abuse trends that Cloudflare normally had reserved only for employees.

Where people accessed the Internet

To get a sense for the Year In Review, let’s zoom in on London (you can do the same with any city from a long list of locations that we’ve analyzed). Here’s a map showing the change in Internet use comparing April (post-lockdown) and February (pre-lockdown). This map compares working hours Continue reading

Internet traffic disruption caused by the Christmas Day bombing in Nashville

On Christmas Day 2020, an apparent suicide bomb exploded in Nashville, TN. The explosion happened outside an AT&T network building on Second Avenue in Nashville at 1230 UTC. Damage to the AT&T building and its power supply and generators quickly caused an outage for telephone and Internet service for local people. These outages continued for two days.

Looking at traffic flow data for AT&T in the Nashville area to Cloudflare we can see that services continued operating (on battery power according to reports) for over five hours after the explosion, but at 1748 UTC we saw a dramatic drop in traffic. 1748 UTC is close to noon in Nashville when reports indicate that people lost phone and Internet service.

We saw traffic from Nashville via AT&T start to recover over a 45 minute period on December 27 at 1822 UTC making the total outage 2 days and 34 minutes.

Traffic flows continue to be normal and no further disruption has been seen.

Privacy needs to be built into the Internet

The first phase of the Internet lasted until the early 1990s. During that time it was created and debugged, and grew globally. Its growth was not hampered by concerns about data security or privacy. Until the 1990s the race was for connectivity.

Connectivity meant that people could get online and use the Internet wherever they were. Because the “inter” in Internet implied interoperability the network was able to grow rapidly using a variety of technologies. Think dialup modems using ordinary phones lines, cable modems sending the Internet over coax originally designed for television, Ethernet, and, later, fibre optic connections and WiFi.

By the 1990s, the Internet was being used widely and for uses far beyond its academic origins. Early web pioneers, like Netscape, realized that the potential for e-commerce was gigantic but would be held back if people couldn’t have confidence in the security of online transactions.

Thus, with the introduction of SSL in 1994, the Internet moved to a second phase where security became paramount. Securing the web, and the Internet more generally, helped create the dotcom rush and the secure, online world we live in today. But this security was misunderstood by some as providing guarantees about privacy Continue reading

Introducing the Cloudflare Data Localization Suite

Today we’re excited to announce the Cloudflare Data Localization Suite, which helps businesses get the performance and security benefits of Cloudflare’s global network, while making it easy to set rules and controls at the edge about where their data is stored and protected.

The Data Localization Suite is available now as an add-on for Enterprise customers.

Cloudflare’s network is private and compliant by design. Preserving end-user privacy is core to our mission of helping to build a better Internet; we’ve never sold personal data about customers or end users of our network. We comply with laws like GDPR and maintain certifications such as ISO-27001.

Today, we're announcing tools that make it simple for our customers to build the same rigor into their own applications. In this post, I'll explain the different types of data that we process and how the Data Localization Suite keeps this data local.

We’ll also talk about how Cloudflare makes it possible to build applications that comply with data locality laws, while remaining fast, secure and scalable.

Why keep data local?

Cloudflare's customers have increasing desire or face legal requirements for data locality: they want to control the geographic location where their data is handled. Continue reading

Welcome to Birthday Week 2020

Each year we celebrate our launch on September 27, 2010 with a week of product announcements. We call this Birthday Week, but rather than receiving gifts, we give them away. This year is no different, except that it is… Cloudflare is 10 years old.

Before looking forward to the coming week, let’s take a look back at announcements from previous Birthday Weeks.

A year into Cloudflare’s life (in 2011) we launched automatic support for IPv6. This was the first of a long line of announcements that support our goal of making available to everyone the latest technologies. If you’ve been following Cloudflare’s growth you’ll know those include SPDY/HTTP/2, TLS 1.3, QUIC/HTTP/3, DoH and DoT, WebP, … At two years old we celebrated with a timeline of our first two years and the fact that we’d reached 500,000 domains using the service. A year later that number had tripled.

In 2014 we released Universal SSL and gave all our customers SSL certificates. In one go we massively increased the size of the encrypted web and made it free and simple to go from http:// to https://. Other HTTPS related features we’ve Continue reading

Cloudflare outage on July 17, 2020

Today a configuration error in our backbone network caused an outage for Internet properties and Cloudflare services that lasted 27 minutes. We saw traffic drop by about 50% across our network. Because of the architecture of our backbone this outage didn’t affect the entire Cloudflare network and was localized to certain geographies.

The outage occurred because, while working on an unrelated issue with a segment of the backbone from Newark to Chicago, our network engineering team updated the configuration on a router in Atlanta to alleviate congestion. This configuration contained an error that caused all traffic across our backbone to be sent to Atlanta. This quickly overwhelmed the Atlanta router and caused Cloudflare network locations connected to the backbone to fail.

The affected locations were San Jose, Dallas, Seattle, Los Angeles, Chicago, Washington, DC, Richmond, Newark, Atlanta, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, St. Petersburg, São Paulo, Curitiba, and Porto Alegre. Other locations continued to operate normally.

For the avoidance of doubt: this was not caused by an attack or breach of any kind.

We are sorry for this outage and have already made a global change to the backbone configuration that will prevent it from being able to occur Continue reading

Cloudflare’s first year in Lisbon

A year ago I wrote about the opening of Cloudflare’s office in Lisbon, it’s hard to believe that a year has flown by. At the time I wrote:

Lisbon’s combination of a large and growing existing tech ecosystem, attractive immigration policy, political stability, high standard of living, as well as logistical factors like time zone (the same as the UK) and direct flights to San Francisco made it the clear winner.

We landed in Lisbon with a small team of transplants from other Cloudflare offices. Twelve of us moved from the UK, US and Singapore to bootstrap here. Today we are 35 people with another 10 having accepted offers; we’ve almost quadrupled in a year and we intend to keep growing to around 80 by the end of 2020.

If you read back to my description of why we chose Lisbon only one item hasn’t turned out quite as we expected. Sure enough TAP Portugal does have direct flights to San Francisco but the pandemic put an end to all business flying worldwide for Cloudflare. We all look forward to getting back to being able to visit our colleagues in other locations.

The pandemic also put us in the Continue reading

When people pause the Internet goes quiet

Recent news about the Internet has mostly been about the great increase in usage as those workers who can have been told to work from home. I've written about this twice recently, first in early March and then last week look at how Internet use has risen to a new normal.

As human behaviour has changed in response to the pandemic, it's left a mark on the charts that network operators look at day in, day out to ensure that their networks are running correctly.

Most Internet traffic has a fairly simple rhythm to it. Here, for example, is daily traffic seen on the Amsterdam Internet Exchange. It's a pattern that's familiar to most network operators. People sleep at night, and there's a peak of usage in the early evening when people get home and perhaps stream a movie, or listen to music or use the web for things they couldn't do during the workday.

But sometimes that rhythm get broken. Recently we've seen the evening peak by joined by morning peaks as well. Here's a graph from the Milan Internet Exchange. There are three peaks: morning, afternoon and evening.  These peaks seem to be caused by people working from Continue reading

Internet performance during the COVID-19 emergency

A month ago I wrote about changes in Internet traffic caused by the COVID-19 emergency. At the time I wrote:

Cloudflare is watching carefully as Internet traffic patterns around the world alter as people alter their daily lives through home-working, cordon sanitaire, and social distancing. None of these traffic changes raise any concern for us. Cloudflare's network is well provisioned to handle significant spikes in traffic. We have not seen, and do not anticipate, any impact on our network's performance, reliability, or security globally.

That holds true today; our network is performing as expected under increased load. Overall the Internet has shown that it was built for this: designed to handle huge changes in traffic, outages, and a changing mix of use. As we are well into April I thought it was time for an update.

Growth

Here's a chart showing the relative change in Internet use as seen by Cloudflare since the beginning of the year. I've calculated moving average of the trailing seven days for each country and use December 29, 2019 as the reference point.

On this chart the highest growth in Internet use has been in Portugal: it's currently running at about a 50% increase Continue reading

Cloudflare Workers Now Support COBOL

Recently, COBOL has been in the news as the State of New Jersey has asked for help with a COBOL-based system for unemployment claims. The system has come under heavy load because of the societal effects of the SARS-CoV-2 virus. This appears to have prompted IBM to offer free online COBOL training.

As old as COBOL is (60 years old this month), it is still heavily used in information management systems and pretty much anywhere there’s an IBM mainframe around. Three years ago Thomson Reuters reported that COBOL is used in 43% of banking systems, is behind 80% of in-person financial transactions and 95% of times an ATM card is used. They also reported 100s of billions of lines of running COBOL.

COBOL is often a source of amusement for programmers because it is seen as old, verbose, clunky, and difficult to maintain. And it’s often the case that people making the jokes have never actually written any COBOL. We plan to give them a chance: COBOL can now be used to write code for Cloudflare’s serverless platform Workers.

Here’s a simple “Hello, World!” program written in COBOL and accessible at https://hello-world.cobol.workers.dev/. It doesn’t do much--it Continue reading

Cloudflare Dashboard and API Outage on April 15, 2020

Starting at 1531 UTC and lasting until 1952 UTC, the Cloudflare Dashboard and API were unavailable because of the disconnection of multiple, redundant fibre connections from one of our two core data centers.

This outage was not caused by a DDoS attack, or related to traffic increases caused by the COVID-19 crisis. Nor was it caused by any malfunction of software or hardware, or any misconfiguration.

What happened

As part of planned maintenance at one of our core data centers, we instructed technicians to remove all the equipment in one of our cabinets. That cabinet contained old inactive equipment we were going to retire and had no active traffic or data on any of the servers in the cabinet. The cabinet also contained a patch panel (switchboard of cables) providing all external connectivity to other Cloudflare data centers. Over the space of three minutes, the technician decommissioning our unused hardware also disconnected the cables in this patch panel.

This data center houses Cloudflare’s main control plane and database and as such, when we lost connectivity, the Dashboard and API became unavailable immediately. The Cloudflare network itself continued to operate normally and proxied customer websites and applications continued to operate. As Continue reading

Announcing the Results of the 1.1.1.1 Public DNS Resolver Privacy Examination

On April 1, 2018, we took a big step toward improving Internet privacy and security with the launch of the 1.1.1.1 public DNS resolver — the Internet's fastest, privacy-first public DNS resolver. And we really meant privacy first. We were not satisfied with the status quo and believed that secure DNS resolution with transparent privacy practices should be the new normal. So we committed to our public resolver users that we would not retain any personal data about requests made using our 1.1.1.1 resolver. We also built in technical measures to facilitate DNS over HTTPS to help keep your DNS queries secure. We’ve never wanted to know what individuals do on the Internet, and we took technical steps to ensure we can’t know.

We knew there would be skeptics. Many consumers believe that if they aren’t paying for a product, then they are the product. We don’t believe that has to be the case. So we committed to retaining a Big 4 accounting firm to perform an examination of our 1.1.1.1 resolver privacy commitments.

Today we’re excited to announce that the 1.1.1.1 resolver examination has been completed Continue reading