Crawler Hints: How Cloudflare Is Reducing The Environmental Impact Of Web Searches
Cloudflare is known for innovation, for needle-moving projects that help make the Internet better. For Impact Week, we wanted to take this approach to innovation and apply it to the environmental impact of the Internet. When it comes to tech and the environment, it’s often assumed that the only avenue tech has open to it is harm mitigation: for example, climate credits, carbon offsets, and the like. These are undoubtedly important steps, but we wanted to take it further — to get into harm reduction. So we asked — how can the Internet at large use less energy and be more thoughtful about how we expend computing resources in the first place?
Cloudflare has a global view into the traffic of the Internet. More than 1 in 6 websites use our network, and we observe the traffic flowing to and from them continuously. While most people think of surfing the Internet as a very human activity, nearly half of all traffic on the global network is generated by automated systems.
We've analyzed this automated traffic, from so-called “bots,” in order to understand the environmental impact. Most of the bot traffic is malicious. Cloudflare protects our clients from this malicious traffic Continue reading
Introducing Smart Edge Revalidation
Today we’re excited to announce Smart Edge Revalidation. It was designed to ensure that compute resources are synchronized efficiently between our edge and a browser. Right now, as many as 30% of objects cached on Cloudflare’s edge do not have the HTTP response headers required for revalidation. This can result in unnecessary origin calls. Smart Edge Revalidation fixes this: it does the work to ensure that these headers are present, even when an origin doesn’t send them to us. The advantage of this? There’s less wasted bandwidth and compute for objects that do not need to be redownloaded. And there are faster browser page loads for users.
So What Is Revalidation?
Revalidation is one part of a longer story about efficiently serving objects that live on an origin server from an intermediary cache. Visitors to a website want it to be fast. One foundational way to make sure that a website is fast for visitors is to serve objects from cache. In this way, requests and responses do not need to transit unnecessary parts of the Internet back to an origin and, instead, can be served from a data center that is closer to the visitor. As such, website operators Continue reading
LISP – OMP – BGP EVPN Interoperability – Part III: SD-WAN Control Plane
Introduction
Figure 2-1 shows the Control Plane operation when host EP1 using IP 172.16.100.10/32 joins the network. In the previous chapter, we saw how Edge-xTR-11 learned the IP address and registered it to MapSrv-22 by using the LISP Map-register Message where the Instance-Id 100 represents Virtual Network-Id (VN-Id). MapSrv-22, in turn, advertised the NLRI to Border-PxTR-13 as BGP VPNv4 Update message where extended community RT 1:100 (=VN-Id) is used as a kind of VPN identifier (BGP VPNv4 route import/export policy is based on RT value). This chapter first explains how Border-PxTR-13 sends BGP IPv4 Update message to local SD-WAN device vEdge-1. The eBGP peering between Border-PxTR-13 and vEdge1 is VRF based and BGP updates over it don’t carry any VN-Id. vEdge-1 imports the routing information from BRIB to RIB. Then it advertises the routing information by using OMP (Overlay Management Protocol) to the SD-WAN centralized Control Plane vSmart over the DTLS tunnel using System-IP as an originator-Id and VPN label 1003 as a VN-Id. vSmart forwards this update to SD-WAN device vEdge-2 located in Datacenter. vEdge-2 installs the route into the routing table. After that, it exports the routing information from the RIB to the BGP process and sends the BGP IPv4 Update message to Border-Leaf-13 over VRF NWKT eBGP peering without VN-Id.
Figure 2-1: Overall Control-Plane Operation.
Continue reading
Network Break Quick Take – New Video Series
New video series Network Break Quick Take
Crystal Notes: Arrays
Arrays in Crystal are an ordered collection of elements, generally of the same type. Although, it is possible to have an array of multiple types. Creating an array Iterating an Array Accessing Elements Array Operations Static Arrays If the size of an array is fixed it is much more...continue reading
Crystal Notes: Variables
Like most languages, variables in Crystal are defined with the = operator. Considerations Variable types are inferred by the compiler and do not have to be specifically defined. When a variable type is declared, it must be assigned a value before it can be accessed. There are no global...continue reading
Crystal Notes: Hashes
Hashes in Crystal are a collection of key/value pairs of defined types. Creating a hash # Deduce the type signature of a hash. typeof(stuff_and_things) # => Hash(String, String) # To create an empty hash, you must define # the intended type of its key/value pairs. # There are 2 ways to define...continue reading
Do you really need a service mesh?
The challenges involved in deploying and managing microservices have led to the creation of the service mesh, a tool for adding observability, security, and traffic management capabilities at the application layer. While a service mesh is intended to help developers and SREs with a number of use cases related to service-to-service communication within Kubernetes clusters, a service mesh also adds operational complexity and introduces an additional control plane for security teams to manage.
What is a service mesh?
A service mesh is a software infrastructure layer for controlling and monitoring internal, service-to-service traffic in microservices applications.
Service mesh provides some of the middleware and some of the components that enable service-to-service communication, such as dynamic discovery. It provides capabilities around service discovery, load balancing traffic across services, security features around encryption and authentication, tracing observability, and more. The service mesh architecture leverages design patterns to enable communication between services without requiring microservices to rewrite applications.
Service mesh architecture
One of the key aspects of how a service mesh works is that it leverages a sidecar design pattern. Services communicate and handle requests via a proxy, which is dynamically injected into each pod. Envoy is one of the most popular proxies Continue reading
Upcoming Live Webinar: Data Center Fabrics
I’ll be teaching a three-hour live webinar on data center fabrics on the 20th of August—
Data centers are the foundation of the cloud, whether private, public, on the edge, or in the center of the network. This training will focus on topologies and control planes, including scale, performance, and centralization. This training is important for network designers and operators who want to understand the elements of data center design that apply across all hardware and software types.
Mitigating the Risks of Multi-Cloud Environments
New security challenges emerge as more organizations launch or expand their multi-cloud strategies as part of their digital transformation initiatives.Helping build a green Internet
When we started Cloudflare, we weren't thinking about minimizing the environmental impact of the Internet. Frankly, I didn't really think of the Internet as having much of an environmental impact. It was just this magical resource that gave access to information and services from anywhere.
But that was before I started racking servers in hyper-cooled data centers. Before Cloudflare started paying the bills to keep those servers powered up and cooled down. Before we became obsessed with maximizing the number of requests we could process per watt of power. And long before we started buying directly from renewable power suppliers to drive down the cost of electricity across our network.
Today, I have a very good understanding of how much power it takes to run the Internet. It therefore wasn't surprising to read the Boston Consulting Group study which found that 2% of all carbon output, about 1 billion metric tons per year, is attributable to the Internet. That’s the equivalent of the entire aviation industry.
Cloudflare: Accidentally Environmentally Friendly By Design
While we didn't set out to reduce the environmental impact of the Internet, Cloudflare has always had efficiency at its core. It comes from our ongoing fight with Continue reading
Announcing Green Compute on Cloudflare Workers
All too often we are confronted with the choice to move quickly or act responsibly. Whether the topic is safety, security, or in this case sustainability, we’re asked to make the trade off of halting innovation to protect ourselves, our users, or the planet. But what if that didn’t always need to be the case? At Cloudflare, our goal is to bring sustainable computing to you without the need for any additional time, work, or complexity.
Enter Green Compute on Cloudflare Workers.
Green Compute can be enabled for any Cron triggered Workers. The concept is simple: when turned on, we’ll take your compute workload and run it exclusively on parts of our edge network located in facilities powered by renewable energy. Even though all of Cloudflare’s edge network is powered by renewable energy already, some of our data centers are located in third-party facilities that are not 100% powered by renewable energy. Green Compute takes our commitment to sustainability one step further by ensuring that not only our network equipment but also the building facility as a whole are powered by renewable energy. There are absolutely no code changes needed. Now, whether you need to update a leaderboard every five Continue reading