Performance isolation in a multi-tenant database environment

Performance isolation in a multi-tenant database environment
Performance isolation in a multi-tenant database environment

Operating at Cloudflare scale means that across the technology stack we spend a great deal of time handling different load conditions. In this blog post we talk about how we solved performance difficulties with our Postgres clusters. These clusters support a large number of tenants and highly variable load conditions leading to the need to isolate activity to prevent tenants taking too much time from others. Welcome to real-world, large database cluster management!

As an intern at Cloudflare I got to work on improving how our database clusters behave under load and open source the resulting code.

Cloudflare operates production Postgres clusters across multiple regions in data centers. Some of our earliest service offerings, such as our DNS Resolver, Firewall, and DDoS Protection, depend on our Postgres clusters' high availability for OLTP workloads. The high availability cluster manager, Stolon, is employed across all clusters to independently control and replicate data across Postgres instances and elect Postgres leaders and failover under high load scenarios.

PgBouncer and HAProxy act as the gateway layer in each cluster. Each tenant acquires client-side connections from PgBouncer instead of Postgres directly. PgBouncer holds a pool of maximum server-side connections to Postgres, allocating those across multiple Continue reading

LastPass breach – Deja Vu all over again

Yesterday I got this notification from my Password Manager, LastPass: The key phrase here is “unusual activity within portions of the LastPass development environment.” This looked too familiar. One of the most well documented attacks of this nature is the SolarWinds supply chain attack in 2020. Although it is too early to say (the news … Continue reading LastPass breach – Deja Vu all over again →

Open sourcing our fork of PgBouncer

Open sourcing our fork of PgBouncer
Open sourcing our fork of PgBouncer

Cloudflare operates highly available Postgres production clusters across multiple data centers, supporting the transactional workloads of our core service offerings such as our DNS Resolver, Firewall, and DDoS Protection.

Multiple PgBouncer instances sit at the front of the gateway layer per each cluster, acting as a TCP proxy that provides Postgres connection pooling. PgBouncer’s pooling enables upstream applications to connect to Postgres, without having to constantly open and close connections (expensive) at the database level, while also reducing the number of Postgres connections used. Each tenant acquires client-side connections from PgBouncer instead of Postgres directly.

Open sourcing our fork of PgBouncer

PgBouncer will hold a pool of maximum server-side connections to Postgres, allocating those across multiple tenants to prevent Postgres connection starvation. From here, PgBouncer will forward backend queries to HAProxy, which load balances across Postgres primary and read replicas.

As an intern at Cloudflare I got to work on improving how our database clusters behave under load and open source the resulting code.

We run our Postgres infrastructure in non-containerized, bare metal environments which consequently leads to multitenant resource contention between Postgres users. To enforce stricter tenant performance isolation at the database level (CPU time utilized, memory consumption, disk IO operations), we’d like to configure Continue reading

Qualcomm planning ARM-based server processor?

Qualcomm may be preparing for another run that the data-center market with a new line of Arm-based processors for servers, according Bloomberg.The company is reportedly seeking customers to test a product from Nuvia, a semiconductor startup it purchased last year that was founded by the former head of Apple CPU development. [ Get regularly scheduled insights by signing up for Network World newsletters. ]To read this article in full, please click here

Network availability: Are you your own worst enemy?

My early enterprise surveys from 30 years ago showed that the largest reported source of network outages was human error. Today, that’s still the case, and in fact human error leads any equipment or transmission cause by a larger margin today than it did 30 years ago. This, despite the fact that enterprises say they’ve invested significantly in improving, simplifying, and automating network operations. The old saying, “We have met the enemy and they are us,” sure seems to apply.If you ask network operations professionals, most will tell you that the problem is that network complexity is growing faster than operations management can cope with. Most, but not all. Operations management believes that acquisition and retention of qualified network experts is a big part of the problem. Some technical pundits think network technology itself is to blame. Almost everyone things that more automation is the solution, but some wonder if our automation tools are just adding another layer of complexity when complexity is the big problem to start with. Hot news: They’re all correct.To read this article in full, please click here

Qualcomm planning ARM-based server processor?

Qualcomm may be preparing for another run that the data-center market with a new line of Arm-based processors for servers, according Bloomberg.The company is reportedly seeking customers to test a product from Nuvia, a semiconductor startup it purchased last year that was founded by the former head of Apple CPU development. [ Get regularly scheduled insights by signing up for Network World newsletters. ]To read this article in full, please click here

Qualcomm planning ARM-based server processor?

Qualcomm may be preparing for another run that the data-center market with a new line of Arm-based processors for servers, according Bloomberg.The company is reportedly seeking customers to test a product from Nuvia, a semiconductor startup it purchased last year that was founded by the former head of Apple CPU development. [ Get regularly scheduled insights by signing up for Network World newsletters. ]To read this article in full, please click here

Qualcomm planning ARM-based server processor?

Qualcomm may be preparing for another run that the data-center market with a new line of Arm-based processors for servers, according Bloomberg.The company is reportedly seeking customers to test a product from Nuvia, a semiconductor startup it purchased last year that was founded by the former head of Apple CPU development. [ Get regularly scheduled insights by signing up for Network World newsletters. ]To read this article in full, please click here

Qualcomm may be planning an ARM-based server processor (again)

Qualcomm may be preparing for another run that the data-center market with a new line of Arm-based processors for servers, according Bloomberg.The company is reportedly seeking customers to test a product from Nuvia, a semiconductor startup it purchased last year that was founded by the former head of Apple CPU development. [ Get regularly scheduled insights by signing up for Network World newsletters. ]To read this article in full, please click here

Network availability: Are you your own worst enemy?

My early enterprise surveys from 30 years ago showed that the largest reported source of network outages was human error. Today, that’s still the case, and in fact human error leads any equipment or transmission cause by a larger margin today than it did 30 years ago. This, despite the fact that enterprises say they’ve invested significantly in improving, simplifying, and automating network operations. The old saying, “We have met the enemy and they are us,” sure seems to apply.If you ask network operations professionals, most will tell you that the problem is that network complexity is growing faster than operations management can cope with. Most, but not all. Operations management believes that acquisition and retention of qualified network experts is a big part of the problem. Some technical pundits think network technology itself is to blame. Almost everyone things that more automation is the solution, but some wonder if our automation tools are just adding another layer of complexity when complexity is the big problem to start with. Hot news: They’re all correct.To read this article in full, please click here

Qualcomm may be planning an ARM-based server processor (again)

Qualcomm may be preparing for another run that the data-center market with a new line of Arm-based processors for servers, according Bloomberg.The company is reportedly seeking customers to test a product from Nuvia, a semiconductor startup it purchased last year that was founded by the former head of Apple CPU development. [ Get regularly scheduled insights by signing up for Network World newsletters. ]To read this article in full, please click here

Verge.io adds GPU virtualization to its HCI software

Verge.io, formerly known as Yottabyte, has added GPU virtualization to its virtual data-center software, making it possible to partition out GPU-based processing tasks the same way a hypervisor divides up a CPU.Hyperconverged infrastructure (HCI) builds a tightly integrated and configured IT data center setup from standard server, storage, and networking servers with everything virtualized. Verge.io’s software, called Verge-OS, goes on step further by partitioning the pooled resources into what it calls virtual data centers (VDCs). From one big pool can come many VDCs made up of different hardware and software configurations.To read this article in full, please click here

Verge.io adds GPU virtualization to its HCI software

Verge.io, formerly known as Yottabyte, has added GPU virtualization to its virtual data-center software, making it possible to partition out GPU-based processing tasks the same way a hypervisor divides up a CPU.Hyperconverged infrastructure (HCI) builds a tightly integrated and configured IT data center setup from standard server, storage, and networking servers with everything virtualized. Verge.io’s software, called Verge-OS, goes on step further by partitioning the pooled resources into what it calls virtual data centers (VDCs). From one big pool can come many VDCs made up of different hardware and software configurations.To read this article in full, please click here

Verge.io adds GPU virtualization to its HCI software

Verge.io, formerly known as Yottabyte, has added GPU virtualization to its virtual data-center software, making it possible to partition out GPU-based processing tasks the same way a hypervisor divides up a CPU.Hyperconverged infrastructure (HCI) builds a tightly integrated and configured IT data center setup from standard server, storage, and networking servers with everything virtualized. Verge.io’s software, called Verge-OS, goes on step further by partitioning the pooled resources into what it calls virtual data centers (VDCs). From one big pool can come many VDCs made up of different hardware and software configurations.To read this article in full, please click here

Verge.io adds GPU virtualization to its HCI software

Verge.io, formerly known as Yottabyte, has added GPU virtualization to its virtual data-center software, making it possible to partition out GPU-based processing tasks the same way a hypervisor divides up a CPU.Hyperconverged infrastructure (HCI) builds a tightly integrated and configured IT data center setup from standard server, storage, and networking servers with everything virtualized. Verge.io’s software, called Verge-OS, goes on step further by partitioning the pooled resources into what it calls virtual data centers (VDCs). From one big pool can come many VDCs made up of different hardware and software configurations.To read this article in full, please click here

1 … 449 450 451 452 453 … 3,841