HTTP Analytics for 6M requests per second using ClickHouse

HTTP Analytics for 6M requests per second using ClickHouse

One of our large scale data infrastructure challenges here at Cloudflare is around providing HTTP traffic analytics to our customers. HTTP Analytics is available to all our customers via two options:

In this blog post I'm going to talk about the exciting evolution of the Cloudflare analytics pipeline over the last year. I'll start with a description of the old pipeline and the challenges that we experienced with it. Then, I'll describe how we leveraged ClickHouse to form the basis of a new and improved pipeline. In the process, I'll share details about how we went about schema design and performance tuning for ClickHouse. Finally, I'll look forward to what the Data team is thinking of providing in the future.

Let's start with the old data pipeline.

Old data pipeline

The previous pipeline was built in 2014. It has been mentioned previously in Scaling out PostgreSQL for CloudFlare Analytics using CitusDB and More data, more data blog posts from the Data team.
HTTP Analytics for 6M requests per second using ClickHouse
It had following components:

  • Log forwarder - collected Cap'n Proto formatted logs from the edge, notably DNS and Nginx logs, Continue reading

China’s Global Cloud and AI Ambitions Keep Extending

Gone are the days of early warehouse scale computing pioneers that were based in the U.S.. Over the last several years, China’s web giants are extending their reach through robust shared AI and cloud efforts—and those are pushing ever further into territory once thought separate.

Alibaba is much like compatriots Baidu and Tencent in its desire to expand well beyond the borders of China and compete with global players like Amazon Web Services, Google, Microsoft and Facebook in such fast-growing areas like the cloud, supercomputing and artificial intelligence (AI).

The tech giant has significant resources at its disposal, pulling in …

China’s Global Cloud and AI Ambitions Keep Extending was written by Jeffrey Burt at The Next Platform.

Machine Learning for Auto-Tuning HPC Systems

On today’s episode of “The Interview” with The Next Platform we discuss the art and science of tuning high performance systems for maximum performance—something that has traditionally come at high time cost for performance engineering experts.

While the role of performance engineer will not disappear anytime soon, machine learning is making tuning systems—everything from CPUs to application specific parameters—less of a burden. Despite the highly custom nature of systems and applications, reinforcement learning is allowing new leaps in time-saving tuning as software learns what works best for user applications and architectures, freeing up performance engineers to focus on the finer …

Machine Learning for Auto-Tuning HPC Systems was written by Nicole Hemsoth at The Next Platform.

BrandPost: Your Roadblock to Becoming Digital: A 1990s Server Room

Digital transformation is fundamentally changing businesses of every size. Larger enterprises no longer have a unique advantage based purely on their scale. Small and midsize businesses (SMBs) can utilize new applications and services that allow them to compete toe to toe against a business or organization of any size. Not only do your customers want the latest and greatest, but employees demand it as well. If your employees can’t get the digital tools they need to be successful, they’ll go where they’re available.The focus of management and employees today is on exciting new apps and services. However, without the appropriate IT infrastructure to support them, an organization will not be able to move forward on the digital transformation journey. New apps and new devices are dependent on the servers, storage, and network gear that provide the data and back-end services.  Unfortunately for many SMBs, IT equipment is stored in a 1990s server room that may be a repurposed conference room, cubicle, or spare office that cannot provide the physical environment the IT hardware needs to support a 21st century organization.To read this article in full, please click here

Startup ZincFive makes old battery technology new again

A startup called ZincFive is set to launch a modular uninterruptible power supply (UPS) for data center computers using the venerable nickel-zinc technology, which it claims is more efficient than lithium-ion.Nickel-zinc batteries were invented by Thomas Edison in 1901 but fell out of favor to newer designs due to their limitations, such as a low number of charge cycles and their inability to hold a charge for long.On the plus side, the batteries could hold a stronger charge and didn’t use toxic metals like other batteries that make them difficult to recycle. And they are not flammable, something lithium-ion batteries certainly can’t claim.To read this article in full, please click here

Startup ZincFive makes old battery technology new again

A startup called ZincFive is set to launch a modular uninterruptible power supply (UPS) for data center computers using the venerable nickel-zinc technology, which it claims is more efficient than lithium-ion.Nickel-zinc batteries were invented by Thomas Edison in 1901 but fell out of favor to newer designs due to their limitations, such as a low number of charge cycles and their inability to hold a charge for long.On the plus side, the batteries could hold a stronger charge and didn’t use toxic metals like other batteries that make them difficult to recycle. And they are not flammable, something lithium-ion batteries certainly can’t claim.To read this article in full, please click here

5 years later, where are you on your Docker journey?

Docker is turning five the week of March 19-25, 2018 and in celebration of this event, we would like to turn the spotlight to our users – the customers, partners, individuals and organizations that are using the Docker platform every day. From scientific use cases in the field of medical and space research to more traditional use cases in large banks and insurance companies, Docker’s adoption is only accelerating with everyone from individual developers to global corporations. Here are some of the key figures showing the widespread adoption of Docker across the community and within enterprises.

 

Docker has evolved from a container runtime to Docker Enterprise Edition (EE), a secure enterprise-ready container platform that brings unmatched freedom of choice to enterprises, while providing a foundation for their digital and multi-cloud initiatives. Millions of users rely on Docker, downloading 100M container images a day, and over 450 companies have turned to Docker Enterprise Edition – including hundreds of the largest enterprises in the world. With such vast adoption, the range of stories to tell and the diverse set of use cases continues to grow. So where are some of these users and enterprise customers on their Docker journey?

Docker users have deployed Docker containers for a Continue reading

Don’t forget to tune into our CCNA Kickoff Session tomorrow!

Watch our March 2018 CCNA Kickoff Session with Keith Bogart TOMORROW at 1:30 PM EST.

 

This kickoff session for those who are interested in, or have started to study for the CCNA certification. In this free session, we will cover common trouble areas that most people experience when getting started with their certification. Topics include: how to approach making a study schedule, strategies for not becoming overwhelmed during the study process, deciding whether to take one test or two to get your CCNA, what to expect when you walk into the testing center, which topics to study and how in depth, and what study tools can be useful. Keith will also discuss the testing experience and the CCNA Certification test format.

When: March 7th at 10:30 am PST/ 1:30 pm EST

Estimated Length: 3 hours

Instructor: Keith Bogart CCIE #4923

Cost: FREE

NAT To Be Missed At SXSW

NAT To Be Missed At SXSW

We’re at the EDGE of our seats, about to LANd in Austin, Texas in route for SXSW. (TKIP, hip, hooray!)

ARP you going to be there? We R going to have three epoch sessions by Cloudflare speakers. Ifdown, seems apt you could SELECT to JOIN. Cat make it? Not a bg deal, wget it (though it mega hertz we won’t C you). All the audio from the three sessions will be recorded, you can listen to the cd.

WPS! I almost forgot to tel(net) you whois going to be there, and WAN and where to go.

On Friday, March 9, I’m moderating a panel with Emily Schechter from Google, Aaron DeVera from Deloitte and Gabe Kassel from eero about how Wi-Fi networks work and WEP happens when attackers coax people into joining insecure networks. It’s at Salon K in the Hilton at 3:30PM.

On Sunday the 11th, Nitin Rao is on a panel with Heather West from Mozilla, Stefan Lederer from Bitmovin and Fred Benenson from Unlimited Liability Corporation LLC about the impact of the recent revocation of Net Neutrality rules on online video streaming. It’s at 11AM at Salon J in the Continue reading

DFS and Low Points

On a recent history of networking episode, Alia talked a little about Maximally Redundant Trees (MRTs), and the concept of Depth First Search (DFS) numbering, along with the idea of a low point. While low points are quickly explained in my new book in the context of MRTs, I thought it worthwhile to revisit the concept in a blog post. Take a look at the following network:

On the left side is a small network with the nodes (think of these as routers) being labeled from A through G. On the right side is the same network, only each node has been numbered by traversing the graph, starting at A. This process, in a network, would either require some device which knows about every node and edge (link) in the network, or it would require a distributed algorithm that “walks” the network from one node to another, numbering each node as it is touched, and skipping any node that has already been visited (again, for more details on this, please see the book).

Once this numbering has been done, the numbers now produce this interesting property: if you remove the parent of any node, and the node can still reach Continue reading

IDG Contributor Network: What to understand about health care IoT and its security

As we have seen, the Internet of Things will disrupt and change every industry and how actors within it do business. Along with new paradigms in services and products that one can offer due to the proliferation of IoT, come business risks as well as heightened security concerns – both physical and cyber. In our prior column, we spoke about this topic in the context of the Smart Electric Grid. Today we’re taking a look at how IoT is disrupting the health care market and how we can take steps to secure it.To read this article in full, please click here

IDG Contributor Network: What to understand about health care IoT and its security

As we have seen, the Internet of Things will disrupt and change every industry and how actors within it do business. Along with new paradigms in services and products that one can offer due to the proliferation of IoT, come business risks as well as heightened security concerns – both physical and cyber. In our prior column, we spoke about this topic in the context of the Smart Electric Grid. Today we’re taking a look at how IoT is disrupting the health care market and how we can take steps to secure it.To read this article in full, please click here

1 … 1,690 1,691 1,692 1,693 1,694 … 3,832