One of our large scale data infrastructure challenges here at Cloudflare is around providing HTTP traffic analytics to our customers. HTTP Analytics is available to all our customers via two options:
In this blog post I'm going to talk about the exciting evolution of the Cloudflare analytics pipeline over the last year. I'll start with a description of the old pipeline and the challenges that we experienced with it. Then, I'll describe how we leveraged ClickHouse to form the basis of a new and improved pipeline. In the process, I'll share details about how we went about schema design and performance tuning for ClickHouse. Finally, I'll look forward to what the Data team is thinking of providing in the future.
Let's start with the old data pipeline.
The previous pipeline was built in 2014. It has been mentioned previously in Scaling out PostgreSQL for CloudFlare Analytics using CitusDB and More data, more data blog posts from the Data team.
It had following components:
The project was the first adopted by CNCF.
AT&T's Chris Rice, SVP of Domain 2.0, praises the new chip.
He'll report to VMware Founder and former CEO Diane Greene at Google Cloud.
Watch our March 2018 CCNA Kickoff Session with Keith Bogart TOMORROW at 1:30 PM EST.
This kickoff session for those who are interested in, or have started to study for the CCNA certification. In this free session, we will cover common trouble areas that most people experience when getting started with their certification. Topics include: how to approach making a study schedule, strategies for not becoming overwhelmed during the study process, deciding whether to take one test or two to get your CCNA, what to expect when you walk into the testing center, which topics to study and how in depth, and what study tools can be useful. Keith will also discuss the testing experience and the CCNA Certification test format.
When: March 7th at 10:30 am PST/ 1:30 pm EST
Estimated Length: 3 hours
Instructor: Keith Bogart CCIE #4923
Cost: FREE
Netsurion's SD-WAN is targeted to multi-tenant enterprises.
We’re at the EDGE of our seats, about to LANd in Austin, Texas in route for SXSW. (TKIP, hip, hooray!)
ARP you going to be there? We R going to have three epoch sessions by Cloudflare speakers. Ifdown, seems apt you could SELECT to JOIN. Cat make it? Not a bg deal, wget it (though it mega hertz we won’t C you). All the audio from the three sessions will be recorded, you can listen to the cd.
WPS! I almost forgot to tel(net) you whois going to be there, and WAN and where to go.
On Friday, March 9, I’m moderating a panel with Emily Schechter from Google, Aaron DeVera from Deloitte and Gabe Kassel from eero about how Wi-Fi networks work and WEP happens when attackers coax people into joining insecure networks. It’s at Salon K in the Hilton at 3:30PM.
On Sunday the 11th, Nitin Rao is on a panel with Heather West from Mozilla, Stefan Lederer from Bitmovin and Fred Benenson from Unlimited Liability Corporation LLC about the impact of the recent revocation of Net Neutrality rules on online video streaming. It’s at 11AM at Salon J in the Continue reading
The new programmable chipset nears Shannon's limit in terms of performance.
On a recent history of networking episode, Alia talked a little about Maximally Redundant Trees (MRTs), and the concept of Depth First Search (DFS) numbering, along with the idea of a low point. While low points are quickly explained in my new book in the context of MRTs, I thought it worthwhile to revisit the concept in a blog post. Take a look at the following network:
On the left side is a small network with the nodes (think of these as routers) being labeled from A through G. On the right side is the same network, only each node has been numbered by traversing the graph, starting at A. This process, in a network, would either require some device which knows about every node and edge (link) in the network, or it would require a distributed algorithm that “walks” the network from one node to another, numbering each node as it is touched, and skipping any node that has already been visited (again, for more details on this, please see the book).
Once this numbering has been done, the numbers now produce this interesting property: if you remove the parent of any node, and the node can still reach Continue reading