We at Cloudflare are long time Kafka users, first mentions of it date back to beginning of 2014 when the most recent version was 0.8.0. We use Kafka as a log to power analytics (both HTTP and DNS), DDOS mitigation, logging and metrics.
Firehose CC BY 2.0 image by RSLab
While the idea of unifying abstraction of the log remained the same since then (read this classic blog post from Jay Kreps if you haven't), Kafka evolved in other areas since then. One of these improved areas was compression support. Back in the old days we've tried enabling it a few times and ultimately gave up on the idea because of unresolved issues in the protocol.
Just last year Kafka 0.11.0 came out with the new improved protocol and log format.
The naive approach to compression would be to compress messages in the log individually:
Edit: originally we said this is how Kafka worked before 0.11.0, but that appears to be false.
Compression algorithms work best if they have more data, so in the new log format messages (now called records) are packed back to back and compressed in Continue reading
Fixing the Internet: Is the Internet broken? Politico’s EU site looks at the work of the Internet & Jurisdiction Policy Network, which met in Ottawa, Canada, last week to discuss how to fix problems like poor cybersecurity, inaccurate information spread on social media, and other bad behavior. The Internet Society covered the first day of the Ottawa event.
The hills are alive with the sound of broadband: Motherboard has a story about the Los Angeles Community Broadband Project, which plans to deliver wireless broadband to parts of the city using inexpensive equipment and dish-shaped antennas on hilltops and rooftops.
AI joins the force: The Verge has a long story about a secretive AI-assisted policing effort that started in 2012 as a partnership between the New Orleans Police and Palantir Technologies, a data-mining company founded with seed money from the CIA’s venture capital firm. The program apparently used AI technologies for predictive policing, a controversial practice used to trace suspects’ ties to other gang members, analyze social media, and predict the likelihood targeted people would commit violence or become a victim. Science Magazine also has a story examining predictive policing.
Women wary of Blockchain bros: The New York Continue reading
Here are some of the top-notch experts who will talk about networking, containers, cloud, open source, and more in Las Vegas.
In the Business Impact of Network Automation podcast Ethan Banks asked an interesting question: “what will happen with older networking engineers who are not willing to embrace automation”
The response somewhat surprised me: Alejandro Salisas said something along the lines “they’ll be just fine” (for a while).
Let me recap his argument and add a few twists of my own:
Read more ...I have to admit I LOVE MPLS. I admit, I didn’t love it so much when I was first learning. I found it kinda hard at first. But then I absolutely loved it once I “saw” it. Newer to MPLS... Read More ›
The post MPLS L3VPN: Label Following Fun with Fish appeared first on Networking with FISH.
Russ White wrote a great article along the lines of what we discussed a while ago. My favorite part:
There are companies who consider the network an asset, and companies that consider the network a necessary evil.
Enjoy!
On a tangential topic: Russ will talk about network complexity in the Building Next-Generation Data Center online course starting on April 25th.
Whether we like it or not, the era of DevOps is upon us, fellow network engineers, and with it come opportunities to approach and solve common networking problems in new, innovative ways. One such problem is automated network change validation and testing in virtual environments, something I’ve already written about a few years ago. The biggest problem with my original approach was that I had to create a custom REST API SDK to work with a network simulation environment (UnetLab) that was never designed to be interacted with in a programmatic way. On the other hand, technologies like Docker have been very interesting since they were built around the idea of non-interactive lifecycle management and came with all API batteries already included. However, Docker was never intended to be used for network simulations and its support for multiple network interfaces is… somewhat problematic.
The easiest way to understand the problem is to see it. Let’s start with a blank Docker host and create a few networks:
docker network create net1
docker network create net2
docker network create net3
Now let’s see what prefixes have been allocated to those networks:
docker network inspect -f "{{range .IPAM.Config }}{{.Subnet}}{{end}}" Continue reading
Cisco and VMware change the SD-WAN playing field.
Minjar sells a service that compares costs across public clouds.