ClickHouse SF Bay Area Meetup: Akvorado
Here are the slides I presented for a ClickHouse SF Bay Area Meetup in July 2022, hosted by Altinity. They are about Akvorado, a network flow collector and visualizer, and notably on how it relies on ClickHouse, a column-oriented database.
The meetup was recorded and available on YouTube. Here is the part relevant to my presentation, with subtitles:1
I got a few questions about how to get information from the higher layers, like HTTP. As my use case for Akvorado was at the network edge, my answers were mostly negative. However, as sFlow is extensible, when collecting flows from Linux servers instead, you could embed additional data and they could be exported as well.
I also got a question about doing aggregation in a single table.
ClickHouse can aggregate automatically data using TTL. My answer for
not doing that is partial. There is another reason: the retention
periods of the various tables may overlap. For example, the main table
keeps data for 15 days, but even in these 15 days, if I do a query on
a 12-hour window, it is faster to use the flows_1m0s aggregated
table, unless I request something about Continue reading
How to Find Skilled Network Operations Talent in a Tight Labor Market
Network operations experts are in extremely short supply. Addressing the challenge requires perseverance, flexibility, and a hefty dose of realism.Day Two Cloud 156: Multi-Cloud Experience Monitoring With Broadcom Software (Sponsored)
It's hard to guarantee quality of experience for users accessing cloud applications. The users are connected via networks we don’t own, and the apps are hosted on networks we don’t own. So what can a network operations team do about quality of experience in a world of cloud-hosted apps and cloud-connected users? Sponsor Broadcom Software is here to help us answer this question.Privacy And Networking Part 5: The Data Lifecycle
In the previous posts in this series, I concluded that privacy is everyone’s responsibility, that IP addresses (and a lot of other information network engineers handle) are protected information, and while processing packets probably doesn’t trigger any privacy warnings, network logging should and does. In this post, I want to start answering the question—okay, what […]
The post Privacy And Networking Part 5: The Data Lifecycle appeared first on Packet Pushers.
Twilight Zone: File Transfer Never Completes
Ages ago when we were building networks using super-expensive 64kbps WAN links, a customer sent us a weird bug report:
Everything works fine, but we cannot transfer one particular file between two locations – the file transfer stalls and eventually times out. At the same time, we’re seeing increased number of CRC errors on the WAN link.
My chat with the engineer handling the ticket went along these lines:
An Overview Of Cisco’s SecureX Device Insights
Device Insights, a feature of Cisco's SecureX XDR service, aggregates, normalizes, and visualizes esssential details about all the devices on your network. SecureX can also automate workflows to respond to device-level security problems.
The post An Overview Of Cisco’s SecureX Device Insights appeared first on Packet Pushers.
Getting started with container security
A couple of days ago, I was checking my Twitter feed and saw a tweet from someone saying how frustrated he was that DockerHub (a renowned container registry) was down. Someone else replied to the tweet, recommending the tweet’s author to check out Google’s repository, where they have DockerHub mirrors in Google Cloud.
My first reaction was “Nice! How clever of this person (or Google) to have thought of this idea.” My next thought was, wait. This could lead to potential security risks for some developers who are not familiar with how these registries are updated and what images go into these mirrored sites. Imagine when application developers are busy scrambling to check-in their latest update to the CI/CD pipeline of the software they are building, and in that time crunch, their go-to container registry is down. Do developers really have the time to check if there are vulnerable images in every registry they use? Will there be an easy, streamlined way to automatically scan the images no matter which registry developers use to pull their images? The short answer is yes, and we will look into that in this blog.
Scan all your container assets with Calico Cloud
In a Digital World, Anti-Fraud and Security Teams Should be Partners
Sharing the right information at the right time is critical to extraordinary digital experiences and protecting customers and the business from fraud.Full Stack Journey 068: An Introduction To Pixie For Open-Source Kubernetes Observability
Today's Full Stack Journey podcast is all about Pixie, an "open source observability tool for Kubernetes applications." Pixie takes advantage of eBPF to capture telemetry data. Joining Scott to provide a beginner-level overview of Pixie is Fabian Ngala.
The post Full Stack Journey 068: An Introduction To Pixie For Open-Source Kubernetes Observability appeared first on Packet Pushers.
Full Stack Journey 068: An Introduction To Pixie For Open-Source Kubernetes Observability
Today's Full Stack Journey podcast is all about Pixie, an "open source observability tool for Kubernetes applications." Pixie takes advantage of eBPF to capture telemetry data. Joining Scott to provide a beginner-level overview of Pixie is Fabian Ngala.
LiveAction Unifies Flow, Packets For Deep Network Visibility
LiveAction provides network and application monitoring that integrates flow records and packets. This integration gives network management teams a clear, clean view of network topology and applications traffic–even for the most complex, multi-vendor networks. LiveAction can also capture and analyze packets to help engineers troubleshoot problems and investigate incidents.
The post LiveAction Unifies Flow, Packets For Deep Network Visibility appeared first on Packet Pushers.
When the window is not fully open, your TCP stack is doing more than you think
Over the years I've been lurking around the Linux kernel and have investigated the TCP code many times. But when recently we were working on Optimizing TCP for high WAN throughput while preserving low latency, I realized I have gaps in my knowledge about how Linux manages TCP receive buffers and windows. As I dug deeper I found the subject complex and certainly non-obvious.
In this blog post I'll share my journey deep into the Linux networking stack, trying to understand the memory and window management of the receiving side of a TCP connection. Specifically, looking for answers to seemingly trivial questions:
- How much data can be stored in the TCP receive buffer? (it's not what you think)
- How fast can it be filled? (it's not what you think either!)
Our exploration focuses on the receiving side of the TCP connection. We'll try to understand how to tune it for the best speed, without wasting precious memory.
A case of a rapid upload
To best illustrate the receive side buffer management we need pretty charts! But to grasp all the numbers, we need a bit of theory.
We'll draw charts from a receive side of a TCP flow, Continue reading