Unbounded memory usage by TCP for receive buffers, and how we fixed it

Unbounded memory usage by TCP for receive buffers, and how we fixed it
Unbounded memory usage by TCP for receive buffers, and how we fixed it

At Cloudflare, we are constantly monitoring and optimizing the performance and resource utilization of our systems. Recently, we noticed that some of our TCP sessions were allocating more memory than expected.

The Linux kernel allows TCP sessions that match certain characteristics to ignore memory allocation limits set by autotuning and allocate excessive amounts of memory, all the way up to net.ipv4.tcp_rmem max (the per-session limit). On Cloudflare’s production network, there are often many such TCP sessions on a server, causing the total amount of allocated TCP memory to reach net.ipv4.tcp_mem thresholds (the server-wide limit). When that happens, the kernel imposes memory use constraints on all TCP sessions, not just the ones causing the problem. Those constraints have a negative impact on throughput and latency for the user. Internally within the kernel, the problematic sessions trigger TCP collapse processing, “OFO” pruning (dropping of packets already received and sitting in the out-of-order queue), and the dropping of newly arriving packets.

This blog post describes in detail the root cause of the problem and shows the test results of a solution.

TCP receive buffers are excessively big for some sessions

Our journey began when we started noticing a lot Continue reading

People Aren’t Stupid Just Because They Don’t Understand Tech

As technical people, we spend immense time and energy mastering the nuances of specific technologies. Esoteric knowledge is our currency, and we often measure our personal value against the yardstick of technical nuance. And sometimes (maybe lots of times) we gauge other people with the same yardstick, and dismiss those who don’t measure up. This […]

The post People Aren’t Stupid Just Because They Don’t Understand Tech appeared first on Packet Pushers.

Kubernetes Unpacked 026: Data Backup And Recovery In Kubernetes

On today's Kubernetes Unpacked podcast, Michael and Kristina catch up with with Geoff Burke, Senior Cloud Solutions Architect, to talk about running backups for Kubernetes, how to recover those backups, and which tools to use for backup and disaster recovery. We're also pleased to welcome Kristina Devochko as full-time co-host of the podcast!

The post Kubernetes Unpacked 026: Data Backup And Recovery In Kubernetes appeared first on Packet Pushers.

Intel revises its XPU strategy

Intel has announced a shift in strategy that impacts its XPU and data-center product roadmap.XPU is an effort by Intel to combine multiple pieces of silicon into one package. The plan was to combine CPU, GPU, networking, FPGA, and AI accelerator and use software to choose the best processor for the task at hand.That’s an ambitious project, and it looks like Intel is admitting that it can’t do it, at least for now.Jeff McVeigh, corporate vice president and general manager of the Super Compute Group at Intel, provided an update to the data-center processor roadmap that involves taking a few steps back. Its proposed combination CPU and GPU, code-named Falcon Shores, will now be a GPU chip only.To read this article in full, please click here

Intel revises its XPU strategy

Intel has announced a shift in strategy that impacts its XPU and data-center product roadmap.XPU is an effort by Intel to combine multiple pieces of silicon into one package. The plan was to combine CPU, GPU, networking, FPGA, and AI accelerator and use software to choose the best processor for the task at hand.That’s an ambitious project, and it looks like Intel is admitting that it can’t do it, at least for now.Jeff McVeigh, corporate vice president and general manager of the Super Compute Group at Intel, provided an update to the data-center processor roadmap that involves taking a few steps back. Its proposed combination CPU and GPU, code-named Falcon Shores, will now be a GPU chip only.To read this article in full, please click here

Inter-VRF DHCP Relaying with Redundant DHCP Servers

Previous posts in this series covered numerous intricacies of DHCP relaying:

Now for the final bit of the puzzle: what if we want to do inter-VRF DHCP relaying with redundant DHCP servers?

Read more …

Inter-VRF DHCP Relaying with Redundant DHCP Servers

Previous posts in this series covered numerous intricacies of DHCP relaying:

Now for the final bit of the puzzle: what if we want to do inter-VRF DHCP relaying with redundant DHCP servers?

Read more …

Why it makes sense to converge the NOC and SOC

It’s been 17 years and counting since Nemertes first wrote about the logic of integrating event response in the enterprise: bringing together the security operations center (SOC) and network operations center (NOC) at the organizational, operational, and technological levels. Needless to say, this has not happened at most organizations, although there has been a promising trend toward convergence in the monitoring and data management side of things. It’s worth revisiting the issue.Why converge? The arguments for convergence remain pretty compelling: Both the NOC and SOC are focused on keeping an eye on the systems and services comprising the IT environment; spotting and understanding anomalies; and spotting and responding to events and incidents that could affect or are affecting services to the business. Both are focused on minimizing the effects of events and incidents on the business. The streams of data they watch overlap hugely. They often use the same systems (e.g. Splunk) in managing and exploring that data. Both are focused on root-cause analysis based on those data streams. Both adopt a tiered response approach, with first-line responders for “business as usual” operations and occurrences, and anywhere from one to three tiers of escalation to more senior engineers, Continue reading

Why it makes sense to converge the NOC and SOC

It’s been 17 years and counting since Nemertes first wrote about the logic of integrating event response in the enterprise: bringing together the security operations center (SOC) and network operations center (NOC) at the organizational, operational, and technological levels. Needless to say, this has not happened at most organizations, although there has been a promising trend toward convergence in the monitoring and data management side of things. It’s worth revisiting the issue.Why converge? The arguments for convergence remain pretty compelling: Both the NOC and SOC are focused on keeping an eye on the systems and services comprising the IT environment; spotting and understanding anomalies; and spotting and responding to events and incidents that could affect or are affecting services to the business. Both are focused on minimizing the effects of events and incidents on the business. The streams of data they watch overlap hugely. They often use the same systems (e.g. Splunk) in managing and exploring that data. Both are focused on root-cause analysis based on those data streams. Both adopt a tiered response approach, with first-line responders for “business as usual” operations and occurrences, and anywhere from one to three tiers of escalation to more senior engineers, Continue reading

Automating Green-House Photos through Event-Bridge Pipes and Lambda

< MEDIUM: https://medium.com/towards-aws/automating-green-house-photos-through-event-bridge-pipes-and-lambda-434461b89f55 >

Image sent to Telegram

I have a small greenhouse which was in the pipeline for over 2 years and I finally decided to build it. Whoever is in gardening will agree that anything grows better in the greenhouse at least it appears to be so.

Now, the initial impression is all good but I have plans to learn and explore both the plant sides of things and also some using some part of image analysis for a predictive action, for all that to happen I need a camera and a picture to start with.

Hardware —

  1. Raspberry Pi — I have an old one at home, you can technically have any shape or size as long as it fits your need, My recommendation — is Raspberry Pi Zero

What are the other simplest alternatives:

  • I could have written a Python script which directly could have sent the image to Telegram storing the image locally or uploading it to S3

The reason I choose to go with Event-bridge Pipe is to put this more into practice and from there on connect more Lambda and step-functions for future expansion of the project.

Architecture Diagram for sending Images Continue reading

Achieving High Availability (HA) Redis Kubernetes clusters with Calico Clustermesh in Microsoft AKS

According to the recent Datadog report on real world container usage, Redis is among the top 5 technologies used in containerized workloads running on Kubernetes.

Redis database is deployed across multi-region clusters to be Highly Available(HA) to a microservices application. However, while Kubernetes mandates how the networking and security policy is deployed and configured in a single cluster it is challenging to enforce inter-cluster communication at pod-level, enforce security policies and connect to services running in pods across multiple clusters.

Calico Clustermesh provides an elegant solution to highly available multiple Redis clusters without any overheads. By default, deployed Kubernetes pods can only see pods within their cluster.

Using Calico Clustermesh, you can grant access to other clusters and the applications they are running. Calico Clustermesh comes with Federated Endpoint Identity and Federated Services.

Federated endpoint identity

Calico federated endpoint identity and federated services are implemented in Kubernetes at the network layer. To apply fine-grained network policy between multiple clusters, the pod source and destination IPs must be preserved. So the prerequisite for enabling federated endpoints requires clusters to be designed with common networking across clusters (routable pod IPs) with no encapsulation.

Federated services

Federated services works with federated endpoint identity, Continue reading

After China’s Micron ban, US lawmakers urge sanctions on chips from CXMT

The US Commerce Department should put trade restrictions on Chinese memory chip maker Changxin Memory Technologies (CXMT), say lawmakers on the US House of Representative’s Committee on China.The comments come in the wake of the Chinese government ban on the use of some Micron chips in certain sectors, citing concerns that the products pose a significant security risk to the country’s key information infrastructure supply chain.However, these claims are “not based in fact” White House press secretary Karine Jean-Pierre told reporters, adding that the Department of Commerce was engaged directly with the PRC (People's Republic of China) to detail the administration’s views on the ban.To read this article in full, please click here

Day Two Cloud 196: Peering Behind The Curtain Of Podsqueeze’s AI Podcasting Service

Today's show gets behind the curtain of a cloud service called Podsqueeze. Podsqueeze is an application that ingests audio and video files and then produces text-based output including a show description, an episode transcript, suggested headlines, segment timestamps, suggested social media posts, and more. The Packet Pushers are experimenting with Podsqueeze as part of our own production. Being curious nerds, we thought this was a good opportunity to see how the service really works. Our guest is Tiago Ferreira, one of the entrepreneurs and developers of Podsqueeze.

Day Two Cloud 196: Peering Behind The Curtain Of Podsqueeze’s AI Podcasting Service

Today's show gets behind the curtain of a cloud service called Podsqueeze. Podsqueeze is an application that ingests audio and video files and then produces text-based output including a show description, an episode transcript, suggested headlines, segment timestamps, suggested social media posts, and more. The Packet Pushers are experimenting with Podsqueeze as part of our own production. Being curious nerds, we thought this was a good opportunity to see how the service really works. Our guest is Tiago Ferreira, one of the entrepreneurs and developers of Podsqueeze.

The post Day Two Cloud 196: Peering Behind The Curtain Of Podsqueeze’s AI Podcasting Service appeared first on Packet Pushers.

Failed Expectations

In a recent workshop I attended, reflecting on the evolution of the Internet over the past 40 years, one of the takeaways for me is how we've managed to surprise ourselves in both the unanticipated successes we've encountered and in the instances of failure when technology has stubbornly resisted to be deployed despite our confident expectations to the contrary! What have we learned from these lessons of our inability to predict technology outcomes?

Leaf and spine network emulation on Mac OS M1/M2 systems


The GitHub sflow-rt/containerlab project contains example network topologies for the Containerlab network emulation tool that demonstrate real-time streaming telemetry in realistic data center topologies and network configurations. The examples use the same FRRouting (FRR) engine that is part of SONiC, NVIDIA Cumulus Linux, and DENT network operating systems. Containerlab can be used to experiment before deploying solutions into production. Examples include: tracing ECMP flows in leaf and spine topologies, EVPN visibility, and automated DDoS mitigation using BGP Flowspec and RTBH controls.

The Containerlab project currently has limited support for Mac OS, stating "ARM-based Macs (M1/2) are not supported, and no binaries are generated for this platform. This is mainly due to the lack of network images built for arm64 architecture as of now." However, this argument doesn't apply to the Linux based images used in these examples.

First install Docker Desktop on your Apple silicon based Mac (select the Apple Chip option).

mkdir clab
cd clab
docker run --rm -it --privileged \
  --network host --pid="host" \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /run/netns:/run/netns \
  -v $(pwd):$(pwd) -w $(pwd) \
  sflow/clab bash

Run Containerlab by typing the above commands in a terminal. This command uses a pre-built multi-architecture Continue reading

1 … 214 215 216 217 218 … 3,765