Archive

Category Archives for "Networking"

What Is EVPN?

EVPN might be the next big thing in networking… or at least all the major networking vendors think so. It’s also a pretty complex technology still facing some interoperability challenges (I love to call it SIP of networking).

To make matters worse, EVPN can easily get even more confusing if you follow some convoluted designs propagated on the ‘net… and the best antidote to that is to invest time into understanding the fundamentals, and to slowly work through more complex scenarios after mastering the basics.

Read more ...

Tracing System CPU on Debian Stretch

Tracing System CPU on Debian Stretch

This is a heavily truncated version of an internal blog post from August 2017. For more recent updates on Kafka, check out another blog post on compression, where we optimized throughput 4.5x for both disks and network.

Tracing System CPU on Debian Stretch
Photo by Alex Povolyashko / Unsplash

Upgrading our systems to Debian Stretch

For quite some time we've been rolling out Debian Stretch, to the point where we have reached ~10% adoption in our core datacenters. As part of upgarding the underlying OS, we also evaluate the higher level software stack, e.g. taking a look at our ClickHouse and Kafka clusters.

During our upgrade of Kafka, we sucessfully migrated two smaller clusters, logs and dns, but ran into issues when attempting to upgrade one of our larger clusters, http.

Thankfully, we were able to roll back the http cluster upgrade relatively easily, due to heavy versioning of both the OS and the higher level software stack. If there's one takeaway from this blog post, it's to take advantage of consistent versioning.

High level differences

We upgraded one Kafka http node, and it did not go as planned:

Tracing System CPU on Debian Stretch

Having 5x CPU usage was definitely an unexpected outcome. For control datapoints, we Continue reading

Networking With Fish: YouTube Channel

Blogging, originally, was my go to and preferred method for sharing information to others – teaching, sharing, etc.  For a few corner case type things I found video (YouTube) to be a better tool for those specific items.  Recently, however, I am finding about half of my ideas of things I want to “pass on” to others… would be best (in my opinion) via video.

I’ve been trying to figure out and think about how best to have the two sharing tools – this blog site and the YouTube channel – best compliment each other.  So I have been experimenting with this.  What I have come up with that I like and works for me is the following…..

  • “Standalone Video” – If the YouTube is really a “standalone” and blogging with additional text around it here doesn’t “help” communicate what I’m trying to get across… then I won’t be blogging about it here.
  • “Video Series” – There will be series that will be building on each other – like the videos in the playlist “BGP Show and Tell: Beginners” and the playlist “Label Swapping Fun”.   Video series, I believe, would definitely benefit from larger big Continue reading

Worth Reading: Cognitive Dissonance

I always wondered why it’s so hard to accept that someone might not find your preferred solution beautiful but would call it complex or even harmful (or from the other side, why someone could not possibly appreciate the beauty of your design)… and then stumbled upon this blog post by Scott Adams describing cognitive dissonance (the actual topic they’re discussing in the mentioned video doesn’t matter – look for the irrational behavior).

You might say “but we could politely agree to disagree” but unfortunately that implies that at least one of us is not fully rational due to Aumann’s Agreement Theorem.

Link Propagation 118

Welcome to Link Propagation, a Packet Pushers newsletter. Link Propagation is included in your free membership. Each week we scour the InterWebs to find the most relevant practitioner blog posts, tech news, and product announcements. We drink from the fire hose so you can sip from a coffee cup. Blogs Foundations of MPLS: Label Switching […]

Time To Get Back To Basics?

I’ve had some fascinating networking discussions over the past couple of weeks at Dell Technologies World, Interop, and the spring ONUG meeting. But two of them have hit on some things that I think need to be addressed in the industry. Both Russ White and Ignas Bagdonas of the IETF have come to me and talked about how they feel networking professionals have lost sight of the basics.

How Stuff Works

If you walk up to any network engineer and ask them to explain how TCP works, you will probably get a variety of answers. Some will try to explain it to you in basic terms to avoid getting too in depth. Others will swamp you with a technical discussion that would make the protocol inventors proud. But still others will just shrug their shoulders and admit they don’t really understand the protocol.

It’s a common problem when a technology gets to the point of being mature and ubiquitous. One of my favorite examples is the fuel system on an internal combustion engine. On older cars or small engines, the carburetor is responsible for creating the correct fuel and air mixture that is used to power the cylinders. Getting that Continue reading

SeaMeWe-3 Experiences Another Cable Break

On Thursday, May 10 at approximately 02:00 UTC, the SeaMeWe-3 (SMW-3) subsea cable suffered yet another cable break. The break disrupted connectivity between Australia and Singapore, causing latencies to spike as illustrated below in our Internet Intelligence tool, because traffic had to take a more circuitous path.

The SMW-3 cable has had a history of outages, which we have reported on multiple times in the past, including August 2017, December 2014, and January 2013.

The incident summary posted by cable owner Vocus Communications for this most recent break noted that “There is no ETR at this stage.” However, based on our observations of past outages, time to recovery has been measured on the order of weeks.

While this subsea cable is currently the only one carrying traffic from Western Australia to South East Asia, there are several additional cable projects in process that will help address this long-standing issue. The Australia-Singapore Cable (ASC) Continue reading

Weekend Reads: 051118: New spectre-class vulnerabilities, scraping data, and no middle ground on encryption

A team of security researchers has reportedly discovered a total of eight new “Spectre-class” vulnerabilities in Intel CPUs, which also affect at least a small number of ARM processors and may impact AMD processor architecture as well. Dubbed Spectre-Next Generation, or Spectre-NG, the partial details of the vulnerabilities were first leaked to journalists at German computer magazine Heise, which claims that Intel has classified four of the new vulnerabilities as “high risk” and remaining four as “medium.” —Mohit Kumar @Hacker News

As cities get smarter, their appetite and access to information is also increasing. The rise of data-generating technologies has given government agencies unprecedented opportunities to harness useful, real-time information about citizens. But governments often lack dedicated expertise and resources to collect, analyze, and ultimately turn such data into actionable information, and so have turned to private-sector companies and academic researchers to get at this information. —Joseph Jerome @CDT

Despite this renewed rhetoric, most experts continue to agree that exceptional access, no matter how you implement it, weakens security. The terminology might have changed, but the essential question has not: should technology companies be forced to develop a system that inherently harms their users? The answer hasn’t changed either: Continue reading

IDG Contributor Network: Defining network performance with Google’s 4 golden signals

You’re supposed to meet someone for coffee. If they’re three minutes late, no problem, but if they’re thirty minutes late, its rude. Was the change from “no problem” to “rude” a straight line, or were there steps of increasing rudeness? Do we care why? A good reason certainly increases our tolerance. Someone who is always late reduces it.Network performance follows many of the same dynamics. We used to talk about outages, but they have become less frequent. “Slow” is the new “out.” But how slow is slow? Do we try to understand the user experience and adjust our performance monitoring to reflect it? Or is the only practical answer to just wait until someone complains?To read this article in full, please click here

IDG Contributor Network: Defining network performance with Google’s 4 golden signals

You’re supposed to meet someone for coffee. If they’re three minutes late, no problem, but if they’re thirty minutes late, its rude. Was the change from “no problem” to “rude” a straight line, or were there steps of increasing rudeness? Do we care why? A good reason certainly increases our tolerance. Someone who is always late reduces it.Network performance follows many of the same dynamics. We used to talk about outages, but they have become less frequent. “Slow” is the new “out.” But how slow is slow? Do we try to understand the user experience and adjust our performance monitoring to reflect it? Or is the only practical answer to just wait until someone complains?To read this article in full, please click here

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Project Jengo Celebrates One Year Anniversary by Releasing Prior Art

Today marks the one year anniversary of Project Jengo, a crowdsourced search for prior art that Cloudflare created and funded in response to the actions of Blackbird Technologies, a notorious patent troll. Blackbird has filed more than one hundred lawsuits asserting dormant patents without engaging in any innovative or commercial activities of its own. In homage to the typical anniversary cliché, we are taking this opportunity to reflect on the last year and confirm that we’re still going strong.

Project Jengo arose from a sense of immense frustration over the way that patent trolls purchase over-broad patents and use aggressive litigation tactics to elicit painful settlements from companies. These trolls know that the system is slanted in their favor, and we wanted to change that. Patent lawsuits take years to reach trial and cost an inordinate sum to defend. Knowing this, trolls just sit back and wait for companies to settle. Instead of perpetuating this cycle, Cloudflare decided to bring the community together and fight back.

After Blackbird filed a lawsuit against Cloudflare alleging infringement of a vague and overly-broad patent (‘335 Patent), we launched Project Jengo, which offered a reward to people who submitted prior art that could Continue reading

Show 389: Using MPLS In The Enterprise

Today on the Weekly show, MPLS in the enterprise, especially for the use case of segmentation.

Should you do it? Why would you do it? Considering the equipment you ve got, could you do it even if you wanted to? And even if you should, you would, and you can, what about all those other people you work with?

Discussing this emotionally fraught topic of MPLS in the enterprise is Tom Ammon, a senior network engineer who s supported it in a few different environments. He blogs at blog.tomammon.net.

We start by discussing the general case for MPLS in the enterprise, walk through the hardware and software requirements necessary to support MPLS, and then drill into segmentation more specifically.

We also look at reasons why you’d want to use MPLS for segmentation, and explores issues around operations, skillsets and training, and troubleshooting.

The post Show 389: Using MPLS In The Enterprise appeared first on Packet Pushers.

Minimizing Risk: How to Be Secure on Twitter and Other Social Networks

Last week’s news that the passwords of every Twitter user around the world had been exposed in plain text is a stark reminder of the sometimes-fragile nature of security in the online places we trust with our personal information.

In this latest example, Twitter says it was a technical error that led to the exposure of the passwords of the social network’s 330 million users. Twitter also says it’s fixed the bug and has no evidence that anyone’s accounts have been breached or misused.

While the error led to exposure only within Twitter’s internal systems, the social network has urged everyone to change their passwords anyway.

“Out of an abundance of caution, we ask that you consider changing your password on all services where you’ve used this password. You can change your Twitter password anytime by going to the password settings page.”

It’s sound advice, and it’s a step we have taken at the Internet Society with our own social media channels.

Yet, you may wonder what the point is in changing your password if a mistake like this can happen. If it happened once, surely it can happen again.

Yes, it could, there’s no doubt. We have compelling Continue reading

Dustin’s Internet Community Roadtrip: In Portland, Recognizing the Role of Women in Tech

Dustin Phillips, Co-Executive Director of ICANNWiki, is traveling across the United States in his red Toyota Corolla, making connections with the people who are making their communities – and the Internet – a better place. First stop: Portland, Oregon, where attended the ICANNWiki’s Girls in ICT Day Edit-a-thon and met up with Hack Oregon.

On the first day of the Internet Community Roadtrip, I headed down to Portland for ICANNWiki’s Girls in ICT Day Edit-a-thon. Making this my first stop was particularly special for me, because it is where I was introduced to the world of Internet governance three years ago. In my time at ICANNNWiki, I’ve experienced the important role that collaborative resources and tools play in the bottom-up, community-driven processes that keep the Internet free and global. At ICANNWiki, we focus on providing capacity-development, an informative resource and a sense of community. We build community by capturing and promoting the work that is being done, in order to provide well-deserved recognition and increase awareness for what is happening in the Internet governance ecosystem.

The stop in Portland focused on increasing the recognition for the role of women in the Internet’s history as planned, but also included Continue reading

Nutanix looks to turn its products into a platform — and attract CIOs

Every big vendor has had to start small and then grow into a major platform. Nutanix is no exception. And this week at its .NEXT user conference, the hyperconverged infrastructure (HCI) provider outlined a plan to move from being a niche vendor that makes VDI work better to becoming the next big enterprise platform vendor.In a recent interview, Nutanix CEO Dheeraj Pandey talked about the building blocks of becoming a major enterprise vendor. Nutanix started off with an integrated hardware/software appliance to deliver its HCI solution. Pandey referred to this as their “iPhone,” meaning it was a fully integrated and turnkey product where Nutanix owns the entire experience.To read this article in full, please click here