It Always Takes Too Long

It always takes longer to find a problem than it should. Moving through the troubleshooting process often feels like swimming in molasses—it’s never fast enough or far enough to get the application back up and running before that crucial deadline. The “swimming in molasses effect” doesn’t end when the problem is found out, either—repairing the problem requires juggling a thousand variables, most of which are unknown, combined with the wit and sagacity of a soothsayer to work with vendors, code releases, and unintended consequences.

It’s enough to make a network engineer want to find a mountain top and assume an all-knowing pose—even if they don’t know anything at all.
The problem of taking longer, though, applies in every area of computer networking. It takes too long for the packet to get there, it takes to long for the routing protocol to converge, it takes too long to support a new application or server. It takes so long to create and validate a network design change that the hardware, software and processes created are obsolete before they are used.

Why does it always take too long? A short story often told to me by my Grandfather—a farmer—might help.

One morning a Continue reading

Weekend Reads 073021

One of the hottest areas of scientific research that peripherally will affect every tech industry is battery research. It seems like every year there are big breakthroughs in battery capability. Today I look at four of the recent announcements.

Tech companies have a long history of mishandling contacts, and the industry has been slow to give people more control.

However, ColdQuanta is betting that a modality that its founders and engineers have been working on for 15 years – cold atom – will establish itself as a method that will establish itself as quantum computing moves from a developing technology to an established global market.

Web infrastructure and website security company Cloudflare last month fixed a critical vulnerability in its CDNJS library that’s used by 12.7% of all websites on the internet.

The end-users of these platforms are not in control of their virtual presence; if anything, they are at the mercy of these platforms to a large extent.

Cyber hygiene and patching are key measures towards protecting data and systems. However, it’s not always possible or practical to patch when vulnerabilities and associated patches are announced. This problem gives rise to day one exploits.

Data-localization policies are spreading Continue reading

Hedge 93: Dinesh Dutt and Observability

We talk a lot of about telemetry in the networking world, but generally as a set of disconnected things we measure, rather than as an entire system. We also tend to think about what we can measure, rather than what is useful to measure. Dinesh Dutt argues we should be thinking about observability, and how to see the network as a system. Listen in as Dinesh, Tom, And Russ talk about observability, telemetry, and Dinesh’s open source network observability project.

download

How to Listen to the Hedge

The Hedge is over 90 episodes now … I’m a little biased, but I believe we’re building the best content in network engineering—a good blend of soft skills, Internet policy, research, open source projects, and relevant technical content. You can always follow the Hedge here on Rule 11, of course, but it’s also available on a number of services, including—

• iTunes
• Spotify
• Deezer
• Castbox
• Google
• Luminary
• Pocket Casts

I think it’s also available on Amazon Music, but I don’t subscribe to that service so I can’t see it. You can check the Podcast Directory for other services, as well. If you enjoy the Hedge, please post a positive rating so others can find it more easily.

Upcoming Live Webinar: Data Center Fabrics

I’ll be teaching a three-hour live webinar on data center fabrics on the 20th of August—

Data centers are the foundation of the cloud, whether private, public, on the edge, or in the center of the network. This training will focus on topologies and control planes, including scale, performance, and centralization. This training is important for network designers and operators who want to understand the elements of data center design that apply across all hardware and software types.

Register here.

Leveraging Similarities

We tend to think every technology and every product is roughly unique—so we tend to stay up late at night looking at packet captures and learning how to configure each product individually, and chasing new ones as if they are the brightest new idea (or, in marketing terms, the best thing since sliced bread). Reality check: they aren’t. This applies across life, of course, but especially to technology. From a recent article—

Whenever I start learning a new programming language, I focus on defining variables, writing a statement, and evaluating expressions. Once I have a general understanding of those concepts, I can usually figure out the rest on my own. Most programming languages have some similarities, so once you know one programming language, learning the next one is a matter of figuring out the unique details and recognizing the differences.

RFC1925 rule 11 states—

Every old idea will be proposed again with a different name and a different presentation, regardless of whether it works.

Rule 11 isn’t just a funny saying—rule 11 is your friend. If want to learn new things quickly, learn rule 11 first. A basic understanding of the theory of networking will carry across all products, all Continue reading

Juniper Master Class: Disaggregation Pros and Cons

On the 28th—in two days—I’m doing a master class over at Juniper on DC fabric disaggregation. I’ll spend some time defining the concept (there are two different ideas we use the word disaggregation to describe), and then consider some of the positive and negative aspects of disaggregation. This is a one hour session, and it’s free. Register here.

Weekend Reads 072321

The next tech talent wars may be less about the free stuff, and more about the freedom to work from anywhere in the world. Those famously expensive Silicon Valley campuses that double as adult playgrounds, with their nap pods and herb gardens and bike-shares, are competing with a newfound love for the home office.

There are some features in any architecture that are essential, foundational, and non-negotiable. Right up to the moment that some clever architect shows us that this is not so.

Looking at the Resource Public Key Infrastructure (RPKI) landscape today, it is vastly different from two to three years ago. At the time, resource holders around the world had created a considerable amount of Route Origin Authorization (ROAs), but actually using RPKI data to perform Route Origin Validation (ROV) was only done by a handful of networks

A newly discovered breed of cyber assault is threatening corporate networks. Dubbed “FragAttacks” (Fragmentation and Aggregation Attacks) by Mathy Vanhoef, the researcher who discovered them, these security breaches are a subcategory of digital airborne attacks performed over Wi-Fi networks.

While there’s enormous promise in AI-powered tools and machine learning, they are very much a double-edged sword. Cybercriminals and other threat Continue reading

Hedge 92: The IETF isn’t the Standards Police

In most areas of life, where the are standards, there is some kind of enforcing agency. For instance, there are water standards, and there is a water department that enforces these standards. There are electrical standards, and there is an entire infrastructure of organizations that make certain the fewest number of people are electrocuted as possible each year. What about Internet standards? Most people are surprised when they realize there is no such thing as a “standards police” in the Internet.

Listen in as George Michaelson, Evyonne Sharp, Tom Ammon, and Russ White discuss the reality of standards enforcement in the Internet ecosystem.

download

Whatever it is, you need more (RFC1925 rule 9)

There is never enough. Whatever you name in the world of networking, there is simply not enough. There are not enough ports. There is not enough speed. There is not enough bandwidth. Many times, the problem of “not enough” manifests itself as “too much”—there is too much buffering and there are too many packets being dropped. Not so long ago, the Internet community decided there were not enough IP addresses and decided to expand the address space from 32 bits in IPv4 to 128 bits in IPv6. The IPv6 address space is almost unimaginably huge—2 to the 128^th power is about 340 trillion, trillion, trillion addresses. That is enough to provide addresses to stacks of 10 billion computers blanketing the entire Earth. Even a single subnet of this space is enough to provide addresses for a full data center where hundreds of virtual machines are being created every minute; each /64 (the default allocation size for an IPv6 address) contains 4 billion IPv4 address spaces.

But… what if the current IPv6 address space simply is not enough? Engineers working in the IETF have created two different solutions over the years for just this eventuality. In 1994 RFC1606 provided a Continue reading

Weekend Reads 071621

Social media platforms like Instagram and Facebook have become key places for businesses to communicate with customers and even sell directly to consumers. Yet when it comes to actually making a purchase, do consumers trust a social media site over a domain?

Ransomware payouts are putting the squeeze on cyber-insurance companies and resulting in higher premiums for organizations that want protection against the threat.

Christopher Belfi was waiting tables in a lakeside resort near this Upstate New York town a decade ago when he got the career break he’d been waiting for — an invitation to work at a semiconductor factory

Until we can solve the cybersecurity problem for the user at home, threats will remain a concern even for enterprises, with many having large numbers of work-from-home employees.

Prompted in part by devastating attacks such as those on SolarWinds Orion, Microsoft Exchange, and Colonial Pipeline, the White House issued an executive order on cybersecurity in May.

The InfiniBand interconnect emerged from the ashes of a fight about the future of server I/O at the end of the last millennium, and instead of becoming that generic I/O it became a low latency, high bandwidth interconnect used for high performance computing.

Continue reading

Hedge 91: Leslie Daigle and IP Addresses Acting Badly

What if you could connect a lot of devices to the Internet—without any kind of firewall or other protection—and observe attackers trying to find their way “in?” What might you learn from such an exercise? One thing you might learn is a lot of attacks seem to originate from within a relatively small group of IP addresses—IP addresses acing badly. Listen in as Leslie Daigle of Thinking Cat and the Techsequences podcast, Tom Ammon, and Russ White discuss just such an experiment and its results.

download

Free Speech is More than Words

A couple of weeks ago, I joined Leslie Daigle and Alexa Reid on Techsequences to talk about free speech and the physical platform—does the right to free speech include the right to build and operate physical facilities like printing presses and web hosting? I argue it does. Listen in if you want to hear my argument, and how this relates to situations such as the “takedown” of Parler.

Listen here

NATs, PATs, and Network Hygiene

While reading a research paper on address spoofing from 2019, I ran into this on NAT (really PAT) failures—

In the first failure mode, the NAT simply forwards the packets with the spoofed source address (the victim) intact … In the second failure mode, the NAT rewrites the source address to the NAT’s publicly routable address, and forwards the packet to the amplifier. When the server replies, the NAT system does the inverse translation of the source address, expecting to deliver the packet to an internal system. However, because the mapping is between two routable addresses external to the NAT, the packet is routed by the NAT towards the victim.

The authors state 49% of the NATs they discovered in their investigation of spoofed addresses fail in one of these two ways. From what I remember way back when the first NAT/PAT device (the PIX) was deployed in the real world (I worked in TAC at the time), there was a lot of discussion about what a firewall should do with packets sourced from addresses not indicated anywhere.

If I have an access list including 192.168.1.0/24, and I get a packet sourced from 192.168.2.24, Continue reading

Controversial Reads 071021

According to the company’s market research, just about every demographic wants more data privacy: young, old, male, female, urban, rural. Public polling backs that up, though the results vary based on how the question is asked. One recent survey found that “93 percent of Americans would switch to a company that prioritizes data privacy if given the option.”

Once upon a very different internet era, law professor Tim Wu rose to intellectual prominence warning of the doom to come without “net neutrality,” a term he coined.

In a blog post on March 3, Google announced that it would be removing third-party cookies from its Chrome browser—a decision that would effectively end use of third-party cookies. Google also pledged to avoid any other technology for tracking individuals as they browse the web.

The $35 million contract given to SKDKnickerbocker was controversial. The state controller refused to pay it, pointing to the fact that there was no authorization in the budget for that spending.

The Judiciary Committee of the U.S. House of Representatives recently released a comprehensive series of bills designed to curb the excesses of Big Tech. One of them, the Platform Competition and Opportunity Act, addresses one of Continue reading

Weekend Reads 070921

A long-standing, generally accepted norm in the computing field distinguishes between software interfaces and implementations: Programmers should have to write their own implementing code, but they should be free to reimplement other developers’ program interfaces.

The traditional approach to statistical disclosure control (SDC) for privacy protection is utility-first. Since the 1970s, national statistical institutes have been using anonymization methods with heuristic parameter choice and suitable utility preservation properties to protect data before release.

Shared libraries encourage code reuse, promote consistency across teams, and ultimately improve product velocity and quality. But application developers are still left to choose the right libraries, figure out how to correctly configure them, and wire everything together.

When October 5 came, there was no vulnerability advisory being published and I still had not heard a CVSS or CVE for the issue, so I reached out again to their PSIRT who this time replied that the release had been postponed until October 14th now due to a delay in QA.

Organizations relying on traditional signature-based tools to detect security threats would likely have missed roughly three-quarters of malware samples that hit their networks and systems last quarter, a new analysis shows.

But as attacks have increased in Continue reading

Hedge 90: Andrew Wertkin and a Naïve Reliance on Automation

Automation is surely one of the best things to come to the networking world—the ability to consistently apply a set of changes across a wide array of network devices has speed at which network engineers can respond to customer requests, increased the security of the network, and reduced the number of hours required to build and maintain large-scale systems. There are downsides to automation, as well—particularly when operators begin to rely on automation to solve problems that really should be solved someplace else.

In this episode of the Hedge, Andrew Wertkin from Bluecat Networks joins Tom Ammon and Russ White to discuss the naïve reliance on automation.

download