Modern Network Troubleshooting

I’ve reformatted and rebuilt my network troubleshooting live training for 2023, and am teaching it on the 26th of January (in three weeks). You can register at Safari Books Online. From the site:

The first way to troubleshoot faster is to not troubleshoot at all, or to build resilient networks. The first section of this class considers the nature of resilience, and how design tradeoffs result in different levels of resilience. The class then moves into a theoretical understanding of failures, how network resilience is measured, and how the Mean Time to Repair (MTTR) relates to human and machine-driven factors. One of these factors is the unintended consequences arising from abstractions, covered in the next section of the class.

The class then moves into troubleshooting proper, examining the half-split formal troubleshooting method and how it can be combined with more intuitive methods. This section also examines how network models can be used to guide the troubleshooting process. The class then covers two examples of troubleshooting reachability problems in a small network, and considers using ChaptGPT and other LLMs in the troubleshooting process. A third, more complex example is then covered in a data center fabric.

A short section on proving Continue reading

The RFC Process

I’ve just finished a seven-part series over at Packets Pushers about the process of writing and publishing an RFC. Even if you don’t ever plan to write a draft or participate in the IETF, this series will give you a better idea of the work that goes into creating new standards and IETF documents.

So … you have an idea you think would fit perfectly into the realm of the Internet Engineering Task Force (IETF)—but where do you start?

This, the second, post, will consider document formatting and two of the (sometimes) more difficult sections of an IETF draft to fill in.

There are other seemingly mystical concepts in the IETF process as well—for instance, what is a “document stream,” and what is a document’s “status?”

You’re almost ready to submit a shiny new document to the IETF for consideration, right? Not quite yet—we still need to deal with mandatory sections and language.

You cannot simply post a draft to the IETF repository and expect “someone, somewhere,” to take action.

The working group chairs asked if your draft should become a working group item, and the consensus was to accept! It might seem like your draft is home free Continue reading

Hedge 206: Taking Care of Yourself with Ethan Banks

As we reach the end of what has been a hard two-year stretch for what seems like the entire world, Ethan Banks joins Tom, Eyvonne, and Russ to talk about the importance of taking care of yourself. In the midst of radical changes, you can apply self-discipline to make your little part of the world a better place by keeping yourself sane, fit, and well-rested.

download

Thoughts on 2023

As we close out 2023, some random observations about engineering, culture, and life.

Network engineering needs help. I am hearing, from all over the place, that network engineering is “not cool.” There is a dearth of students entering the pipeline. College programs are struggling, and many organizations are struggling with a lack of engineering talent—in fact, I would guess the most common reason for companies to move to “the cloud” is because they cannot find anyone who knows how to build an operate a network any longer.

It probably didn’t help that for the last few years many “thought leaders” in the network engineering space have been saying there is no future in network engineering. It also doesn’t help that network engineering training has become stilted and … boring. Coders are off talking about how to solve problems. Robotics folks are working on cool projects that solve problems.

Network engineers are being taught how to spend less money and told to “find another career.”

I don’t know how we think we can sustain a healthy world of IT without network engineers.

And yes, I know there are folks who think networking problems are simple, easy enough to solve Continue reading

Weekend Reads 121523

NTT Data has opened a hotel at which it plans to watch people sleep, as part of a plan to gather – and of course sell – data about the snoozing habits of ten million people.

Business and technical leaders should prepare to focus on memory safety in software development, the US Cybersecurity and Infrastructure Agency (CISA) urged on Wednesday.

A proposed fork of the OpenPGP standard, called “LibrePGP” and initiated by GnuPG’s maintainer Werner Koch, has made a series of statements on its own website1 in order to justify its existence.

Cisco has quietly introduced changes to the licensing model for its Catalyst range, and will bring it to more products over time.

ICANN’s response to the European Union’s Network and Information Security Directive (NIS2) is a litmus test on whether its policy processes can address the needs of all stakeholders, instead of only satisfying the needs of the domain industry.

One of the joys of operational privacy professionals is getting that random, Friday afternoon Slack from someone on the product team asking, “Can we [insert questionable action] with our customer data?”

Lackluster security controls in one of Google’s cloud services for data scientists could allow hackers to Continue reading

Weekend Reads 120923

Google has revealed a new multilingual text vectorizer called RETVec (short for Resilient and Efficient Text Vectorizer) to help detect potentially harmful content such as spam and malicious emails in Gmail.

Carding has been around since the 1980s but has evolved to the point that even less experienced cybercriminals can now launch campaigns.

Enter Cilium’s advanced Border Gateway Protocol (BGP) implementation, powered by the GoBGP control plane, a solution that not only addresses these challenges but also adds unprecedented flexibility to your network configurations.

The most curious part of this is how people working inside the macroculture are the only folks who don’t understand what’s going on.

Efforts to convince remote workers to return to corporate offices appear to have stalled, based on data from the government, academia, and private-sector organizations.

Once in your home, different individuals have differing authority based on who they are. Family members have access to your whole home.

HP is squeezing more margin out of print customers, the result of a multi-year strategy to convert unprofitable business into something more lucrative, and says its subscription model is “locking” in people.

Microsoft helped Chinese state-run media outlets disseminate propaganda as part of previously unreported partnership agreements, Continue reading

Hedge 205: OId Engineering Quotes

For this month’s roundtable, Eyvonne, Tom, and I return to Addresses to Engineering Students by Harrington and Waddell. This book, published in 1912, is a “product of its time,” and hence deserves some trigger warnings. But it is also interesting to see how advice given to engineering students over 100 years ago holds up for today. Have engineering challenges, and the engineering life, changed all that much? What kinds of advice stand the test of time, what kinds do not?

download

Upcoming Pearson Class: Modern Network Troubleshooting

On the 26th of January, I’ll be teaching a webinar over at Safari Books Online (subscription service) called Modern Network Troubleshooting. From the blurb:

The first section of this class considers the nature of resilience, and how design tradeoffs result in different levels of resilience. The class then moves into a theoretical understanding of failures, how network resilience is measured, and how the Mean Time to Repair (MTTR) relates to human and machine-driven factors. One of these factors is the unintended consequences arising from abstractions, covered in the next section of the class.
The class then moves into troubleshooting proper, examining the half-split formal troubleshooting method and how it can be combined with more intuitive methods. This section also examines how network models can be used to guide the troubleshooting process. The class then covers two examples of troubleshooting reachability problems in a small network, and considers using ChaptGPT and other LLMs in the troubleshooting process. A third, more complex example is then covered in a data center fabric.

Register here.

Weekend Reads 120223

Yet despite the constant accretion of new tools to solve new problems, the most common root cause of serious cybersecurity incidents remains failed processes.

The House of Representatives’ failure to spike a federal “kill switch” mandate means that outside of a political miracle, all new vehicles from 2026 onward will be required to incorporate “advanced drunk and impaired driving prevention technology.”

Gone are the days when a car was a dumb machine you turned on and drove from A to B. Today it’s a smartphone on wheels, and your data is possibly being taken for a ride.

APT29, believed to be an espionage group from Russia, became known for launching targeted attacks against organizations in Ukraine.

Mozilla has slapped its “Privacy Not Included” labels on several products from Google, Amazon and Microsoft – just in time for Christmas shopping.

Despite more than a decade of reminding, prodding, and downright nagging, a surprising number of developers still can’t bring themselves to keep their code free of credentials that provide the keys to their kingdoms to anyone who takes the time to look for them.

Special report Web advert blockers and other Chrome extensions will stop working by June 2024 unless Continue reading

Hedge 204: Terry Slattery on Network Automation

Terry Slattery joins Tom and Russ to continue the conversation on network automation—and why networks are not as automated as they should be. This is part one of a two-part series; the first part of this conversation was posted as episode 203.

download

Weekend Reads 111923

Bad queries tend to propagate to the root zone due to the hierarchical nature of DNS, so studying traffic at a root server can provide key insights into overall network usage.

This blog covers an interesting case of suspected abuse in a gTLD registry between February and April 2023.

YouTube wants its pound of flesh. Disable your ad blocker or pay for Premium, warns a new message being shown to an unsuspecting test audience, with the barely hidden subtext of “you freeloading scum.”

The shift towards chiplet architecture is inevitable for almost all high-end CPUs/GPUs, accelerators, and networking silicon vendors. It is not a question of ‘if’ but ‘when’.

The Global Coalition on Telecommunications (GCOT) purports to be about synching up how the five countries approach telecoms. The scope of corporation includes information sharing, joint R&D, funding alignment, the development of standards, skills, supply chain diversification, security, and 6G, so says the release.

Securing mainframes remains top of mind, with 61% of mainframe and IT professionals ranking security as the top problem they are facing, according to BMC’s annual survey of mainframe users for 2023.

Gartner has raised the specter of departments outside of tech running their own IT Continue reading

Hedge 203: Terry Slattery on Network Automation

Terry Slattery joins Tom and Russ to continue the conversation on network automation—and why networks are not as automated as they should be. This is part one of a two-part series; the second part will be published in two weeks as Hedge episode 204.

download

Hedge 202: Internet Governance with George Michaelson

How is the Internet governed? Who sets the rules for the Internet, civil society, and government control? How much input should techies have, and how much should government control things? These are questions we don’t often ask, and yet are crucial to building and operating networks connected to the global Internet. George Michaelson joins Toms and Russ to talk about Internet governance—including contrary views of where things should be versus where they are.

download

On the ‘net: Two on AI

I occasionally write over at Mind Matters on topics “other than technical.” Here are my two latest posts over there.

But what if you could steal something just as valuable as the contents of a lady’s handbag without anyone suspecting it and without impacting your user’s trust? What if you could take private information about millions of people, across the world, using that information to create what Shoshana Zuboff calls “behavioral surplus?” What if you could use that information to discover — and shape — people’s preferences without them even realizing it is happening? What if you could sell your user’s attention to the highest bidder?

While it seems evident that content created by a user prompt should not be copyrightable by the user, what about the designer and operator of the AI system? It might seem reasonable to infer the humans who create a system that, in turn, creates new “works” should be able to copyright those works.

On the ‘net: Model Based Thinking

Running a little late on cross posting stuff from Packet Pushers … but I suppose better late than never.

Model-based thinking can help engineers understand protocols and networks at a fundamental level. My previous post on network models focused on the Four Things model, an alternative to the OSI model for understanding network transport. I’ll continue with the Four Things, this time with a focus on routing.

Weekend Reads 110323

With security, the battle between good and evil is always a swinging pendulum. Traditionally, the shrewdness of the attack has depended on the skill of the attacker and the sophistication of the arsenal.

While cyberattacks on websites receive much attention, there are often unaddressed risks that can lead to businesses facing lawsuits and privacy violations even in the absence of hacking incidents.

A new login technique is becoming available in 2023: the passkey. The passkey promises to solve phishing and prevent password reuse.

Security researchers have discovered what they believe may be a government attempt to covertly wiretap an instant messaging service in Germany — an attempt that was blown because the potential intercepting authorities failed to reissue a TLS certificate.

Artists suing generative artificial intelligence art generators have hit a stumbling block in a first-of-its-kind lawsuit over the uncompensated and unauthorized use of billions of images downloaded from the internet to train AI systems, with a federal judge’s dismissal of most claims.

Professional artists and photographers annoyed at generative AI firms using their work to train their technology may soon have an effective way to respond that doesn’t involve going to the courts.

Intel is shedding its silicon photonics transceiver module business as part of restructuring and cost-cutting measures, offloading it to manufacturing company Jabil.

Domain Name System (DNS) abuse stands has proven a constant in the internet threat landscape, posing risk to the overall digital trust.

SpaceX is equipping its new satellites with inter-satellite laser links (ISLLs). They now have over 8,000 optical terminals in orbit (3 per satellite) and they communicate at up to 100 Gbps.

Hedge 201: Roundtable

It’s time to gather round the hedge and discuss whatever Eyvonne, Tom, and Russ find interesting! In this episode we discuss business logic vulnerabilities, and how we often forget to think outside the box to understand the attack surfaces that matter. We also discuss upcoming network speed increases like Wi-Fi 7 and 800G Ethernet. Do we really need these speeds, or are we just getting caught up in a hype cycle?

download

Weekend Reads 102723

Passkeys are appearing more and more in tech news, with support for them increasing. Since many administrators test out new technologies themselves first, we at CIS embarked on a short project to see what happened when an intern with our CTO team had the opportunity to implement passkeys.

Draft proposals setting a “remuneration obligation” for digital platforms started to pop up in the Brazilian congress after Australia adopted its own News Media Bargaining Code.

And can companies finally trust their data to cloud providers and actually trust that CC removes the need to worry about the cloud provider as a sub-processor with access to the data?

Google says it plans to prototype a technique to mask IP addresses via network proxies in future versions of its Chrome browser, a privacy protection similar to Apple’s iCloud Private Relay service for its Safari browser.

The cryptanalytic threat of quantum computing, particularly the “store data; decrypt later” approach, is drawing ever nearer. Unsurprisingly, the U.S. government is among those most alert to the danger.

A common critique of HTTP/3 and QUIC is that they primarily benefit the big players and companies (for example, Google and Meta), who often control one or even Continue reading

Hedge 200: Automation with Ethan Banks

We’ve been on a long streak of discussions about automation, why it works, why it isn’t working, and what the networking industry can do about it. For this episode, we’re joined by the indubitable Ethan Banks. If you don’t think there’s anything left to say, you’ve not yet listened to Ethan!

download

Hedge 199: Automation with Carl Buchmann

Automation is a big topic–folks had a lot of feedback on our first couple of Hedge episodes on the topic. We return to automation in this episode of the Hedge with Carl Buchmann to discuss one effort at unifying automation with humble beginnings.

download