How GitHub Learned How Hard Distributed Systems Are

Anne Baretta found a great video describing the October 2018 GitHub failure. Here’s the TL&DW:

• The failure was caused by a short (~ 1 minute) disconnect of the primary data center
• The database replicas failed over to the secondary data center, but that failover was never tested and of course some stuff didn’t work.
• In the meantime, batch jobs modified data in the primary data center, making the two replicas out-of-sync.
• It took them over 24 hours to clean up the mess.

How GitHub Learned How Hard Distributed Systems Are

Anne Baretta found a great video describing the October 2018 GitHub failure. Here’s the TL&DW:

• The failure was caused by a short (~ 1 minute) disconnect of the primary data center
• The database replicas failed over to the secondary data center, but that failover was never tested and of course some stuff didn’t work.
• In the meantime, batch jobs modified data in the primary data center, making the two replicas out-of-sync.
• It took them over 24 hours to clean up the mess.

Engagement Farming

One of my readers asked for my opinion about the following masterpiece posted on (where else) LinkedIn¹:

Engagement Farming

One of my readers asked for my opinion about the following masterpiece posted on (where else) LinkedIn¹:

netlab 1.6.1: BGP Reports, Markdown Support

We added just a few small features in netlab release 1.6.1¹:

• Markdown reports: netlab report command can produce Markdown-formatted reports, making it extremely easy to include them in your documentation (assuming you’re using Markdown to write it)
• If you’re using BGP in your labs, you can generate reports on BGP autonomous systems and BGP neighbors.
• I made locations of default files configurable. I’m using this feature in large projects where I want to have a shared set of project-wide defaults for topologies stored in different directories.

netlab 1.6.1: BGP Reports, Markdown Support

We added just a few small features in netlab release 1.6.1¹:

• Markdown reports: netlab report command can produce Markdown-formatted reports, making it extremely easy to include them in your documentation (assuming you’re using Markdown to write it)
• If you’re using BGP in your labs, you can generate reports on BGP autonomous systems and BGP neighbors.
• I made locations of default files configurable. I’m using this feature in large projects where I want to have a shared set of project-wide defaults for topologies stored in different directories.

Getting Comfortable with the Command Line

More than a dozen years after the SDN brouhaha erupted, some people still haven’t got the memo on the obsolescence of CLI. For example, Julia Evans tries to make people comfortable with the command line. Has nobody told her it’s like teaching COBOL?

On a more serious note: you OUGHT TO master Linux CLI and be comfortable using CLI commands on network devices and servers. Her article has tons of useful tips and is definitely worth reading.

Getting Comfortable with the Command Line

More than a dozen years after the SDN brouhaha erupted, some people still haven’t got the memo on the obsolescence of CLI. For example, Julia Evans tries to make people comfortable with the command line. Has nobody told her it’s like teaching COBOL?

On a more serious note: you OUGHT TO master Linux CLI and be comfortable using CLI commands on network devices and servers. Her article has tons of useful tips and is definitely worth reading.

Free Subscription No Longer Needed to Watch the ipSpace.net Videos

I’m publishing a link to a free ipSpace.net video several times each month, usually with a notice saying you need free subscription to watch the video. I had to put that limitation in place when I was hosting videos on AWS S3 – unlimited streaming could explode my AWS bill.

Recently I moved the video storage to Cloudflare R2. Cloudflare claims they will never charge egress fees, and as long as that’s true (and they don’t start chasing me for generating too much traffic) I see no reason to bother you with registration and login procedures – starting immediately, you can watch the free ipSpace.net videos without an ipSpace.net account.

Free Subscription No Longer Needed to Watch the ipSpace.net Videos

I’m publishing a link to a free ipSpace.net video several times each month, usually with a notice saying you need free subscription to watch the video. I had to put that limitation in place when I was hosting videos on AWS S3 – unlimited streaming could explode my AWS bill.

Recently I moved the video storage to Cloudflare R2. Cloudflare claims they will never charge egress fees, and as long as that’s true (and they don’t start chasing me for generating too much traffic) I see no reason to bother you with registration and login procedures – starting immediately, you can watch the free ipSpace.net videos without an ipSpace.net account.

Worth Reading: Networking for AI Workloads

Sharada Yeluri (Senior Director of Engineering at Juniper Networks) wrote a long article describing the connectivity requirements of AI workloads and new approaches to Ethernet fabrics. Definitely worth reading if you’re interested in these topics.

Worth Reading: Networking for AI Workloads

Sharada Yeluri (Senior Director of Engineering at Juniper Networks) wrote a long article describing the connectivity requirements of AI workloads and new approaches to Ethernet fabrics. Definitely worth reading if you’re interested in these topics.

New Project: BGP Hands-On Labs

Approximately 30 years ago I managed to persuade the powers-that-be within Cisco’s European training organization that they needed a deep-dive BGP course, resulting in a 3 (later 5) day Advanced BGP Configuration and Troubleshooting (ABCT) course¹. I was delivering that course for close to a decade, and gradually built a decent story explaining the reasoning and use cases behind most of (then available) BGP features, from simple EBGP sessions to BGP route reflectors and communities².

Now imagine having more than a dozen hands-on labs that go with the “BGP from rookie to hero” story available for any platform of your choice³. I plan to make that work (eventually) as an open-source project that you’ll be able to download and run free-of-charge.

New Project: BGP Hands-On Labs

Long story short: I decided to create open-source BGP configuration labs, and (so far) created a superset of labs we used in an ancient Advanced BGP Configuration and Troubleshooting (ABCT) course

netlab 1.6.0: New Commands, Reports, and External Connectivity

netlab release 1.6.0 has (probably) the longest release notes so far as it contains so many user-visible new features including:

New Commands

Some users were complaining how complex it was to use netlab create command to create graphs, inspect data structures, or create custom reports. They might find the new commands easier to use:

netlab 1.6.0: New Commands, Reports, and External Connectivity

netlab release 1.6.0 has (probably) the longest release notes so far as it contains so many user-visible new features including:

New Commands

Some users were complaining how complex it was to use netlab create command to create graphs, inspect data structures, or create custom reports. They might find the new commands easier to use:

Worth Reading: MP-TCP in Hybrid Access Networks

Wouldn’t it be nice if your home router (CPE) could use DSL (or slow-speed fibre) and LTE connection at the same time? Even better: run a single TCP session over both links? The answer to both questions is YES, of course it could do that, if only your service provider would be interested in giving you that option.

We solved similar problems with multilink PPP in the networking antiquity, today you could use a CPE with an MP-TCP proxy combined with a Hybrid Access Gateway in the service provider network. For more details, read the excellent Increasing broadband reach with Hybrid Access Networks article by prof. Olivier Bonaventure and his team.

Worth Reading: MP-TCP in Hybrid Access Networks

Wouldn’t it be nice if your home router (CPE) could use DSL (or slow-speed fibre) and LTE connection at the same time? Even better: run a single TCP session over both links? The answer to both questions is YES, of course it could do that, if only your service provider would be interested in giving you that option.

We solved similar problems with multilink PPP in the networking antiquity, today you could use a CPE with an MP-TCP proxy combined with a Hybrid Access Gateway in the service provider network. For more details, read the excellent Increasing broadband reach with Hybrid Access Networks article by prof. Olivier Bonaventure and his team.

Worth Reading: Eyes Like Saucers

Gerben Wierda published a nice description of common reactions to new unicorn-dust-based technologies:

• Eyes that glaze over
• Eyes like saucers
• Eyes that narrow

He uses generative AI as an example to explain why it might be a bad idea that people in the first two categories make strategic decisions, but of course nothing ever stops people desperately believing in vendor fairy tales, including long-distance vMotion, SDN or intent-based networking.

Worth Reading: Eyes Like Saucers

Gerben Wierda published a nice description of common reactions to new unicorn-dust-based technologies:

• Eyes that glaze over
• Eyes like saucers
• Eyes that narrow

He uses generative AI as an example to explain why it might be a bad idea that people in the first two categories make strategic decisions, but of course nothing ever stops people desperately believing in vendor fairy tales, including long-distance vMotion, SDN or intent-based networking.