A While, Career update and Ends with various Junos load commands.

It has been quite some time since my last blog post. The past few months have been busy, leaving little time to write. I am happy to share that I have started working as a site-reliability developer at Oracle. While it has only been a short while, I am enjoying the work.

Reflecting on the past year, I am happy to say it has been productive overall. I have had the opportunity to collaborate with various networking and software teams, which has taught me a lot about high-scale traffic patterns and migrations. Working in an operations role has been beneficial since it requires constant fire-fighting and documentation for improvements. This has made me more aware of traffic patterns and ways to improve reliability.

What can be improved?

However, I could benefit from more study on the software architecture and distributed system design parameters from a business perspective. I want to write more this year; the first half was better, but lacked in the second half, where most of the writing was private.

From a personal standpoint, 2024 needs more travel absolutely and some improvements in garden automation for summer 2024

Other updates

Finally, I have been working on some JNCIA-Devops Continue reading

About password security and credential stuffing

Common Falsehoods About Password Security and Credential Stuffing

TechCrunch published an article that gives class-action lawyers suing 23andMe a mouthpiece to editorialize about password security practices, masquerading as a news article. The upshot of the ~article~ editorial is this:

  1. A large number of 23andMe’s customers were subject to a credential stuffing attack.
  2. Design choices in 23andMe’s site allowed other customer data to be exposed via accounts compromised in the credential stuffing attack.
  3. 23andMe’s mitigations for credential stuffing attacks were inadequate.
  4. 23andMe should be held liable for this.

I want to focus here on the third point: credential stuffing attack mitigations. I’ve worked quite a bit on analyzing large credential stuffing attacks and recommending mitigations for them. I also served as a technical escalation point for customers who had a wide variety of strongly held false beliefs about password security credential stuffing mitigation. In reading various social media responses the 23andMe case, I see all these false beliefs turning up again. Let’s have a look at some.

False: Rate limiting login attempts is a great mitigation

The first question we need to answer here is: what do you mean by rate limiting? Usually there are two main rate limits that people Continue reading

Meshtastic quick setup

I wanted some nice offline mid range chat app, for when I don’t have data, or data roaming is too expensive. I also want it to work for people who are not amateur radio licensed, since my girlfriend stubbornly refuses to be interested in that.

Looks like the answer I’m looking for is Meshtastic, preferably with LoRalora]. I bought a couple of Heltec V3 ESP32 LoRa OLED and the matching case.

Maybe I’ll buy a battery, but I’m fine just powering it from a USB power bank.

The documentation makes a fair bit of assumptions about the user knowing the name for what they want, and what firmware provides what.

In short, what I think I want is to ignore the Heltec firmware, and instead just treat the Heltec V3 as the hardware that Meshtastic runs on.

The recommended way to flash, and for some cases even use, is the Meshtastic Web UI. It uses browser integration for serial ports and bluetooth. A nice idea, but it was extremely unreliable for me. The flasher worked for one device, but not the other. The chat client never worked at all.

Here’s what worked reliably for me:

  1. Download “stable” firmware Continue reading

With “Big Chip,” China Lays Out Aspirations For Waferscale

The end of Moore’s Law – the real Moore’s Law where transistors get cheaper and faster with every process shrink – is making chip makers crazy. …

The post With “Big Chip,” China Lays Out Aspirations For Waferscale first appeared on The Next Platform.

With “Big Chip,” China Lays Out Aspirations For Waferscale was written by Timothy Prickett Morgan at The Next Platform.

Selectively Replacing Resources with Pulumi

Because Pulumi operates declaratively, you can write a Pulumi program that you can safely run (via pulumi up) multiple times. If no changes are needed—meaning that the current state of the infrastructure matches what you’ve defined in your Pulumi program—then nothing happens. If only one resource needs to be updated, then it will update only that one resource (and any dependencies, if there are any). There may be times, however, when you want to force the replacement of specific resources. In this post, I’ll show you how to target specific resources for replacement when using Pulumi.

Here’s an example: I use Pulumi to manage my AWS-based lab resources, including SSH bastion hosts. However, because my code uses a dynamic AMI lookup, I’ve instructed Pulumi to ignore changes in the AMI ID for the bastion hosts (by appending pulumi.IgnoreChanges([]string{"ami"}) as a resource option). This gives me the control over when the bastion hosts get replaced, instead of Pulumi wanting to replace them every time the AMI ID changes.

With this in place, then, how do I tell Pulumi that I’m ready to replace the bastion hosts? Tearing down the entire stack isn’t an option. Fortunately, the pulumi CLI Continue reading

Ethernet Switching Bucks The Server Recession Trend

You might be thinking that with all of the investment in AI systems these days that the boom in InfiniBand interconnect sales would be eating into sales of high-end Ethernet interconnects in the datacenter. …

The post Ethernet Switching Bucks The Server Recession Trend first appeared on The Next Platform.

Ethernet Switching Bucks The Server Recession Trend was written by Timothy Prickett Morgan at The Next Platform.

BGP EVPN Part II: Network Virtualization Overlay with BGP EVPN and VXLAN – Introduction

In Figure 1-1, we have a routed 3-stage Clos Fabric, where all Inter-Switch links are routed point-to-point layer-3 connections. As explained in previous sections, a switched layer-2 network with an STP control plane allows only one active path per VLAN/Instance and VLAN-based traffic load sharing. Due to the Equal Cost Multi-Path (ECMP) supported by routing protocols, a routed Clos Fabric enables flow-based traffic load balancing using all links from the ingress leaf via the spine layer down to the egress leaf. The convergence time for routing protocols is faster and less disruptive than STP topology change. Besides, a routed Clos Fabric architecture allows horizontal bandwidth scaling. We can increase the overall bandwidth capacity between switches, by adding a new spine switch. Dynamic routing protocols allow standalone and virtualized devices lossless In-Service Software Update (ISSU) by advertising infinite metrics or withdrawing all advertised routes.

But how do we stretch layer-2 segments over layer-3 infrastructure in a Multipoint-to-Multipoint manner, allowing tenant isolation and routing between segments? The answer relies on the Network Virtualization Overlay (NVO3) framework. 

BGP EVPN, as an NVO3 control plane protocol, uses EVPN Route Types (RT) in update messages for identifying the type of advertised EVPN NLRIs (Network Continue reading

Public Cloud Networking Hands-On Exercises

I got this request from someone who just missed the opportunity to buy the ipSpace.net subscription (or so he claims) earlier today

I am inspired to learn AWS advanced networking concepts and came across your website and webinar resources. But I cannot access it.

That is not exactly true. I wrote more than 4000 blog posts in the past, and some of them dealt with public cloud networking. There are also the free videos, some of them addressing public cloud networking.

Read more …

Public Cloud Networking Hands-On Exercises

I got this request from someone who just missed the opportunity to buy the ipSpace.net subscription (or so he claims) earlier today

I am inspired to learn AWS advanced networking concepts and came across your website and webinar resources. But I cannot access it.

That is not exactly true. I wrote more than 4000 blog posts in the past, and some of them dealt with public cloud networking. There are also the free videos, some of them addressing public cloud networking.

Read more …

EVPN Deepdive Route Types 2 and 3

In my last post on Configuring EVPN, we setup EVPN but configured no services. In this post we will configure a basic L2 service so we can dive into the different EVPN route types. This post will cover route type 2 and 3 together as you will commonly see these together. This post will cover:

  • Discovery of VTEPs.
  • How to map a VLAN to a VNI.
  • Automatic generation of RD and RT.
  • Advertising MAC- and optionally IP address (route type 2).
  • Ingress replication with dynamic discovery of VTEPs (route type 3).

The topology we will use for this post is shown below:

Before diving into configuration, let’s discuss something that is often overlooked, VTEP discovery.

VTEP discovery

Without EVPN, VXLAN uses flood and learn behavior for discovery of VTEPs. This means that any host sending VXLAN frames would be considered a trusted VTEP in the network. This is obviously not great from a security perspective. When using EVPN, adding VTEPs is based on BGP messages. A VTEP will learn about other VTEPs based on these BGP updates. It’s not a specific route type, but rather any type of EVPN message. This makes it more difficult to add a rogue Continue reading

Modern Network Troubleshooting

I’ve reformatted and rebuilt my network troubleshooting live training for 2023, and am teaching it on the 26th of January (in three weeks). You can register at Safari Books Online. From the site:

The first way to troubleshoot faster is to not troubleshoot at all, or to build resilient networks. The first section of this class considers the nature of resilience, and how design tradeoffs result in different levels of resilience. The class then moves into a theoretical understanding of failures, how network resilience is measured, and how the Mean Time to Repair (MTTR) relates to human and machine-driven factors. One of these factors is the unintended consequences arising from abstractions, covered in the next section of the class.

The class then moves into troubleshooting proper, examining the half-split formal troubleshooting method and how it can be combined with more intuitive methods. This section also examines how network models can be used to guide the troubleshooting process. The class then covers two examples of troubleshooting reachability problems in a small network, and considers using ChaptGPT and other LLMs in the troubleshooting process. A third, more complex example is then covered in a data center fabric.

A short section on proving Continue reading

VRF Without Route Target – Will the Route Be Exported?

Yesterday I posted a tricky question to Twitter. If you have a working VPNv4 environment and create a VRF with only a Route Distinguisher (RD) but without Route Targets (RT), will the route be exported? The answer may surprise you! The configuration supplied in the question was similar to the one below:

vrf definition QUIZ
 rd 198.51.100.1:100 
 !
 address-family ipv4
 exit-address-family
!
interface GigabitEthernet2
 vrf forwarding QUIZ
 ip address 203.0.113.1 255.255.255.0
!
router bgp 65000
 !
 address-family ipv4 vrf QUIZ
  network 203.0.113.0

Notice how this VRF has a RD but no RT. Will this router, PE1, advertise the route into VPNv4? Most would say no, but the answer is yes. Let’s first check that we see the route locally on PE1 in VRF QUIZ:

PE1#show bgp vpnv4 uni vrf QUIZ 203.0.113.0
BGP routing table entry for 198.51.100.1:100:203.0.113.0/24, version 4
Paths: (1 available, best #1, table QUIZ)
  Advertised to update-groups:
     1         
  Refresh Epoch 1
  Local
    0.0.0.0 (via vrf QUIZ) from 0.0.0.0 (198.51.100.1)
      Origin IGP, metric 0, localpref 100, weight 32768, valid, sourced, local, best
      mpls  Continue reading

Back On Track in 2024

It’s time to look back at my year that was and figure out where this little train jumped off the rails. I’ll be the first to admit that I ran out of steam chugging along toward the end of the year. My writing output was way down for reasons I still can’t quite figure out. Everything has felt like a much bigger task to accomplish throughout the year. To that end, let’s look at what I wanted to do and how it came out:

  • Keeping Track of Things: I did a little bit better with this one, aside from my post schedule. I tried to track things much more and understand deadlines and such. I didn’t always succeed like I wanted to but at least I made the effort.
  • Creating Evergreen Content: This one was probably a miss. I didn’t create nearly as much content this year as I have in years past. What little I did create sometimes felt unfocused and less impactful. Part of that has to do with the overall move away from written content to something more video and audio focused. However, even my other content like Tomversations was significantly reduced this year. I will Continue reading

BGP EVPN Part-I: Challenges in Traditional Switched Datacenter Networks

Inefficient Link Utilization

The default Layer 2 Control Plane protocol in Cisco NX-OS is a Rapid Per-VLAN Spanning Tree Plus (Rapid PVST+), which runs 802.1w standard Rapid Spanning Tree Protocol (RSTP) instance per VLAN. Rapid PVST+ builds a VLAN-specific, loop-free Layer 2 data path from the STP root switch to all non-root switches. Spanning Tree Protocol, no matter which mode we use, allows only one active path at a time and blocks all redundant links. One general solution for activating all Inter-switch links is placing an STP root switch for odd and even VLANs into different switches. However, STP allows only a VLAN-based traffic load balancing.


CPU and Memory Usage

After building a loop-free data path, switches running Rapid PVST+ monitor the state of the network by using Spanning Tree instance-based Bridge Protocol Data Units (BPDU). By default, each switch sends instance-based BPDU messages from their designated port in two-second intervals. If we have 2000 VLANs, all switches must process 2000 BPDUs. To reduce CPU and Memory consumption caused by BPDU processing, we can use Multiple Spanning Tree – MSTP (802.1s), where VLANs are associated with Instances. For example, we can attach VLANs 1-999 to one instance and Continue reading

Goodbye, ipSpace.net Subscription

I ran the first webinar as an independent author almost exactly fourteen years ago1, with the first ticket sold just before New Year’s Eve. I kept focusing on individual webinars until someone asked me, “Would it be possible to buy access to everything you did?” His question effectively created the ipSpace.net subscription, with the first one sold in late 2010 (I still have the email that triggered the whole process).

Read more …
1 … 183 184 185 186 187 … 3,844