Robotcop: enforcing your robots.txt policies and stopping bots before they reach your website

Cloudflare’s AI Audit dashboard allows you to easily understand how AI companies and services access your content. AI Audit gives a summary of request counts broken out by bot, detailed path summaries for more granular insights, and the ability to filter by categories like AI Search or AI Crawler.

Today, we're going one step further. You can now quickly see which AI services are honoring your robots.txt policies, which aren’t, and then programmatically enforce these policies. 

What is robots.txt?

Robots.txt is a plain text file hosted on your domain that implements the Robots Exclusion Protocol, a standard that has been around since 1994. This file tells crawlers like Google, Bing, and many others which parts of your site, if any, they are allowed to access. 

There are many reasons why site owners would want to define which portions of their websites crawlers are allowed to access: they might not want certain content available on search engines or social networks, they might trust one platform more than another, or they might simply want to reduce automated traffic to their servers.

With the advent of generative AI, AI services have started crawling the Internet to Continue reading

IBGP Source Interface Selection Still Requires Configuration

A fellow networking engineer recently remarked, “FRRouting automatically selects the correct [IBGP] source interface even when not configured explicitly.”

TL&DR: No, it does not. You were just lucky.

Basics first1. BGP runs over TCP sessions. One of the first things a router does when establishing a BGP session with a configured neighbor is to open a TCP session with the configured neighbor’s IP address.

Read more …

NB507: Arista Announces Stackable Switches; FBI Recommends Encrypted Messaging

Take a Network Break! This week we warn of significant vulnerabilities in WhatsUp Gold and Cisco NX-OS. We also discuss US federal reaction to the penetration of US telco networks by Chinese state actors. These reactions include network and visibility hardening guidelines from CISA, the FBI (reluctantly) recommending the use of encrypted messaging apps, and... Read more »

Cloudflare 2024 Year in Review

The 2024 Cloudflare Radar Year in Review is our fifth annual review of Internet trends and patterns observed throughout the year at both a global and country/region level across a variety of metrics. In this year’s review, we have added several new traffic, adoption, connectivity, and email security metrics, as well as the ability to do year-over-year and geographic comparisons for selected metrics. 

Below, we present a summary of key findings, and then explore them in more detail in subsequent sections.

Key Findings

Traffic

  • Global Internet traffic grew 17.2% in 2024. 🔗

  • Google maintained its position as the most popular Internet service overall. OpenAI remained at the top of the Generative AI category. Binance remained at the top of the Cryptocurrency category. WhatsApp remained the top Messaging platform, and Facebook remained the top Social Media site. 🔗

  • Global traffic from Starlink grew 3.3x in 2024, in line with last year’s growth rate. After initiating service in Malawi in July 2023, Starlink traffic from that country grew 38x in 2024. As Starlink added new markets, we saw traffic grow rapidly in those locations. 🔗

  • Googlebot, Google’s web crawler, was responsible for the highest volume of request traffic to Continue reading

From ChatGPT to Temu: ranking top Internet services in 2024

Since the late 1990s, millions have relied on the Internet for searching, communicating, shopping, and working, though 2.6 billion people (about 31% of the global population) still lack Internet access. Over the years, use of the Internet has evolved from email and static sites to social media, streaming, e-commerce, cloud tools, and more recently AI chatbots, reflecting its constant adaptation to users' needs. This post explores how people interacted online in 2024, based on Cloudflare’s observations and a review of the year’s DNS trends.

Building on similar reports we’ve done over the past several years, we have compiled a ranking of the top Internet properties of 2024, with the same categories included in 2023, including Generative AI. In addition to our overall ranking, we chose 9 categories to focus on:

  1. Generative AI

  2. Social Media

  3. Ecommerce

  4. Video Streaming

  5. News

  6. Messaging

  7. Metaverse & Gaming

  8. Financial Services

  9. Cryptocurrency Services

As we have done since 2022, our analysis uses anonymized DNS query data from our 1.1.1.1 public DNS resolver, used by millions globally. We aggregate domains for each service (e.g., twitter.com, t.co, and x.com for X) and identify the sites that Continue reading

Managing AWS Complexity: Insights from Dr. Werner Vogels

https://www.youtube.com/watch?v=aim5x73crbM

Dr. Werner Vogels’ keynote at AWS re:Invent 2024 explores how simplicity can lead to complexity, highlighting innovations in AWS services and the importance of maintaining manageable systems.

Highlights

  • 🚀 Simplicity breeds complexity: AWS services like S3 exemplify the journey from simple beginnings to complex systems.
  • 🍕 The Two-Pizza Team: Small, autonomous teams enhance innovation while managing complexity effectively.
  • 🔄 Continuous learning: Emphasis on adapting structures and processes to accommodate growth and change.
  • 🌎 Global scalability: AWS focuses on building technologies that enable businesses to expand effortlessly across regions.
  • 🔍 Importance of observability: Understanding and managing system complexity through effective monitoring and metrics.
  • 🔒 Security by design: Embedding security measures from the outset to ensure robust systems.
  • 🤝 Community involvement: Encouraging tech professionals to support initiatives that address global challenges.

Key Insights

  • 🧩 Managing Complexity: Systems evolve over time, and complexity is inevitable. Organizations must strategically manage this complexity to avoid fragility while ensuring functionality.
  • ⚙ Evolvability as a Requirement: Building systems with the ability to evolve in response to user needs is essential. Flexibility in architecture allows for future changes without major disruptions.
  • 🔗 Decoupling Systems: Breaking down monolithic systems into smaller, independently functioning components enhances Continue reading

From Python to Go 006. Dictionaries and Maps.

Hello my friend,

We continue our journey from Python to Go (Golang), or more right to say with Python and Go (Golang) together. Today we are going to talk about a data structure, which is by far the most widely used in Python when it comes to a network and IT infrastructure automation and management. This data structure is called dictionaries in Python, or Map in Go (Golang).

Black Friday Is Over, Can I Still Buy Your Trainings?

Of course, you can. Our self-paced network automation trainings are the perfect place to start your journey in network and IT infrastructure automation or to upskill yourself further if you are seasoned engineer. There is no such thing as excessive knowledge, therefore we encourage you to join our network automation programs and start your study today:

We offer the following training programs in network automation for you:

During these trainings you will learn the following topics:

  • Success and failure strategies to build the automation tools.
  • Principles of software developments and the most useful and convenient tools.
  • Data Continue reading

TNO010: Navigating Network Automation Complexities: Insights from AutoCon 2 (Sponsored)

On today’s show, we recap some highlights of AutoCon2 with guest Jeremy Rossbach from sponsor Broadcom. Jeremy gives some background on his career, and then elaborates on conversations he had at AutoCon2. He also shares observations on the present and future of network automation, which include AI and robust observability solutions that integrate with the... Read more »

Why You Should Change Palo Alto Master Key?

Why You Should Change Palo Alto Master Key?

Palo Alto firewalls come with a default master key used to encrypt passwords, secrets, and certificates. If your firewall is compromised or someone gains unauthorized access, they can easily decrypt these secrets, posing a significant security risk. In this blog post, let's explore why you should change the master key, important considerations, and how to configure it. Let's get started.

Why Change the Master Key?

Palo Alto firewalls come with a default master key. Anyone with unauthorized access to the firewall can easily decrypt your secrets or export the configuration to another firewall to retrieve those secrets. For this reason, Palo Alto strongly recommends changing the master key as soon as possible.

Master Key Considerations

Configuring the master key isn’t something you can just set and forget; it requires careful consideration. Here are some important points to keep in mind.

  • The new master key must be exactly 16 characters long.
  • If your firewalls are in an HA pair, you need to disable 'Config Sync' before configuring the key, as the key does not sync across the pair. You must configure the exact same key on each firewall individually.
  • If the master key expires, the firewall or Panorama will Continue reading

HN760: Mitigate IoT/OT Vulnerabilities with Guided Virtual Patching (Sponsored)

Today on Heavy Networking, sponsored by Palo Alto Networks, we explore how virtual patching can be used to protect IoT and OT devices. Virtual patching leverages intrusion detection and intrusion prevention, combined with threat research, to block exploits targeting IoT and OT devices. Why would you use virtual patching? When it comes to IoT and... Read more »

Cutting to the Quick

No doubt you’ve seen the news that Intel has parted ways with Pat Gelsinger. There is a lot of info to unpack on that particular story but we did a good job of covering it on the Rundown this week. What I really wanted to talk about was a quote that I brought up in the episode that I heard from my friend Michael Bushong a couple of months ago:

No one cuts their way back into relevance.

It’s been rattling around in my head for a while and I wanted to talk about why he’s absolutely right.

Outcomes Need Incomes

Do you remember the coupon clipping craze of ten years ago? I think it started from some show on TLC about people that were ultra crazy couponers. They would do the math and they could buy like 100 lbs of rice for $2. They would stock up on a year’s worth of toothpaste at a time because you could pay next to nothing for it. However, the trend died out after a year or so. In part, that was because the show wasn’t very exciting after the shock of buying two years of hand soap wore off. The other Continue reading

IPB165: IPv6 Basics – Address Planning

Continuing the IPv6 Basics series, today’s podcast addresses IPv6 address planning.  Special “guest” Tom Coffeen who literally wrote the book, IPv6 Address Planning, helps answer questions and gives advice on how to effectively plan IPv6 addresses. We discuss topics such as the importance of long-term planning and understanding prefix sizes, common design pitfalls, and the... Read more »

N4N005: The Sort-of-Useful OSI Model

Network engineers should be familiar with the Open Systems Interconnection (OSI) model, a framework for understanding network communications. On today’s show, Ethan and Holly delve into each of the model’s seven layers to discuss their functions, associated protocols, and practical implications for modern networking. They also talk about how the OSI model is, in fact,... Read more »
1 … 87 88 89 90 91 … 3,840