Archive

Category Archives for "MTU Ninja | Vincent Bernat"

Serving WebP & AVIF images with Nginx

WebP and AVIF are two image formats for the web. They aim to produce smaller files than JPEG and PNG. They both support lossy and lossless compression, as well as alpha transparency. WebP was developed by Google and is a derivative of the VP8 video format.1 It is supported on most browsers. AVIF is using the newer AV1 video format to achieve better results. It is supported by Chromium-based browsers and has experimental support for Firefox.2

Your browser supports WebP and AVIF image formats. Your browser supports none of these image formats. Your browser only supports the WebP image format. Your browser only supports the AVIF image format.

Without JavaScript, I can’t tell what your browser supports.

Converting and optimizing images

For this blog, I am using the following shell snippets to convert and optimize JPEG and PNG images. Skip to the next section if you are only interested in the Nginx setup.

JPEG images

JPEG images are converted to WebP using cwebp.

find media/images -type f -name '*.jpg' -print0 \
  | xargs -0n1 -P$(nproc) -i \
      cwebp -q 84 -af '{}' -o '{}'.webp

They are converted to AVIF using avifenc Continue reading

Jerikan: a configuration management system for network teams

There are many resources for network automation with Ansible. Most of them only expose the first steps or limit themselves to a narrow scope. They give no clue on how to expand from that. Real network environments may be large, versatile, heterogeneous, and filled with exceptions. The lack of real-world examples for Ansible deployments, unlike Puppet and SaltStack, leads many teams to build brittle and incomplete automation solutions.

We have released under an open-source license our attempt to tackle this problem:

  • Jerikan, a tool to build configuration files from a single source of truth and Jinja2 templates, along with its integration into the GitLab CI system,
  • an Ansible playbook to deploy these configuration files on network devices, and
  • a redacted version of the configuration data and the templates for our, now defunct, datacenters in San Francisco and South Korea, covering many vendors (Facebook Wedge 100, Dell S4048 and S6010, Juniper QFX 5110, Juniper QFX 10002, Cisco ASR 9001, Cisco Catalyst 2960, Opengear console servers, and Linux), and many functionalities (provisioning, BGP-to-the-host routing, edge routing, out-of-band network, DNS configuration, integration with NetBox and IRRs).

Here is a quick demo to configure a new peering:

This work is the collective effort of Continue reading

Transient prompt with Zsh

Powerlevel10k is a prompt for Zsh. It contains some powerful features, is astoundingly fast, and easy to customize. I am quite amazed at the skills of its main author. Be sure to also have a look at Zsh for Humans, a complete Zsh configuration including this theme.

One of the nice features of Powerlevel10k is transient prompts: past prompts are reduced to a more minimal configuration to save space by removing unneeded information.

Demonstration of a transient prompt with Zsh: past prompts use a
more compact form
My implementation of a transient prompt with Zsh. Past prompts are compact and include the time of the command execution, the hostname, and the status of the previous command while the complete prompt contains more information like the current directory and the Git branch.

When it comes to configuring my shell, I still prefer writing and understanding each line going into it. Therefore, I am still building my Zsh configuration from scratch. Here is how I have integrated the above transient feature into my prompt.

The first step is to configure the appearance of the prompt in its compact form. Let’s assume we have a variable, $_vbe_prompt_compact set to 1 when we want a compact prompt. We use the following function to define the prompt Continue reading

Zero-Touch Provisioning for Juniper

Juniper’s official documentation on ZTP explains how to configure the ISC DHCP Server to automatically upgrade and configure on first boot a Juniper device. However, the proposed configuration could be a bit more elegant. This note explains how.

TL;DR

Do not redefine option 43. Instead, specify the vendor option space to use to encode parameters with vendor-option-space.


When booting for the first time, a Juniper device requests its IP address through a DHCP discover message, then request additional parameters for autoconfiguration through a DHCP request message:

Dynamic Host Configuration Protocol (Request)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x44e3a7c9
    Seconds elapsed: 0
    Bootp flags: 0x8000, Broadcast flag (Broadcast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: 02:00:00:00:00:01 (02:00:00:00:00:01)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (54) DHCP Server Identifier (10.0.2.2)
    Option: (55) Parameter Request List
        Length: 14
        Parameter Request List Item: (3) Router
        Parameter Request List Item: (51) IP  Continue reading

My collection of vintage PC cards

Recently, I have been gathering some old hardware at my parents’ house, notably PC extension cards, as they don’t take much room and can be converted to a nice display item. Unfortunately, I was not very concerned about keeping stuff around. Compared to all the hardware I have acquired over the years, only a few pieces remain.

Tseng Labs ET4000AX (1989)

This SVGA graphics card was installed into a PC powered by a 386SX CPU running at 16 MHz. This was a good card at the time as it was pretty fast. It didn’t feature 2D acceleration, unlike the later ET4000/W32. This version only features 512 KB of RAM. It can display 1024×768 images with 16 colors or 800×600 with 256 colors. It was also compatible with CGA, EGA, VGA, MDA, and Hercules modes. No contemporary games were using the SVGA modes but the higher resolutions were useful with Windows 3.

This card was manufactured directly by Tseng Labs.

Carte Tseng Labs ET4000AX ISA au-dessus de la boîte "Planète Aventure"
Tseng Labs ET4000 AX ISA card

AdLib clone (1992)

My first sound card was an AdLib. My parents bought it in Canada during the summer holidays in 1992. It uses a Yamaha OPL2 chip to produce sound via FM synthesis. Continue reading

Running Isso on NixOS in a Docker container

This short article documents how I run Isso, the commenting system used by this blog, inside a Docker container on NixOS, a Linux distribution built on top of Nix. Nix is a declarative package manager for Linux and other Unix systems.


While NixOS 20.09 includes a derivation for Isso, it is unfortunately broken and relies on Python 2. As I am also using a fork of Isso, I have built my own derivation, heavily inspired by the one in master:1

issoPackage = with pkgs.python3Packages; buildPythonPackage rec {
  pname = "isso";
  version = "custom";

  src = pkgs.fetchFromGitHub {
    # Use my fork
    owner = "vincentbernat";
    repo = pname;
    rev = "vbe/master";
    sha256 = "0vkkvjcvcjcdzdj73qig32hqgjly8n3ln2djzmhshc04i6g9z07j";
  };

  propagatedBuildInputs = [
    itsdangerous
    jinja2
    misaka
    html5lib
    werkzeug
    bleach
    flask-caching
  ];

  buildInputs = [
    cffi
  ];

  checkInputs = [ nose ];

  checkPhase = ''
    ${python.interpreter} setup.py nosetests
  '';
};

I want to run Isso through Gunicorn. To this effect, I build an environment combining Isso and Gunicorn. Then, I can invoke the latter with "${issoEnv}/bin/gunicorn".

issoEnv = pkgs.python3.buildEnv.override {
    extraLibs = [
      issoPackage
      pkgs.python3Packages. Continue reading

Speeding up bgpq4 with IRRd in a container

When building route filters with bgpq4 or bgpq3, the speed of rr.ntt.net or whois.radb.net can be a bottleneck. Updating many filters may take several tens of minutes, depending on the load:

$ time bgpq4 -h whois.radb.net AS-HURRICANE | wc -l
909869
1.96s user 0.15s system 2% cpu 1:17.64 total
$ time bgpq4 -h rr.ntt.net AS-HURRICANE | wc -l
927865
1.86s user 0.08s system 12% cpu 14.098 total

A possible solution is to have your own IRRd instance in your network, mirroring the main routing registries. A close alternative is to bundle IRRd with all the data in a ready-to-use Docker image. This also has the advantage of easy integration into a Docker-based CI/CD pipeline.

$ git clone https://github.com/vincentbernat/irrd-legacy.git -b blade/master
$ cd irrd-legacy
$ docker build . -t irrd-snapshot:latest
[…]
Successfully built 58c3e83a1d18
Successfully tagged irrd-snapshot:latest
$ docker container run --rm --detach --publish=43:43 irrd-snapshot
4879cfe7413075a0c217089dcac91ed356424c6b88808d8fcb01dc00eafcc8c7
$ time bgpq4 -h localhost AS-HURRICANE | wc -l
904137
1.72s user 0.11s system 96% cpu 1.881 total

The Dockerfile contains three stages:

  1. building IRRd,1
  2. retrieving various IRR databases, and
  3. assembling Continue reading

Syncing RIPE, ARIN and APNIC objects with a custom Ansible module

Internet is split into five regional Internet registry: AFRINIC, ARIN, APNIC, LACNIC and RIPE. Each RIR maintains an Internet Routing Registry. An IRR allows one to publish information about the routing of Internet number resources.1 Operators use this to determine the owner of an IP address and to construct and maintain routing filters. To ensure your routes are widely accepted, it is important to keep the prefixes you announce up-to-date in an IRR.

There are two common tools to query this database: whois and bgpq4. The first one allows you to do a query with the WHOIS protocol:

$ whois -BrG 2a0a:e805:400::/40
[…]
inet6num:       2a0a:e805:400::/40
netname:        FR-BLADE-CUSTOMERS-DE
country:        DE
geoloc:         50.1109 8.6821
admin-c:        BN2763-RIPE
tech-c:         BN2763-RIPE
status:         ASSIGNED
mnt-by:         fr-blade-1-mnt
remarks:        synced with cmdb
created:        2020-05-19T08:04:58Z
last-modified:  2020-05-19T08:04:58Z
source:         RIPE

route6:         2a0a:e805:400::/40
descr:          Blade IPv6 - AMS1
origin:         AS64476
mnt-by:         fr-blade-1-mnt
remarks:        synced with cmdb
created:        2019-10-01T08:19:34Z
last-modified:  2020-05-19T08:05:00Z
source:         RIPE

The second one allows you to build route filters using the information contained in the IRR database:

$ bgpq4 -6 -S RIPE -b AS64476
NN = [
    2a0a:e805::/40,
    2a0a:e805:100::/40,
    2a0a:e805:300::/40,
    2a0a:e805:400::/40,
    2a0a:e805:500::/40
];

There is no module available on Ansible Galaxy Continue reading

Keepalived and unicast over multiple interfaces

Keepalived is a Linux implementation of VRRP. The usual role of VRRP is to share a virtual IP across a set of routers. For each VRRP instance, a leader is elected and gets to serve the IP address, ensuring the high availability of the attached service. Keepalived can also be used for a generic leader election, thanks to its ability to use scripts for healthchecking and run commands on state change.

A simple configuration looks like this:

vrrp_instance gateway1 {
  state BACKUP          # ❶
  interface eth0        # ❷
  virtual_router_id 12  # ❸
  priority 101          # ❹
  virtual_ipaddress {
    2001:db8:ff/64
  }
}

The state keyword in ❶ instructs Keepalived to not take the leader role when starting. Otherwise, incoming nodes create a temporary disruption by taking over the IP address until the election settles. The interface keyword in ❷ defines the interface for sending and receiving VRRP packets. It is also the default interface to configure the virtual IP address. The virtual_router_id directive in ❸ is common to all nodes sharing the virtual IP. The priority keyword in ❹ helps choosing which router will be elected as leader. If you need more information around Keepalived, be sure to check Continue reading

Syncing NetBox with a custom Ansible module

The netbox.netbox collection from Ansible Galaxy provides several modules to update NetBox objects:

- name: create a device in NetBox
  netbox_device:
    netbox_url: http://netbox.local
    netbox_token: s3cret
    data:
      name: to3-p14.sfo1.example.com
      device_type: QFX5110-48S
      device_role: Compute Switch
      site: SFO1

However, if NetBox is not your source of truth, you may want to ensure it stays in sync with your configuration management database1 by removing outdated devices or IP addresses. While it should be possible to glue together a playbook with a query, a loop and some filtering to delete unwanted elements, it feels clunky, inefficient and an abuse of YAML as a programming language. A specific Ansible module solves this issue and is likely more flexible.

Notice

I recommend that you read “Writing a custom Ansible module” as an introduction, as well as “Syncing MySQL tables” for a first simpler example.

Code

The module has the following signature and it syncs NetBox with the content of the provided YAML file:

netbox_sync:
  source: netbox.yaml
  api: https://netbox.example.com
  token: s3cret

The synchronized objects are:

Syncing MySQL tables with a custom Ansible module

The community.mysql collection from Ansible Galaxy provides a mysql_query module to run arbitrary MySQL queries. Unfortunately, it does not support check mode nor the --diff flag. It is also unable to tell if there was a change. Let’s write a specific Ansible module to workaround these issues.

Notice

I recommend that you read “Writing a custom Ansible module” as an introduction.

Code

The module has the following signature and it executes the provided SQL statements in a single transaction. It needs a list of the affected tables to be able to detect and show the changes.

mysql_sync:
  sql: |
    DELETE FROM rules WHERE name LIKE 'CMDB:%';
    INSERT INTO rules (name, rule) VALUES
      ('CMDB: check for cats', ':is(object, "CAT")'),
      ('CMDB: check for dogs', ':is(object, "DOG")');
    REPLACE INTO webhooks (name, url) VALUES
      ('OpsGenie', 'https://opsgenie/something/token'),
      ('Slack', 'https://slack/something/token');
  user: monitoring
  password: Yooghah5
  database: monitoring
  tables:
    - rules
    - webhooks

Prerequisites

The module does not enforce idempotency, but it is expected you provide appropriate SQL queries. In the above example, idempotency is achieved because the content of the rules table is deleted and recreated from scratch while the rows in the webhooks table are Continue reading

Syncing SSH keys on Cisco IOS-XR with a custom Ansible module

The cisco.iosxr collection from Ansible Galaxy provides an iosxr_user module to manage local users, along with their SSH keys. However, the module is quite slow, do not display a diff for changed SSH keys, never signal change when a key is modified, and does not delete obsolete keys. Let’s write a custom Ansible module managing only the SSH keys while fixing these issues.

Notice

I recommend that you read “Writing a custom Ansible module” as an introduction.

How to add an SSH key to a user

Adding SSH keys to users in Cisco IOS-XR is quite undocumented. First, you need to encode the key with the “ssh-rsa” key ASN.1 format, like an OpenSSH public key, but without the base64-encoding:

$ awk '{print $2}' id_rsa.pub \
    | base64 -d \
    > publickey_vincent.raw

Then, you upload the key with SCP to harddisk:/publickey_vincent.raw and import it for the current user with the following IOS command:

crypto key import authentication rsa harddisk:/publickey_vincent.b64

However, if you want to import a key for another user, you need to be part of the root-system group:

username vincent
 group root-lr
 group root-system

With the following admin command, you Continue reading

Writing a custom Ansible module

Ansible ships a lot of modules you can combine for your configuration management needs. However, the quality of these modules may vary widely. Sometimes, it may be quicker and more robust to write your own module instead of shopping and assembling existing ones.1

In my opinion, a robust module exhibits the following characteristics:

  • idempotency,
  • diff support,
  • check mode compatibility,
  • correct change signaling, and
  • lifecycle management.

In a nutshell, it means the module can run with --diff --check and shows the changes it would apply. When run twice in a row, the second run won’t apply or signal changes. The last bullet point suggests the module should be able to delete outdated objects configured during previous runs.2

The module code should be minimal and tailored to your needs. Making the module generic for use by other users is a non-goal. Less code usually means less bugs and easier to understand.

I do not cover testing here. It is undeniably a good practice, but it requires a significant effort. In my opinion, it is preferable to have a well written module matching the above characteristics rather than a module that is well tested but without them or a module requiring Continue reading

Zero-Touch Provisioning for Cisco IOS

The official documentation to automatically upgrade and configure on first boot a Cisco switch running on IOS, like a Cisco Catalyst 2960-X Series switch, is scarce on details. This note explains how to configure the ISC DHCP Server for this purpose.


When booting for the first time, Cisco IOS sends a DHCP request on all ports:

Dynamic Host Configuration Protocol (Discover)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x0000117c
    Seconds elapsed: 0
    Bootp flags: 0x8000, Broadcast flag (Broadcast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: Cisco_6c:12:c0 (b4:14:89:6c:12:c0)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Discover)
    Option: (57) Maximum DHCP Message Size
    Option: (61) Client identifier
        Length: 25
        Type: 0
        Client Identifier: cisco-b414.896c.12c0-Vl1
    Option: (55) Parameter Request List
        Length: 12
        Parameter Request List Item: (1) Subnet Mask
        Parameter Request List Item: (66) TFTP Server Name
        Parameter Request List Item: (6) Domain Name Server
        Parameter Request List Item:  Continue reading

Safer SSH agent forwarding

ssh-agent is a program to hold in memory the private keys used by SSH for public-key authentication. When the agent is running, ssh forwards to it the signature requests from the server. The agent performs the private key operations and returns the results to ssh. It is useful if you keep your private keys encrypted on disk and you don’t want to type the password at each connection. Keeping the agent secure is critical: someone able to communicate with the agent can authenticate on your behalf on remote servers.

ssh also provides the ability to forward the agent to a remote server. From this remote server, you can authenticate to another server using your local agent, without copying your private key on the intermediate server. As stated in the manual page, this is dangerous!

Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent’s UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent. A safer alternative Continue reading

Downgrade all Debian packages to a specific date

Unlike NixOS, Debian doesn’t have a builtin mechanism to rollback an installation to a specific point in time. However, thanks to snapshot.debian.org, a wayback machine for Debian packages, it is possible to downgrade all packages to the versions from a chosen date.

Let’s suppose we want to go back to January, 20th 2020. In /etc/apt/sources.list.d/snapshot.list, we add a date-specific snapshot as a source:

deb [check-valid-until=no] https://snapshot.debian.org/archive/debian/20200120T111800Z/ unstable main contrib non-free

In /etc/apt/preferences.d/snapshot.pref, we set the priority of all packages from this source to 1001. This is above the default priority of 500 and over 1000 to allow downgrade. See apt_preferences(5) manual page for more details.

Package: *
Pin: origin snapshot.debian.org
Pin-Priority: 1001

After running apt update, we can check the result with apt policy:

$ apt policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
1001 https://snapshot.debian.org/archive/debian/20200120T111800Z unstable/non-free amd64 Packages
     release o=Debian,a=unstable,n=sid,l=Debian,c=non-free,b=amd64
     origin snapshot.debian.org
1001 https://snapshot.debian.org/archive/debian/20200120T111800Z unstable/contrib amd64 Packages
     release o=Debian,a=unstable,n=sid,l=Debian,c=contrib,b=amd64
     origin snapshot.debian.org
1001 https://snapshot.debian.org/archive/debian/20200120T111800Z unstable/main amd64 Packages
     release o=Debian,a=unstable,n=sid,l=Debian,c=main,b=amd64
     origin snapshot.debian.org
[…]

When requesting an upgrade, we Continue reading

ThinkPad X1 Carbon 2014: 5 years later

I have recently replaced my ThinkPad X1 Carbon 2014 (second generation). I have kept it for more than five years, using it every day and carrying it everywhere. The expected lifetime of a laptop is always an unknown. Let me share my feedback.

ThinkPad X1 Carbon with the lid closed
ThinkPad X1 Carbon 20A7 with its lid closed

My configuration embeds an Intel vPro Core i7-4600U, 8 Gib of RAM, a 256 Gib SATA SSD, a matte WQHD display and a WWAN LTE card. I got it in June 2014. It has spent these years running Debian Sid, starting from Linux 3.14 to Linux 5.4.

Inside the X1 Carbon
The inside is still quite dust-free! In the bottom left, there is the Intel WLAN card, the Sierra WWAN card as well as the SSD.

This generation of ThinkPad X1 Carbon has been subject to a variety of experiences around the keyboard. We are still hunting the culprits. The layout is totally messed up, with many keys displaced.1 I have remapped most of them. It also lacks physical function keys: they have been replaced by a non-customizable touch bar. I do not like it due to absence of tactile feedback and it is quite easy to hit a key by mistake. I would recommend to Continue reading

Replacing Orange Livebox router by a Linux box

A few months ago, I moved back to France and I settled for Orange as an ISP with a bundle combining Internet and mobile subscription. In Switzerland, I was using my own router instead of the box provided by Swisscom. While there is an abundant documentation to replace the box provided by Orange, the instructions around a plain Linux box are kludgy. I am exposing here my own variation. I am only interested in getting IPv4/IPv6 access: no VoIP, no TV.

Hardware

Orange is using GPON for its FTTH deployment. Therefore, an ONT is needed to encapsulate and decapsulate Ethernet frames into GPON frames. Two form-factors are available. It can be small Huawei HG8010H box also acting as a media converter to Ethernet 1000BASE-T:

Huawei ONT rebranded as Orange
The rebranded Huawei HG8010H is acting as an ONT and media converter

With a recent Livebox, Orange usually provides an SFP to be plugged inside the Livebox. For some reason I got the external ONT instead of the SFP version. As I have a Netgear GS110TP with two SFP ports, I have bought an SFP GPON FGS202 on eBay. It is the same model than Orange is providing with its Livebox 4. However, I didn’t get Continue reading

Self-hosted videos with HLS: subtitles

In a previous article, I have described a solution to self-host videos while offering a delivery adapted to each user’s bandwith, thanks to HLS and hls.js. Subtitles1 were not part of the game. While they can be declared inside the HLS manifest or embedded into the video, it is easier to include them directly in the <video> element, using the WebVTT format:

<video poster="poster.jpg"
       controls preload="none">
  <source src="index.m3u8"
          type="application/vnd.apple.mpegurl">
  <source src="progressive.mp4"
          type='video/mp4; codecs="avc1.4d401f, mp4a.40.2"'>
  <track src="de.vtt"
         kind="subtitles" srclang="de" label="Deutsch">
  <track src="en.vtt"
         kind="subtitles" srclang="en" label="English">
</video>

Watch the following demonstration, featuring Agent 327: Operation Barbershop, a video created by Blender Animation Studio and currently released under the Creative Commons Attribution No Derivatives 2.0 license:

You may want to jump to 0:12 for the first subtitle. Most browsers should display a widget to toggle subtitles. This works just fine with Chromium but Firefox will not show the menu Continue reading

Asynchronous Zsh prompt with Git status

Zsh ships vcs_info, a function fetching information about the VCS state for the current directory and populating a variable that can be used in a shell prompt. It supports several VCS, including Git and SVN. Here is an example of configuration:

autoload -Uz vcs_info
zstyle ':vcs_info:*' enable git

() {
    local formats="${PRCH[branch]} %b%c%u"
    local actionformats="${formats}%{${fg[default]}%} ${PRCH[sep]} %{${fg[green]}%}%a"
    zstyle ':vcs_info:*:*' formats           $formats
    zstyle ':vcs_info:*:*' actionformats     $actionformats
    zstyle ':vcs_info:*:*' stagedstr         "%{${fg[green]}%}${PRCH[circle]}"
    zstyle ':vcs_info:*:*' unstagedstr       "%{${fg[yellow]}%}${PRCH[circle]}"
    zstyle ':vcs_info:*:*' check-for-changes true
}

add-zsh-hook precmd vcs_info

You can use ${vcs_info_msg_0_} in your prompt to display the current branch, the presence of staged and unstaged changes, as well as the ongoing action.1 Have a look at the documentation for more details.

Prompt with Git-related information, including branch name and
presence of tracked and untracked
changes
Example of prompt including information from the vcs_info function.

On large repositories, some information are expensive to fetch. While vcs_info queries Git, interactions with Zsh are stuck. A possible solution is to execute vcs_info asynchronously with zsh-async.

Continue reading

1 2 3