Jerikan: a configuration management system for network teams
There are many resources for network automation with Ansible. Most of them only expose the first steps or limit themselves to a narrow scope. They give no clue on how to expand from that. Real network environments may be large, versatile, heterogeneous, and filled with exceptions. The lack of real-world examples for Ansible deployments, unlike Puppet and SaltStack, leads many teams to build brittle and incomplete automation solutions.
We have released under an open-source license our attempt to tackle this problem:
- Jerikan, a tool to build configuration files from a single source of truth and Jinja2 templates, along with its integration into the GitLab CI system,
- an Ansible playbook to deploy these configuration files on network devices, and
- a redacted version of the configuration data and the templates for our, now defunct, datacenters in San Francisco and South Korea, covering many vendors (Facebook Wedge 100, Dell S4048 and S6010, Juniper QFX 5110, Juniper QFX 10002, Cisco ASR 9001, Cisco Catalyst 2960, Opengear console servers, and Linux), and many functionalities (provisioning, BGP-to-the-host routing, edge routing, out-of-band network, DNS configuration, integration with NetBox and IRRs).
Here is a quick demo to configure a new peering:
This work is the collective effort of Continue reading
Transient prompt with Zsh
Powerlevel10k is a prompt for Zsh. It contains some powerful features, is astoundingly fast, and easy to customize. I am quite amazed at the skills of its main author. Be sure to also have a look at Zsh for Humans, a complete Zsh configuration including this theme.
One of the nice features of Powerlevel10k is transient prompts: past prompts are reduced to a more minimal configuration to save space by removing unneeded information.
When it comes to configuring my shell, I still prefer writing and understanding each line going into it. Therefore, I am still building my Zsh configuration from scratch. Here is how I have integrated the above transient feature into my prompt.
The first step is to configure the appearance of the prompt in its
compact form. Let’s assume we have a variable, $_vbe_prompt_compact
set to 1 when we want a compact prompt. We use the
following function to define the prompt Continue reading
Zero-Touch Provisioning for Juniper
Juniper’s official documentation on ZTP explains how to configure the ISC DHCP Server to automatically upgrade and configure on first boot a Juniper device. However, the proposed configuration could be a bit more elegant. This note explains how.
TL;DR
Do not redefine option 43. Instead, specify the vendor
option space to use to encode parameters with vendor-option-space.
When booting for the first time, a Juniper device requests its IP address through a DHCP discover message, then request additional parameters for autoconfiguration through a DHCP request message:
Dynamic Host Configuration Protocol (Request)
Message type: Boot Request (1)
Hardware type: Ethernet (0x01)
Hardware address length: 6
Hops: 0
Transaction ID: 0x44e3a7c9
Seconds elapsed: 0
Bootp flags: 0x8000, Broadcast flag (Broadcast)
Client IP address: 0.0.0.0
Your (client) IP address: 0.0.0.0
Next server IP address: 0.0.0.0
Relay agent IP address: 0.0.0.0
Client MAC address: 02:00:00:00:00:01 (02:00:00:00:00:01)
Client hardware address padding: 00000000000000000000
Server host name not given
Boot file name not given
Magic cookie: DHCP
Option: (54) DHCP Server Identifier (10.0.2.2)
Option: (55) Parameter Request List
Length: 14
Parameter Request List Item: (3) Router
Parameter Request List Item: (51) IP Continue readingMy collection of vintage PC cards
Recently, I have been gathering some old hardware at my parents’ house, notably PC extension cards, as they don’t take much room and can be converted to a nice display item. Unfortunately, I was not very concerned about keeping stuff around. Compared to all the hardware I have acquired over the years, only a few pieces remain.
Tseng Labs ET4000AX (1989)
This SVGA graphics card was installed into a PC powered by a 386SX CPU running at 16 MHz. This was a good card at the time as it was pretty fast. It didn’t feature 2D acceleration, unlike the later ET4000/W32. This version only features 512 KB of RAM. It can display 1024×768 images with 16 colors or 800×600 with 256 colors. It was also compatible with CGA, EGA, VGA, MDA, and Hercules modes. No contemporary games were using the SVGA modes but the higher resolutions were useful with Windows 3.
This card was manufactured directly by Tseng Labs.
AdLib clone (1992)
My first sound card was an AdLib. My parents bought it in Canada during the summer holidays in 1992. It uses a Yamaha OPL2 chip to produce sound via FM synthesis. Continue reading
Running Isso on NixOS in a Docker container
This short article documents how I run Isso, the commenting system used by this blog, inside a Docker container on NixOS, a Linux distribution built on top of Nix. Nix is a declarative package manager for Linux and other Unix systems.
While NixOS 20.09 includes a derivation for Isso, it is unfortunately broken and relies on Python 2. As I am also using a fork of Isso, I have built my own derivation, heavily inspired by the one in master:1
issoPackage = with pkgs.python3Packages; buildPythonPackage rec { pname = "isso"; version = "custom"; src = pkgs.fetchFromGitHub { # Use my fork owner = "vincentbernat"; repo = pname; rev = "vbe/master"; sha256 = "0vkkvjcvcjcdzdj73qig32hqgjly8n3ln2djzmhshc04i6g9z07j"; }; propagatedBuildInputs = [ itsdangerous jinja2 misaka html5lib werkzeug bleach flask-caching ]; buildInputs = [ cffi ]; checkInputs = [ nose ]; checkPhase = '' ${python.interpreter} setup.py nosetests ''; };
I want to run Isso through Gunicorn. To this effect, I build an
environment combining Isso and Gunicorn. Then, I can invoke the latter
with "${issoEnv}/bin/gunicorn".
issoEnv = pkgs.python3.buildEnv.override { extraLibs = [ issoPackage pkgs.python3Packages. Continue reading
Speeding up bgpq4 with IRRd in a container
When building route filters with bgpq4 or bgpq3, the speed of
rr.ntt.net or whois.radb.net can be a bottleneck. Updating many
filters may take several tens of minutes, depending on the load:
$ time bgpq4 -h whois.radb.net AS-HURRICANE | wc -l 909869 1.96s user 0.15s system 2% cpu 1:17.64 total $ time bgpq4 -h rr.ntt.net AS-HURRICANE | wc -l 927865 1.86s user 0.08s system 12% cpu 14.098 total
A possible solution is to have your own IRRd instance in your network, mirroring the main routing registries. A close alternative is to bundle IRRd with all the data in a ready-to-use Docker image. This also has the advantage of easy integration into a Docker-based CI/CD pipeline.
$ git clone https://github.com/vincentbernat/irrd-legacy.git -b blade/master $ cd irrd-legacy $ docker build . -t irrd-snapshot:latest […] Successfully built 58c3e83a1d18 Successfully tagged irrd-snapshot:latest $ docker container run --rm --detach --publish=43:43 irrd-snapshot 4879cfe7413075a0c217089dcac91ed356424c6b88808d8fcb01dc00eafcc8c7 $ time bgpq4 -h localhost AS-HURRICANE | wc -l 904137 1.72s user 0.11s system 96% cpu 1.881 total
The Dockerfile contains three stages:
- building IRRd,1
- retrieving various IRR databases, and
- assembling Continue reading
Syncing RIPE, ARIN and APNIC objects with a custom Ansible module
Internet is split into five regional Internet registry: AFRINIC, ARIN, APNIC, LACNIC and RIPE. Each RIR maintains an Internet Routing Registry. An IRR allows one to publish information about the routing of Internet number resources.1 Operators use this to determine the owner of an IP address and to construct and maintain routing filters. To ensure your routes are widely accepted, it is important to keep the prefixes you announce up-to-date in an IRR.
There are two common tools to query this database: whois and
bgpq4. The first one allows you to do a query with the WHOIS
protocol:
$ whois -BrG 2a0a:e805:400::/40 […] inet6num: 2a0a:e805:400::/40 netname: FR-BLADE-CUSTOMERS-DE country: DE geoloc: 50.1109 8.6821 admin-c: BN2763-RIPE tech-c: BN2763-RIPE status: ASSIGNED mnt-by: fr-blade-1-mnt remarks: synced with cmdb created: 2020-05-19T08:04:58Z last-modified: 2020-05-19T08:04:58Z source: RIPE route6: 2a0a:e805:400::/40 descr: Blade IPv6 - AMS1 origin: AS64476 mnt-by: fr-blade-1-mnt remarks: synced with cmdb created: 2019-10-01T08:19:34Z last-modified: 2020-05-19T08:05:00Z source: RIPE
The second one allows you to build route filters using the information contained in the IRR database:
$ bgpq4 -6 -S RIPE -b AS64476 NN = [ 2a0a:e805::/40, 2a0a:e805:100::/40, 2a0a:e805:300::/40, 2a0a:e805:400::/40, 2a0a:e805:500::/40 ];
There is no module available on Ansible Galaxy Continue reading
Keepalived and unicast over multiple interfaces
Keepalived is a Linux implementation of VRRP. The usual role of VRRP is to share a virtual IP across a set of routers. For each VRRP instance, a leader is elected and gets to serve the IP address, ensuring the high availability of the attached service. Keepalived can also be used for a generic leader election, thanks to its ability to use scripts for healthchecking and run commands on state change.
A simple configuration looks like this:
vrrp_instance gateway1 { state BACKUP # ❶ interface eth0 # ❷ virtual_router_id 12 # ❸ priority 101 # ❹ virtual_ipaddress { 2001:db8:ff/64 } }
The state keyword in ❶ instructs Keepalived to not take the leader
role when starting. Otherwise, incoming nodes create a temporary
disruption by taking over the IP address until the election settles.
The interface keyword in ❷ defines the interface for sending and
receiving VRRP packets. It is also the default interface to configure
the virtual IP address. The virtual_router_id directive in ❸ is
common to all nodes sharing the virtual IP. The priority keyword in
❹ helps choosing which router will be elected as leader. If you need
more information around Keepalived, be sure to check Continue reading
Syncing NetBox with a custom Ansible module
The netbox.netbox collection from Ansible Galaxy
provides several modules to update NetBox objects:
- name: create a device in NetBox netbox_device: netbox_url: http://netbox.local netbox_token: s3cret data: name: to3-p14.sfo1.example.com device_type: QFX5110-48S device_role: Compute Switch site: SFO1
However, if NetBox is not your source of truth, you may want to ensure it stays in sync with your configuration management database1 by removing outdated devices or IP addresses. While it should be possible to glue together a playbook with a query, a loop and some filtering to delete unwanted elements, it feels clunky, inefficient and an abuse of YAML as a programming language. A specific Ansible module solves this issue and is likely more flexible.
Notice
I recommend that you read “Writing a custom Ansible module” as an introduction, as well as “Syncing MySQL tables” for a first simpler example.
Code
The module has the following signature and it syncs NetBox with the content of the provided YAML file:
netbox_sync: source: netbox.yaml api: https://netbox.example.com token: s3cret
The synchronized objects are:
- sites,
- manufacturers,
- device types,
- device Continue reading
Syncing MySQL tables with a custom Ansible module
The community.mysql collection from Ansible Galaxy
provides a mysql_query module to run arbitrary MySQL queries.
Unfortunately, it does not support check mode nor the --diff flag.
It is also unable to tell if there was a change. Let’s write a
specific Ansible module to workaround these issues.
Notice
I recommend that you read “Writing a custom Ansible module” as an introduction.
Code
The module has the following signature and it executes the provided SQL statements in a single transaction. It needs a list of the affected tables to be able to detect and show the changes.
mysql_sync: sql: | DELETE FROM rules WHERE name LIKE 'CMDB:%'; INSERT INTO rules (name, rule) VALUES ('CMDB: check for cats', ':is(object, "CAT")'), ('CMDB: check for dogs', ':is(object, "DOG")'); REPLACE INTO webhooks (name, url) VALUES ('OpsGenie', 'https://opsgenie/something/token'), ('Slack', 'https://slack/something/token'); user: monitoring password: Yooghah5 database: monitoring tables: - rules - webhooks
Prerequisites
The module does not enforce idempotency, but it is expected you
provide appropriate SQL queries. In the above example, idempotency is
achieved because the content of the rules table is deleted and
recreated from scratch while the rows in the webhooks table are
Continue reading
Syncing SSH keys on Cisco IOS-XR with a custom Ansible module
The cisco.iosxr collection from Ansible Galaxy
provides an iosxr_user module to manage local users,
along with their SSH keys. However, the module is quite slow, do not
display a diff for changed SSH keys, never signal change when a key is
modified, and does not delete obsolete keys. Let’s write a custom
Ansible module managing only the SSH keys while fixing these issues.
Notice
I recommend that you read “Writing a custom Ansible module” as an introduction.
How to add an SSH key to a user
Adding SSH keys to users in Cisco IOS-XR is quite undocumented. First, you need to encode the key with the “ssh-rsa” key ASN.1 format, like an OpenSSH public key, but without the base64-encoding:
$ awk '{print $2}' id_rsa.pub \ | base64 -d \ > publickey_vincent.raw
Then, you upload the key with SCP to harddisk:/publickey_vincent.raw
and import it for the current user with the following IOS command:
crypto key import authentication rsa harddisk:/publickey_vincent.b64
However, if you want to import a key for another user, you need to be
part of the root-system group:
username vincent group root-lr group root-system
With the following admin command, you Continue reading
Writing a custom Ansible module
Ansible ships a lot of modules you can combine for your configuration management needs. However, the quality of these modules may vary widely. Sometimes, it may be quicker and more robust to write your own module instead of shopping and assembling existing ones.1
In my opinion, a robust module exhibits the following characteristics:
- idempotency,
- diff support,
- check mode compatibility,
- correct change signaling, and
- lifecycle management.
In a nutshell, it means the module can run with --diff --check and
shows the changes it would apply. When run twice in a row, the second
run won’t apply or signal changes. The last bullet point suggests the
module should be able to delete outdated objects configured during
previous runs.2
The module code should be minimal and tailored to your needs. Making the module generic for use by other users is a non-goal. Less code usually means less bugs and easier to understand.
I do not cover testing here. It is undeniably a good practice, but it requires a significant effort. In my opinion, it is preferable to have a well written module matching the above characteristics rather than a module that is well tested but without them or a module requiring Continue reading
Zero-Touch Provisioning for Cisco IOS
The official documentation to automatically upgrade and configure on first boot a Cisco switch running on IOS, like a Cisco Catalyst 2960-X Series switch, is scarce on details. This note explains how to configure the ISC DHCP Server for this purpose.
When booting for the first time, Cisco IOS sends a DHCP request on all ports:
Dynamic Host Configuration Protocol (Discover)
Message type: Boot Request (1)
Hardware type: Ethernet (0x01)
Hardware address length: 6
Hops: 0
Transaction ID: 0x0000117c
Seconds elapsed: 0
Bootp flags: 0x8000, Broadcast flag (Broadcast)
Client IP address: 0.0.0.0
Your (client) IP address: 0.0.0.0
Next server IP address: 0.0.0.0
Relay agent IP address: 0.0.0.0
Client MAC address: Cisco_6c:12:c0 (b4:14:89:6c:12:c0)
Client hardware address padding: 00000000000000000000
Server host name not given
Boot file name not given
Magic cookie: DHCP
Option: (53) DHCP Message Type (Discover)
Option: (57) Maximum DHCP Message Size
Option: (61) Client identifier
Length: 25
Type: 0
Client Identifier: cisco-b414.896c.12c0-Vl1
Option: (55) Parameter Request List
Length: 12
Parameter Request List Item: (1) Subnet Mask
Parameter Request List Item: (66) TFTP Server Name
Parameter Request List Item: (6) Domain Name Server
Parameter Request List Item: Continue readingSafer SSH agent forwarding
ssh-agent is a program to hold in memory the private keys used by
SSH for public-key authentication. When the agent is running, ssh
forwards to it the signature requests from the server. The agent
performs the private key operations and returns the results to ssh.
It is useful if you keep your private keys encrypted on disk and you
don’t want to type the password at each connection. Keeping the agent
secure is critical: someone able to communicate with the agent can
authenticate on your behalf on remote servers.
ssh also provides the ability to forward the agent to a remote
server. From this remote server, you can authenticate to another
server using your local agent, without copying your private key on the
intermediate server. As stated in the manual page, this is
dangerous!
Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent’s UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent. A safer alternative Continue reading
Downgrade all Debian packages to a specific date
Unlike NixOS, Debian doesn’t have a builtin mechanism to rollback an installation to a specific point in time. However, thanks to snapshot.debian.org, a wayback machine for Debian packages, it is possible to downgrade all packages to the versions from a chosen date.
Let’s suppose we want to go back to January, 20th 2020. In
/etc/apt/sources.list.d/snapshot.list, we add a date-specific
snapshot as a source:
deb [check-valid-until=no] https://snapshot.debian.org/archive/debian/20200120T111800Z/ unstable main contrib non-free
In /etc/apt/preferences.d/snapshot.pref, we set the priority of all
packages from this source to 1001. This is above the default priority
of 500 and over 1000 to allow downgrade. See apt_preferences(5)
manual page for more details.
Package: * Pin: origin snapshot.debian.org Pin-Priority: 1001
After running apt update, we can check the result with apt policy:
$ apt policy Package files: 100 /var/lib/dpkg/status release a=now 1001 https://snapshot.debian.org/archive/debian/20200120T111800Z unstable/non-free amd64 Packages release o=Debian,a=unstable,n=sid,l=Debian,c=non-free,b=amd64 origin snapshot.debian.org 1001 https://snapshot.debian.org/archive/debian/20200120T111800Z unstable/contrib amd64 Packages release o=Debian,a=unstable,n=sid,l=Debian,c=contrib,b=amd64 origin snapshot.debian.org 1001 https://snapshot.debian.org/archive/debian/20200120T111800Z unstable/main amd64 Packages release o=Debian,a=unstable,n=sid,l=Debian,c=main,b=amd64 origin snapshot.debian.org […]
When requesting an upgrade, we Continue reading
ThinkPad X1 Carbon 2014: 5 years later
I have recently replaced my ThinkPad X1 Carbon 2014 (second generation). I have kept it for more than five years, using it every day and carrying it everywhere. The expected lifetime of a laptop is always an unknown. Let me share my feedback.
My configuration embeds an Intel vPro Core i7-4600U, 8 Gib of RAM, a 256 Gib SATA SSD, a matte WQHD display and a WWAN LTE card. I got it in June 2014. It has spent these years running Debian Sid, starting from Linux 3.14 to Linux 5.4.
This generation of ThinkPad X1 Carbon has been subject to a variety of experiences around the keyboard. We are still hunting the culprits. The layout is totally messed up, with many keys displaced.1 I have remapped most of them. It also lacks physical function keys: they have been replaced by a non-customizable touch bar. I do not like it due to absence of tactile feedback and it is quite easy to hit a key by mistake. I would recommend to Continue reading
Replacing Orange Livebox router by a Linux box
A few months ago, I moved back to France and I settled for Orange as an ISP with a bundle combining Internet and mobile subscription. In Switzerland, I was using my own router instead of the box provided by Swisscom. While there is an abundant documentation to replace the box provided by Orange, the instructions around a plain Linux box are kludgy. I am exposing here my own variation. I am only interested in getting IPv4/IPv6 access: no VoIP, no TV.
Hardware
Orange is using GPON for its FTTH deployment. Therefore, an ONT is needed to encapsulate and decapsulate Ethernet frames into GPON frames. Two form-factors are available. It can be small Huawei HG8010H box also acting as a media converter to Ethernet 1000BASE-T:
With a recent Livebox, Orange usually provides an SFP to be plugged inside the Livebox. For some reason I got the external ONT instead of the SFP version. As I have a Netgear GS110TP with two SFP ports, I have bought an SFP GPON FGS202 on eBay. It is the same model than Orange is providing with its Livebox 4. However, I didn’t get Continue reading
Self-hosted videos with HLS: subtitles
In a previous article, I have described a solution to self-host
videos while offering a delivery adapted to each user’s bandwith,
thanks to HLS and hls.js. Subtitles1 were not part of
the game. While they can be declared inside the HLS manifest or
embedded into the video, it is easier to include them directly in the
<video> element, using the WebVTT format:
<video poster="poster.jpg" controls preload="none"> <source src="index.m3u8" type="application/vnd.apple.mpegurl"> <source src="progressive.mp4" type='video/mp4; codecs="avc1.4d401f, mp4a.40.2"'> <track src="de.vtt" kind="subtitles" srclang="de" label="Deutsch"> <track src="en.vtt" kind="subtitles" srclang="en" label="English"> </video>
Watch the following demonstration, featuring Agent 327: Operation Barbershop, a video created by Blender Animation Studio and currently released under the Creative Commons Attribution No Derivatives 2.0 license:
You may want to jump to 0:12 for the first subtitle. Most browsers should display a widget to toggle subtitles. This works just fine with Chromium but Firefox will not show the menu Continue reading
Asynchronous Zsh prompt with Git status
Zsh ships vcs_info, a function fetching information about the
VCS state for the current directory and populating a variable that can
be used in a shell prompt. It supports several VCS, including Git and
SVN. Here is an example of configuration:
autoload -Uz vcs_info zstyle ':vcs_info:*' enable git () { local formats="${PRCH[branch]} %b%c%u" local actionformats="${formats}%{${fg[default]}%} ${PRCH[sep]} %{${fg[green]}%}%a" zstyle ':vcs_info:*:*' formats $formats zstyle ':vcs_info:*:*' actionformats $actionformats zstyle ':vcs_info:*:*' stagedstr "%{${fg[green]}%}${PRCH[circle]}" zstyle ':vcs_info:*:*' unstagedstr "%{${fg[yellow]}%}${PRCH[circle]}" zstyle ':vcs_info:*:*' check-for-changes true } add-zsh-hook precmd vcs_info
You can use ${vcs_info_msg_0_} in your prompt to display the current
branch, the presence of staged and unstaged changes, as well as the
ongoing action.1 Have a look at the documentation for more
details.
vcs_info function.On large repositories, some information are expensive to fetch. While
vcs_info queries Git, interactions with Zsh are stuck. A possible
solution is to execute vcs_info asynchronously with zsh-async.
Securing BGP on the host with the RPKI
An increasingly popular design for a data-center network is BGP on the host: each host ships with a BGP daemon to advertise the IPs it handles and receives the routes to its fellow servers. Compared to a L2-based design, it is very scalable, resilient, cross-vendor and safe to operate.1 Take a look at “L3 routing to the hypervisor with BGP” for a usage example.
While routing on the host eliminates the security problems related to
Ethernet networks, a server may announce any IP prefix. In the above
picture, two of them are announcing 2001:db8:cc::/64. This could be
a legit use of anycast or a prefix hijack. BGP offers several
solutions to improve this aspect and one of them is to leverage the
features around the RPKI infrastructure.
Short introduction to the RPKI
On the Internet, BGP is mostly relying on trust. This contributes to various incidents due to operator errors, like the one that affected Cloudflare a few months ago, or to malicious attackers, like the hijack of Amazon Continue reading