Archive

Category Archives for "Thomas Habets blog"

The uselessness of bash

The way I write automation for personal projects nowadays seems to follow a common pattern:

  1. A command line, that’s getting a bit long
  2. A bash script
  3. Rewrite in Go

Occasionally I add a step between 2 and 3 where I write it in Python, but it’s generally not actually gaining me anything. Python’s concurrency primitives are pretty bad, and it’s pretty wasteful.

Maybe there’s an actually good scripting language somewhere.

I should remember that writing a bash script (step 2) seems to almost never be worth it. If it’s so complicated that it doesn’t fit on one line, then it’ll become complicated enough to not work with bash.

There are two main things that don’t work well. Maybe there are good solutions to these problems, but I’ve not found them.

1. Concurrency

There are no good primitives. Basically only xargs -P and &. It’s annonying when you have an embarrassingly parallelizable problem where you want to run exactly nproc in parallel.

Especially error handling becomes terrible here.

2. Error handling

You can handle errors in bash scripts in various ways:

  1. || operator. E.g. gzip -9 < a > a.gz || (echo "handling error…")
  2. set -e at the top Continue reading

More FT8 propagation

Last month I graphed the distance to remote stations as a function of time of day.

Today I plotted the gridsquare locations on a world map:

Grid squares heard

Ignore the top right one. That’s “RR73”, and not a real grid square. The rest should be accurate.

More that can be done (more interesting with more data than I can get, though):

  • also take into account the received signal strength
  • …and number of unique callsigns per grid square
  • create animations over time

If I had access to the data from pskreporter I could even, instead of using just a callsign as input data, use a grid square as input.

So for example I could create an animation to show what the propagation was over the last week from any given gridsquare, and generate them on-demand.

Like last time the scripts are pretty hacky proof of concepts. But they work.

Measuring propagation using FT8

One obvious thing that you can do after putting up an amateur radio antenna is to operate a bit on FT8, to see how the propagation goes. Just transmit on all bands and see how for you get.

E.g. this map on pskreporter.info with 10W on my EFHW:

10W EFHW propagation

You can also use the [reverse beacon network][rev] with morse code:

Reverse beacon network for M0THC

But that’s just a few samples. What about more statistical data? And propagation over time? I don’t have access to the raw data from pskreporter.info, and even if I did I can’t just set up an automatic beacon tx round the clock every day without requesting a Notice of Variation.

I may do that some day, but it’s a project for another time.

For this post what I want to know is if my antenna setup is better for 20m or 40m. Subjectively it seems like more is trickling in on 40m. And when they say that 40m is better “at night”, what time exactly do they mean?

For passive listening my data will, of course, be heavily skewed by when people are awake and active. But that means it’s skewed towards representing “if I call CQ, how Continue reading

Unifi controller with a real cert

I finally got sick of seeing a certificate error when connecting to my Ubuiquiti Unifi WiFi controller.

There are a bunch of shitty howtos describing how to install a cert, and one good one. But in order to make it more copy-paste for future me when the certificate needs renewing, and because the paths are not quite the same since I run the controller in a Docker container on a raspberry pi, here are the commands (after copying fullchain.pem and privkey.pem into the stateful data dir):

host$ docker ps  # make note of the docker ID
host$ docker exec ID_HERE -ti bash
docker$ openssl pkcs12 \
        -export \
        -inkey privkey.pem \
        -in fullchain.pem \
        -out cert.p12 \
        -name unifi \
        -password pass:secret
docker$ keytool \
        -importkeystore \
        -deststorepass aircontrolenterprise \
        -destkeypass aircontrolenterprise \
        -destkeystore /usr/lib/unifi/data/keystore \
        -srckeystore cert.p12 \
        -srcstorepass secret \
        -alias unifi \
        -noprompt
docker$ exit
host$ docker stop ID_HERE
host$ docker start ID_HERE

I’m mostly happy with the Ubiquiti access points. I have an AP-AC-LR and an AP-M. My complaints are:

  • When I reported a bug about access to SSH on non-management interfaces, they responded by turning off management over IPv6 Continue reading

Tiling window manager

A couple of months ago it occurred to me that I’ve been manually tiling my windows. That is, I use all the screen real estate, and don’t have windows overlapping each other.

In various window manages (and on Windows) I have used Super+Left and Super+Right to divide the screen 50/50.

So why am I not running a tiling window manager? That’s literally what they do, and they allow more flexibility in how to tile, without wasting space.

Switching to tiling

A quick googling says that i3 is what I want. Fast, small, efficient. No bells and whistles.

I used it for a little while, but then because I wanted to make it even harder on myself, err… I mean to join the 21st century, I thought I’d switch from X11 to Wayland, too. Luckily there’s a Wayland Compositor that’s equilavent to the i3 Window Manager called Sway.

It’s great! I knew X11 and Gnome had issues, but I didn’t realize just how much better I feel when I don’t have to deal with their deficiencies.

Like:

  • screen tearing when scrolling in terminal windows
  • changing focus can take up to a second, sometimes
  • X11 resets keyboard settings when it bloody feels Continue reading

Bypassing safety check for an obviously safe change

This is less concrete technical than my usual blog post.

For every 100 changes we’re 99% sure won’t cause an outage, one will

It’s actually hard to be 99% sure of anything. I’m not 99% sure today’s Thursday. I say that because more often than one day in a hundred, I’ll think “hmm… feels like Wednesday” when it’s not.

I just closed my eyes and tried to remember what time it is. I don’t think I can guess with 99% accuracy what hour I’m in. (but to be fair, it’s de-facto Friday afternoon today, as I’m off tomorrow).

Anyway… the reason I say this is that this should be kept in mind every time someone comes and says they want to circumvent some process for a change that they are absolutely sure won’t cause an outage, that can actually be put into numbers. And those numbers are “you are not 100% sure of anything”.

By saying you are 99% sure this won’t cause an outage (and are you right about that?) you are saying that for every 100 requests like yours that will bypass normal checks, there will be an outage. You are taking on an amortized 1% of Continue reading

BPF: The future of configs

BPF has some wow-presentations, showing how it enables new performance measuring and tracing. Brendan Gregg has a whole bunch, for example. But I don’t think’s it’s very well explained just why BPF is such a big deal.

Most of the demos are essentially cool and useful looking tools, with an “oh by the way BPF made this happen”. Similar to how it’s common to see announcements about some software, where the very title of the announcement ends with “written in Go”. It gives a vibe of “so what?”.

If you’re interested in system tooling and configuration, and aren’t already aware of BPF, then this is for you.

I’m not an expert on BPF, but this will hopefully help someone else bootstrap faster.

bpftrace

bpftrace is really cool. Clearly it’s inspired by dtrace. But one should not mistake bpftrace for BPF. bpftrace is only yet another tool that uses BPF, albeit one that allows you to create trace points in a domain specific language.

This is not the full power of BPF. It’s not at all the big picture.

BPF and configs

Let’s take packet filtering as an example. Once upon a time in Linux there was ipfwadm. I Continue reading

A smarter emacs

I’ve been running Emacs for like 25 years. But I’ve never really configured it with anything fancy.

Sure, I’ve set some shortcut keys, and enabled global-font-lock-mode and set indent size, but that’s almost it.

All my coding is done in tmux&Emacs. One project gets exactly one tmux session. Window 0 is emacs. Window 1 is make && ./a.out (sometimes split panes to tail logs or run both server and client), and to run git commands. The remaining windows are used for various things like reading manpages etc….

I have that same workflow whether I’m editing a blog post or doing kernel programming.

This way I can work at my desk with large and plentiful screens, and then move to my laptop and everything continues working exactly the same.

tmux I’ve customized, but not that much with Emacs.

So, step one to get my coding environment to be less 1995, and more 2020: make my editor understand my code, and show me stuff about it.

I’m learning as I’m going, and writing what I’m learning. As always if you see something wrong then please leave a comment.

Code annotations and other semantic understanding

The way to do this is Continue reading

Measuring USB with bpftrace

File usb-bw.b:

#include <linux/usb.h>

interval:s:1 {
  printf("--------------------------\n");
  print(@total);
  print(@sum);
  clear(@sum);
  clear(@total);
}

kprobe:__usb_hcd_giveback_urb {
  $urb = (struct urb*)arg0;
  $dev = $urb->dev;
  @total = stats((uint64)$urb->actual_length);
  @sum[$dev->descriptor.idVendor,
       $dev->descriptor.idProduct,
       str($dev->product),
       str($dev->manufacturer)] = stats((uint64)$urb->actual_length);
}

Example run with a USB stick idling (appears to be probed once every two seconds), and starting and stopping some GNURadio sniffing with an USRP B200 at 10Msps:

$ sudo bpftrace usb-bw.b
Attaching 2 probes...
--------------------------
@total: count 317, average 20, total 6641

@sum[9472, 32, USRP B200, Ettus Research LLC]: count 315, average 20, total 6597
@sum[4871, 357, USB Mass Storage Devie, USBest Technology]: count 2, average 22, total 44

--------------------------
@total: count 6807, average 20, total 136552

@sum[9472, 32, USRP B200, Ettus Research LLC]: count 6807, average 20, total 136552

--------------------------
@total: count 8507, average 20, total 170852

@sum[9472, 32, USRP B200, Ettus Research LLC]: count 8505, average 20, total 170808
@sum[4871, 357, USB Mass Storage Devie, USBest Technology]: count 2, average 22, total 44

--------------------------
@total: count 979, average 20, total 20288

@sum[9472, 32, USRP B200, Ettus Research LLC]: count 979, average 20, total 20288

--------------------------
@total: count 2141, average 7319, total 15670428

@sum[4871, 357, USB Mass Storage Devie, USBest Technology]:  Continue reading

Transferring pictures with DStar

I’ve successfully experimented with sending pictures using the data portion of D-Star.

I did it in multiple ways, starting with the simplest and ending with the longest path (though not most complex).

Equipment is an Android phone, a Kenwood TH-D74, and an ICom IC-9700.

Simplex

First I did it the simplest way, using simplex between the radios.

You install the ICom RS-MS1A app (sigh, yes that’s the kind of useful naming scheme they have). You’d think this app is needed for the ICom radio, but no. The IC-9700 has Picture mode built in. I used this app for the Kenwood D74.

You start the app, select “Others (Bluetooth)”, and select the D74.

On the D74 you need to:

  1. Press 1 to go into VFO mode
  2. Select the right frequency
  3. Set the mode to digital (DV/DR)
  4. If it’s DR, switch it to DV in the digital menu.
  5. In the digital menu, switch it to DATA

Annoyingly, unlike the native picture mode in the IC9700, setting DATA mode on the D74 will not allow any voice transmission at all.

On the IC9700, just set the right frequency, switch to DV mode, and select Picture from the menu.

I won’t go into detail Continue reading

Troubleshooting KISS with bpftrace

This is the troubleshooting story about me finding out why some packets were getting dropped when running AX.25 over D-Star DV between a Kenwood TH-D74 and an Icom 9700.

Troubleshooting: “Trouble”, from the latin “turbidus” meaning “a disturbance”. “Shooting”, from American English meaning “to solve a problem”.

The end result is this post, and this is the troubleshooting story.

The setup: laptop->bluetooth->D74->rf->9700->usb->raspberry pi.

I’m downloading from the raspberry pi, with the laptop sending back ACKs. But one of the ACKs is not getting through.

axlisten -a clearly showed that the dropped packet was being sent from the laptop:

radio: fm M0XXX to 2E0XXX-9 ctl RR6-

But nothing received on the receiver side. I saw the D74 light up red to TX, and the 9700 light up green on RX, but then nothing. Error counters in ifconfig ax0 were counting up on the receiver side. So something is being sent over the air.

And it wasn’t the first packet. All the ones before it were fine. They were always fine. This packet was always dropped. It was always only that packet that caused it to stall. The window size was set to 2, so session establishment, RR0, RR2 Continue reading

Amateur packet radio walkthrough

An earlier version of this post that did data over D-Star was misleading. This is the new version.

This blog post aims do describe the steps to setting up packet radio on modern hardware with Linux. There’s lots of ham radio documentation out there about various setups, but they’re usually at least 20 years old, and you’ll find recommendations to use software that’s not been updated is just as long.

Specifically here I’ll set up a Kenwood TH-D74 and ICom 9700 to talk to each other over D-Star and AX.25. But for the latter you can also use use cheap Baofengs just as well.

Note that 9600bps AX.25 can only be generated by a compatible radio. 1200bps can be send to a non-supporting radio as audio, but 9600bps cannot. So both D-Star and AX.25 here will give only 1200bps. But with hundreds of watts you can get really far with it, at least.

I’ll assume that you already know how to set up APRS (and therefore KISS) on a D74. If not, get comfortable with that first by reading the manual.

DMR doesn’t seem to have a data mode, and SystemFusion radios don’t give the user access Continue reading

FT8 and IC9700

Setup

The basis for these instructions is this guide, but updated to reflect that the IC9700 is now directly supported by wsjtx and js8call.

Step one: connect a normal USB-A-B cable between the computer and the radio.

IC-9700 settings

  • Menu
    • Set
      • Connectors
        • MOD Input
          • USB Mod Level: 30% (default 50%)
          • DATA MOD: USB (default: ACC)
          • DATA OFF MOD: Leave as default (MIC, ACC)
        • CI-V
          • CI-V Baud Rate = 19200 (default: Auto)
          • CI-V address = keep default A2h
          • CI-V USB Port = Link to [REMOTE] (default: Unlink from [REMOTE])
          • CI-V DATA Baud Rate = 19200 (default: OFF)
  • When running, choose FIL1 (next to the mode in the top left)

wsjtx/js8call settings

  • Radio
    • Rig: Icom IC-9700
    • Baud rate: 9700
    • Data bits: 8
    • Stop bits: 1
    • Handshake: XON/XOFF
    • PTT method: CAT
    • Mode: Data/Pkt (USB also works, but will use DATA OFF MOD as audio, so that needs to be USB)
    • Serial port: /dev/ttyUSB0 (on my Linux machine at least)
  • Audio
    • Input: also_input.usb-Burr-Brown_from_TI_USB_Audio_CODEC-0.analog-stereo
    • Output: also_output.usb-Burr-Brown_from_TI_USB_Audio_CODEC-0.analog-stereo
  • Frequencies
    • Right click on the “Working frequencies” table and insert 144.174 and 432.065 for FT8. I’m not sure about the 70cm frequency, as Continue reading

APRS

Another post in my burst of amateur radio blog posts.

To say that the documentation for APRS is not great is an understatement. What should be the best source of information, aprs.org, is just a collection of angry rants by the inventor of APRS, angrily accusing implementations and operators of using his invention the wrong way. There’s no documentation about what the right way is, just that everyone is wrong.

So here I’ll attempt to write down what it is, in one place, in an effort to both teach others, and for people who know more than me to correct me.

The best source of APRS information for me has actually been Kenwood radio manuals. See resources at the bottom.

APRS in short

APRS is a way to send short pieces of digital information as packets of data. The messages are:

  • Status about you
    • Your position (optionally not exact)
    • Your heading
    • Your QSY (frequency you’re tuned to if someone wants to call)
  • Weather reports
  • Status about “items” and “objects”. This is objects that are not you, and aren’t a radio. For example where the meeting point is, or a hurricane.
  • Short messages

The protocol

As an operator you Continue reading

Amateur radio digital voice

It’s a mess.

This post is my attempt at a summary of amateur radio digital voice modes, and what I think of them.

I’m not an expert, so if you have more experience then your opinion is likely more valid than mine. But hopefully at least I’m getting the facts right. Please correct me where I’m mistaken.

Analog and digital voice

In the beginning there was only analog. Traditionally on HF you used SSB, and on VHF/UHF you use FM. Analog works, and while yes there are different modes, radios tend to support all of them, or at least the common ones (e.g. most VHF/UHF radios don’t support SSB, because most traffic there is FM). Usually HT traffic is VHF/UHF FM, and for SSB while there is LSB and USB, radios will support both.

But analog isn’t perfect. By going digital we can send metadata such as call signs, positions, and even pictures and files. And for audio quality digital will get rid of the static of analog noise. Digital works better for longer distances, uses less spectrum, and retains voice clarity much longer.

Yes, there’s a sharp cliff when digital voice modes can no longer Continue reading

Yaesu FT3D vs Kenwood D74

I’ve had a Kenwood TH-D74 for almost two years now, and was curious to get a sense of what the competition is like. Seems like everyone’s recommending the Yaesu FT3D. So I got one, and I think I’ve played around with it enough now to have an informed opinion.

Summarizing the feeling of them, while I have my complaints about the usability of the D74, the FT3D is like a time machine back to the 90s in how well the interface is though through.

I’m sneaking in some mentions of the AnyTone 878UV too. But I’ve not used it enough to have a solid opinion yet.

Programming

With the FT3D upgrading the firmware is a two step process, where you have to flip a little hidden switch first to “up”, to upgrade one firmware, then to “down”, to upgrade the other. And then flip it back to “middle” for normal mode.

The FT3D programming software costs $25 and comes with a special cable, but the software also seems downloadable from their website. The USB cable seems to require a special driver. I guess that’s what you’re paying for. At least you can download the software and put the data on Continue reading

20 years of maintaining an open source program

It’s been almost 10 years since my previous post about this. And 20 years since 2000-02-24, which is when arping 0.1 was released. It was a 208 line C file, with a hand made Makefile.

As of today when Arping 2.21 is overdue to be released, the code in .c and .h files (excluding tests) is 3863 lines, and it uses the amazing autotools framework for analyzing dependencies.

I’ve recently had the displeasure of working with cmake, which is just the worst. Why anyone would think cmake is even remotely acceptable I’ll never understand.

CMake sucks

But the Arping story continues. It isn’t getting many new major features. Still, since the last post there’s been 205 commits, and 10 releases.

Things like:

  • Change from gettimeofday() to clock_gettime(), when available. More info about that in this blog post.
  • Don’t check for uid=0 and stop. Capabilities can come in other ways
  • Change from poll() to select() to work around bug in MacOS X
  • Use nice and modern getifaddrs() to resolve interfaces
  • Update documentation
  • Improve error messages
  • Update author email address
  • Fix warnings and general code cleanup
  • Used coverity to find and fix suspicious code
  • Add some more stats to output
  • Continue reading

Broadband RF scanner

One great thing about software defined radio is that you can become less blind to the invisible world of radio waves that’s all around us. One simple thing is to do a survey of the spectrum, to see what parts are busy.

More practically you can also use this to find which Wifi channels are least busy, so that you can get optimal performance on your network. Counting the number of networks is not a good indicator, since one network may be completely unused, while another is used 24/7 to stream Netflix. And some networks are hidden anyway, making them no more secure, but more annoying.

[GNU Radio][gnuradio] has a bunch of building blocks for some interactive peeking at spectrums, but there’s still some assembly required in order to make actually useful things.

To do a survey I used a USRP B200 with a [broadband spiral antenna. If you’re only interested in the Wifi spectrum then a 2.4/5GHz antenna is a better choice.

You can probably use a cheaper SDR, but you need to make sure it sends frequency tag updates in GNU Radio, so the block knows which frequency is tuned, as it moves across the spectrum.

Overall Continue reading

TCP MD5

TCP_MD5 (RFC 2385) is something that doesn’t come up often. There’s a couple of reasons for that, good and bad.

I used it with tlssh, but back then (2010) it was not practical due to the limitations in the API on Linux and OpenBSD.

This is an updated post, written after I discovered TCP_MD5SIG_EXT.

What it is

In short it’s a TCP option that adds an MD5-based signature to every TCP packet. It signs the source and destination IP addresses, ports, and the payload. That way the data is both authenticated and integrity protected.

When an endpoint enables TCP MD5, all unsigned packets (including SYN packets) are silently dropped. For a signed connection it’s not even possible for an eavesdropper to reset the connection, since the RST would need to be signed.

Because it’s on a TCP level instead of part of the protocol on top of TCP, it’s the only thing that can protect a TCP connection against RST attacks.

It’s used by the BGP protocol to set a password on the connection, instead of sending the password in the handshake. If the password doesn’t match the TCP connection doesn’t even establish.

But outside of BGP it’s essentially Continue reading

1 2 3 4 5 6