RustRadio improved API 0.4

Since last time, I’ve improved the API a bit. That last post was about API version 0.3. Now it’s on 0.4, and I think it’s getting pretty decent.

0.3 could never have worked very well. The API was VecDeque-based, which means it could not provide a linear view (a slice) of all the data in the buffer.

The 0.4 API is simpler. You get a typed slice, and you read or write to, it as appropriate. Because all streams are currently single writer, single reader, the code is simple, and requires minimal amount of locking.

It’s simpler, but I switched to using memory mapped circular buffers, with a slice as the stream interface. This means that the buffer is allocated only once, yet both reader and writer can use all space available to them, linearly, without having to worry about wrapping around.

The code is still at https://github.com/ThomasHabets/rustradio. I registered the github org rustyradio, too. rustradio was taken. I sent a message to the owner, since it seems to not have any real content, but have not heard back.

Unsafe code

To make this multiuser stream I did have to write some Continue reading

SDR transmit and clean signals

If you have a transmit capable SDR, you may have heard that you need to filter its output, before transmitting to the world. Certainly before amplifying the signal.

I have a TinySA Ultra spectrum analyzer, and will here show you some screenshots about just how true that is.

I tested this with my USRP B200, transmitting a pure carrier around 145MHz and 435MHz.

Oh, and a word of caution: If you want to replicate this, make sure to add an inline attenuator, to not damage your spectrum analyzer. I had a cheap 40dB one, but the values in the graphs have been adjusted to show the real signal strength, as if I hadn’t.

tl;dr

1. Harmonics can be almost as strong as the fundamental. You need to filter these.
2. Transmitting at maximum output gain may cause lots of unwanted signals right around your fundamental. You cannot filter these. You need to not generate them.

Harmonics

Reducing the output gain did not meaningfully fix the problem. The best I saw from using half output gain was to make the strongest harmonic 9dB less than the fundamental. That’s way too strong.

I added a cheap band pass filter (FBP-144), which made Continue reading

AX.25 and 9600bps G3RUH decoding

I’ve been coding more on my rust SDR framework, and want to improve my ability to send/receive data packets efficiently and reliably.

There are two main ways I use learn to do this better: designing a new protocol, and making the best implementation possible for an existing one. This post is about refining the latter.

AX.25 and APRS

First a detour, or background.

AX.25 is the standard amateur radio data protocol. It’s mostly an OSI layer 2-4 protocol, mashing the layers together into one. Contrast this with IP, which just encapsulates the next layer.

Layer 3 (IP stack equivalent: IP itself) consists of the ability to add, in addition to source and destination, a variable number of intermediate repeaters. This allows limited source routing. In APRS the repeaters are usually not named, but instead uses “virtual” hops like WIDE1-1.

Layer 4 (IP stack equivalent: TCP and UDP) allows both connected and disconnected communication channels. In my experience connected AX.25 works better over slow simplex radio than TCP. If TCP was ever optimized for high delay low bandwidth, it’s not anymore.

For the physical layer, there are three main “modems”:

1. 300 baud bell 103, used Continue reading

Setting up secure wifi

If you don’t set a password on your wifi, then not only can anyone connect, but it’s not even encrypted. This means that even when an open network gives you a captive portal, that could actually be an attacker giving you a fake portal. Even if the portal is HTTPS, because you may be connected to https://evil-fake-portal.com.

That is solved in WPA3, where even open networks become encrypted.

Of course, the attacker can just set up a fake access point, and you’ll connect, none the wiser. Even if the network has a password, the attacker only needs to know that password in order to fake it.

Before WPA3, passwords can easily be brute forced offline. A few years ago I calculated that it would cost about $70 to crack the default generated 8 character random passwords used by a popular ISP here in London, using some GPUs in Google Cloud. I’m sure it’s cheaper now.

That’s potentially years of free use of your neighbours wifi, for just the cost of a couple of months of paying for your own.

But that’s illegal, of course. This post is about protecting you against these attacks, not performing them.

If you Continue reading

RustRadio, and Roast My Rust

I’m learning Rust. And I like playing with software defined radio (SDR). So the natural project to take on to learn Rust is to write a crate for making SDR applications. I call it RustRadio.

I have something that works, and seems pretty OK. But before marking a 1.0.0 release I want to see if I can get some opinions on my use of the Rust language. Both in terms of design, and more clippy-like suggestions.

Hence: Roast My Rust. File a github issue, email me, or tweet at me. Tell me I’m doing it wrong.

• RustRadio code: https://github.com/ThomasHabets/rustradio
• RustRadio docs: https://docs.rs/rustradio/latest/rustradio/
• The first application: https://github.com/ThomasHabets/sparslog

What my priorities are

There are two API surfaces in RustRadio; the Block API (for writing blocks), and the Application API (for writing applications that use blocks). I want them to be good, and future proof, so that I don’t have to change every block and every application, after adding a feature or improving the API.

The blocks will need to be thread safe, even though the scheduler is currently single threaded.

For the streams between blocks I’ll eventually want to make a more fancy, but unsafe circular Continue reading

Downloading web resources

Last time I went to the dentist, they offered to use a fancy scanner to better be able to show me my teeth.

Who can say no to that? I already for fun got a 3D scan of my brain, so why not teeth too?

I requested the data, and got a link to a web UI. Unfortunately it was just a user friendly 3D viewer, without any download button.

Here’s how I extracted the 3D data:

1. Open Chrome developer console, e.g. by pressing Ctrl-Shift-C (I hate it that Chrome hijacked this. Every single day I press Ctrl-Shift-C to copy, and it throws up this thing)
2. Close the stupid “what’s new” spam, that nobody in the history of ever has wanted to see.
3. Go to the ‘Network’ tab.
4. Reload the page.
5. Right click on any item in the list, and choose “Save all as HAR with content”. No, I don’t know why I can’t just save that one resource.
6. A HAR file is a JSON file archive, essentially.

   $ jq '.log | keys' foo.har
   [
     "creator",
     "entries",
     "pages",
     "version"
   ]
   $ jq '.log | .entries[0].request | keys' foo.har
   [
     "bodySize",
     "cookies",
     "headers",
     "headersSize",
      Continue reading

The unreasonable effectiveness of radio

Light and radio transmissions are the same thing, just using different frequencies.

How is it reasonable that I can transmit with a radio using the energy of a low energy light bulb (10 watts), and easily chat with someone 1800km away?

Even if you imagine a perfectly dark world, where the only light bulb is a 10W bulb in Sweden; How is it even possible that this could be seen in Italy? It’s not even a spotlight! It’s only vaguely aimed away from east-west, in favour of up, north, and south.

My little light bulb (radio) could be seen in (approximately) all of Europe. ~750 million people potentially could have received it at the same time. With 10 watts.

“Blink blink” — All of Europe can see my little lightbulb.

And this isn’t some specialized super duper antenna, nor was it set up by an expert, fine tuning everything. I just put up the antenna in a PVC pipe and connected it. We can’t even credit fancy computers digging signals out of the noise. This was not FT8, this was PSK31. I’m sure voice would also have worked.

I also used FT8 with these same 10W to check off Continue reading

Multichannel fast file transfers over AX.25

Lately I’ve been thinking about a better data protocol for amateur radio.

“Better” is, of course, relative. And the space is so big. Are we talking HF or VHF/UHF? Should it work with existing radios (just working the audio spectrum), or be its own radio? Should it be just RF improvements, or higher networking layers?

File transfers on the application layer

In my previous post I started off trying ZMODEM, but was fairly disappointed. The Linux AX.25 implementation sucks, and ZMODEM is too chatty. Every roundtrip is expensive. But even tuning the parameters, there are better ways to avoid needless retransmits and roundtrips.

I’ve started a tool called hamtransfer. The implementation is currently only point-to-point, but the protocol will work for more “bittorrent” style too.

It uses Raptor codes, but I’ll save you some time: It encodes the file (it calls a “block”) into smaller chunks (it calls “symbols”). It then sends the symbols to the receiver, which will be able to reassemble the original block.

The trick is that the set of symbols is infinite, and the block can be assembled by almost any subset of symbols. If the block is 10kB, then with more than Continue reading

ZModem over amateur radio

While I have built a file transfer protocol for AX.25, I also wanted to try old BBS era protocols. They’re more tested and standardized.

The easiest way is probably to take lrzsz and let it talk over AX.25 connected sockets. Ideally socat should be enough, but it seems that it does not support AX.25.

That’s actually fine, because ideally I want to run on my authenticated wrapped AX.25 (while encryption, obscuring the meaning, is banned, signatures are not).

So I had to make an adapter that bridges stdin/stdout to AX.25. Simple enough.

The setup is two Kenwood TH-D74s, set up the same way as before, in 9600bps.

D74 -> rfcomm -> kissattach -> axpipe -> socat -> lrzsz

The D74 is a great radio. It has the best text entry, menu system, and APRS support of any handheld radio I’ve seen, and it also has a built-in TNC (“modem”) that works both in 1200bps and 9600bps.

First, just for fun, let’s try YModem.

YModem

socat EXEC:'sz --ymodem axpipe.cc' EXEC:'./axpipe -r radio1 -l 0'
socat EXEC:'rz --ymodem -t 100'    EXEC:'./axpipe -r radio6 -s M6VMB-12 -c M0THC-1'

On ARM and RISC-V I Continue reading

Linking statically, and glibc breaking userspace for fun

glibc is annoyingly eager to break userspace. You can’t just build something that only depends on libc and expect it to work on all linux systems of that architecture.

I don’t know why Linus Torvalds keeps insisting “we do not break userspace” as a policy for the kernel when libc seems to make that exact thing a hobby. And either way the userspace programs break.

Compiling static (including libc) is frowed upon, and has even had known breakages left unaddressed.

E.g. setlocale() had a strange bug where for new threads you had to first set the locale to the wrong locale, and then call it again to set it to the right one. Otherwise the new thread would be in a weird state where the local is wrong, but it thought it’s right, so won’t allow you to change it to what it thought it already was.

I can’t find the bug now (I ran into this around 2004-2005), but the official response was basically “well don’t compile statically, then”.

And DNS can be broken with static glibc. “a statically linked glibc can’t use NSS (Name Service Switch) modules from a different glibc version, so if you statically link Continue reading

Counting current live readers

Once upon a time it was popular to put a counter on your web page, to show how many people had visited the site before you. I thought it be more fun, and less bragging about how long the page has existed, if it just showed who’s reading it now.

As I mentioned in a previous post, I’m learning Rust. My teaching project has been to make this web widget that shows the current number of browsers that that have the page open.

You see this counter here on the blog in the top right.

The idea is pretty simple. Have some javascript open a websocket to a server, and stream down messages with the current count, as it changes. When a client connects or disconnects, inform all other clients of the new total.

This does mean that it needs to keep one TCP connection open per client, which may be too costly for some sites. Especially since I’m putting it behind an nginx, so the machine needs to keep 3x the state.

I’m not logging anything to disk, nor sharing anything between the clients except for the current count. It’s just an amusing publicly visible presence counter.

Actually, because Continue reading

Tracing function calls

Sometimes you want to see functions of a library, as they’re called. I know of two ways of doing this.

Let’s have a super simple test program:

#include<iostream>
#include<unistd.h>

void func1() {}
void func2() {}

int main()
{
  std::cout << "Hello world\n";
  func1();
  func2();
  func1();

  // Wait a bit for bpftrace to be able to aquire the function name.
  // Not applicable for something that doesn't exist.
  sleep(1);
}

bpftrace

Start a bpftrace in one terminal, and run the program in another.

$ sudo bpftrace -e 'uprobe:./a.out:func* { print(func); }'
Attaching 2 probes...
func1()
func2()
func1()

GDB

$ gdb a.out
[…]
(gdb) rbreak func.*
[…]
(gdb) commands
Type commands for breakpoint(s) 1-3, one per line.
End with a line saying just "end".
>silent
>bt 1
>cont
>end
(gdb) r
Starting program: […]/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
#0  0x0000555555555215 in _GLOBAL__sub_I__Z5func1v ()
Hello world
#0  0x000055555555516d in func1() ()
#0  0x0000555555555174 in func2() ()
#0  0x000055555555516d in func1() ()
[Inferior 1 (process 424744) exited normally]
(gdb)

Which to use?

bpftrace is lower (but Continue reading

RISC-V optimization and -mtune

I’ve been getting into RISC-V optimization recently. Partly because I got my SiFive VisionFive 2, and partly because unlike x86 the number of RISC-V instructions is so managable that I may actually have a chance at beating the compiler.

I’m optimizing the inner loops of GNURadio, or in other words the volk library. I’ve been getting up to a about a doubling of the speed compared to the compiled C code, depending on the function.

But it got me thinking how far I could tweak the compiler and its options, too.

Yes, I should have done this much sooner.

Many years ago now I built some data processing thing in C++, and thought it ran too slowly. Sure, I did a debug build, but how much slower could that be? Half speed? Nope. 20x slower.

Of course this time I never compared to a debug build, so don’t expect that kind of difference. Don’t expect that it’ll reach my hand optimized assembly either, imperfect as it may be.

The test code

This may look like a synthetic benchmark, in simplified C++:

complex volk_32fc_x2_dot_prod_32fc_generic(const vector<complex> &in1,
                                           const vector<complex> &in2)
{
  complex res;
  for (unsigned int i = 0; i  Continue reading

VisionFive 2 quickstart

RISC-V small computer

For a long time I’ve wanted something Raspberry-pi-like but with RISC-V. And finally there is one, and a defensible price! Especially with the Raspberry Pi 4 shortage this seemed like a good idea.

This post is my first impressions and setup steps.

It’s just like when I was a kid!

When I was in my late teens I was playing with different architectures, mostly using discarded university computers. It was fun to have such different types of computers. Back then it was SPARC (And UltraSparc), Alpha, and x86. Maybe access to some HPPA. I even had a MIPS (SGI Indigo 2).

Nowadays instead of SPARC, Alpha, and x86 it’s ARM, RISC-V, and x64.

Luckily they can be smaller nowadays. Before I left home my room had more towers of computers than it had furniture. In my first flat I had a full size rack!

Write SD card

pv starfive-jh7110-VF2_515_v2.5.0-69-minimal-desktop.img \
   | sudo dd of=/dev/sda

Repartition SD card

We need to repartition, because the boot partition is way too small. It only fits one kernel/initrd, which became a problem I ran into.

Unfortunately gparted doesn’t seem to work on disk images. It Continue reading

Learning Rust, assisted by ChatGPT

I finally got around to learn Rust. Well, starting to.

It’s amazing.

I’m comparing this to learning Basic, C, C++, Erlang, Fortran, Go, Javascript, Pascal, Perl, PHP, Prolog and Python. I wouldn’t say I know all these languages well, but I do know C, C++, Go, and Python pretty well.

I can critique all these languages, but I’ve not found anything frustrating or stupid in Rust yet.

Rust is like taking C++, but all the tricky parts are now the default, and checked at compile time.

Copies

With C++11 we got move semantics, so we have to carefully disable the (usually) default-created copy constructor and copy assignments, or if you have to allow copies then every single use of the type has to be very careful to not trigger copies unless absolutely necessary.

And you have to consider exactly when RVO kicks in. And even the best of us will sometimes get it wrong, especially with refactors. E.g. who would have thought that adding this optimization would suddenly trigger a copy of the potentially very heavy Obj object, a copy that was not there before?

--- before.cc   2022-12-28 10:32:50.969273274 +0000
+++ after.cc    2022-12-28 10:32:50.969273274 +0000
 Continue reading

Better pipes

In my blog post The uselessness of bash I made a tool to improve pipes in shell, to assemble a better pipeline.

It solves the problem, but it’s a bit too different, with its own language.

While complaining with some people at work that one of the main features of shell (the pipe operator) is broken, someone joked that it should be replaced by a protobuf based protocol.

But on second thought it’s not really a joke.

How about instead of this:

$ goodpipe <<EOF
[
  ["gsutil", "cat", "gs://example/input-unsorted.txt"],
  ["sort", "-S300M", "-n"],
  ["gzip", "-9"],
  ["gsutil", "cp", "-", "gs://example/input-sorted-numerically.txt.gz"]
]
EOF

how about this:

$ wp -o gsutil cat gs://example/input-unsorted.txt \
  | wp -io sort -S300M -n \
  | wp -io gzip -9 \
  | wp -i gsutil cp - gs://example/input-sorted-numerically.txt.gz

It doesn’t use protobufs, but a simpler regular protocol. This in order to avoid well known bugs types. Before implementing any protocol also see formal theory and science of insecurity.

First I hacked it together in Go, but I think the main implementation I’ll maintain is the one I made while porting it to Rust, as a way to learn Rust. The Continue reading

Can ChatGPT debug my code? Can it write better code?

Everyone’s talking about it, so why should I not add something too.

Interview question

ChatGPT is pretty amazing. I asked it an interview question, and it just plain solved it. Well, there was one real world imperfection. I asked it to fix that problem, and then it went off the rails. The more I hinted, the worse the code got.

I won’t give the details here, because I want to keep using the question / topic.

Finding bugs

So I wondered if it could find bugs. First an easy one:

Q:

What's the problem with this C++ code:
int* foo() {
  auto ret = new int;
  int r2 = *ret * 2;
  return &r2;
}

A:
There are several problems with the code you provided:

1. The function foo is returning a pointer to a local variable r2,
   which is not safe because the memory for r2 is deallocated when the
   function returns. This can cause undefined behavior when you try to
   access the returned pointer after the function has returned.
2. The function foo is allocating memory for an integer with new int,
   but it is not deallocating that memory before the function
   returns. This can lead to a  Continue reading

Terminator 1 is the best Terminator movie

And now for something completely different.

I’ve off and on thought about this for years, so it needed to be written down.

Terminator 1 is the best Terminator movie

Obviously SPOILERS, for basically all Terminator movies.

Summary of reasons

• The robot is really not human.
• It’s a proper time loop, with a bonus that none of the players in the movie know it.

I’m aware of The Terminator Wiki, but I don’t care about it. My opinions are on the movies as movies.

The behavior of the terminator

In Terminator 1 (T1) Arnold is clearly a robot in human skin. At no point do you believe it’s a human. The only reason people don’t stop and scream and point, is that “I’m being silly, that’s clearly impossible”. But Arnold spends the whole movie in the uncanny valley, the kind in 2022 reserved for realistically generated CGI characters.

It’s very nearly a perfect movie. Just take his first dialog. “Nice night for a walk”, the punks say. They are saying this to a machine that has never talked to a human before, so its response is complete nonsense. It just repeats the words back to them.

It’s a Continue reading

Fast zero copy static web server with KTLS

I’m writing a webserver optimized for serving a static site with as high performance as possible. Counting every syscall, and every copy between userspace and kernel space.

It’s called “tarweb”, because it serves a website entirely from a tar file.

I’m optimizing for latency of requests, throughput of the server, and scalability over number of active connections.

I won’t go so far as to implement a user space network driver to bypass the kernel, because I want to be able to just run it in normal setups, even as non-root.

I’m not even close to done, and the code is a mess, but here are some thoughts for now.

First optimize syscall count

Every syscall costs performance, so we want to minimize those.

The minimum set of syscalls for a webserver handling a request is:

1. accept() to acquire the new connection.
2. epoll_ctl() to add the fd.
3. epoll_wait() & read() or similar. (ideally getting the whole request in one read() call)
4. epoll_wait() & write() or similar. (again ideally in one call)
5. close() the connection.

There’s not much to do about accept() and read(), as far as I can see. You need to accept the connection, and you need to Continue reading

Integer handling is broken

Floating point can be tricky. You can’t really check for equality, and with IEEE 754 you have a bunch of fun things like values of not a number, infinities, and positive and negative zero.

But integers are simple, right? Nope.

I’ll use “integers” to refer to all integer types. E.g. C’s int, unsigned int, gid_t, size_t, ssize_t, unsigned long long, and Java’s int, Integer, etc…

Let’s list some problems:

• You can’t parse unsigned integers in C/C++ with standard functions.
• Casting between types silently discards information (C, Java, Go, Rust, Haskell).
• Casting between some other types changes the semantic meaning of the number, even when technically no information is lost.
• The state of integers in Javascript is its own mess.

What’s wrong with casting?

Casting an integer from one type to another changes three things:

1. The type in the language’s type system.
2. Crops values that don’t fit.
3. May change the semantic value, by changing sign.

The first is obvious, and is even safe for the language to do implicitly. Why even bother telling the human that a conversion was done?

But think about the other two for a minute. Is there any reason that you want your Continue reading