Blargh, Author at NetworkingNexus.net

Blargh

Author Archives: Blargh

io_uring, kTLS and Rust for zero syscall HTTPS server

Around the turn of the century we started to get a bigger need for high capacity web servers. For example there was the C10k problem paper.

At the time, the kinds of things done to reduce work done per request was pre-forking the web server. This means a request could be handled without an expensive process creation.

Because yes, creating a new process for every request was something perfectly normal.

Things did get better. People learned how to create threads, making things more light weight. Then they switched to using poll()/select(), in order to not just spare the process/thread creation, but the whole context switch.

I remember a comment on Kuro5hin from anakata, the creator of both The Pirate Bay and the webserver that powered it, along the lines of “I am select() of borg, resistance is futile”, mocking someone for not understanding how to write a scalable webserver.

But select()/poll() also doesn’t scale. If you have ten thousand connections, that’s an array of ten thousand integers that need to be sent to the kernel for every single iteration of your request handling loop.

Enter epoll (kqueue on other operating systems, but I’m focusing Continue reading

Exploring RISC-V vector instructions

It finally happened! A raspberry pi like device, with a RISC-V CPU supporting the v extension. Aka RVV. Aka vector instructions.

I bought one, and explored it a bit.

SIMD background

First some background on SIMD.

SIMD is a set of instructions allowing you to do the same operation to multiple independent pieces of data. As an example, say you had four 8 bit integers, and you wanted to multiply them all by 2, then add 1. You could do this with a single operation without any special instructions.

    # x86 example assembly.

    mov eax, [myvalues]  # load our four bytes.
    mov ebx, 2           # we want to multiply by two
    imul eax, ebx        # single operation, multiple data!
                         # After this, eax contains 0x02040608
    add eax, 0x01010101  # single operation, multiple data!
                         # After this, eax contains 0x03050709
    mov [myvalues], eax  # store back the new value.

section .data
  myvalues db 1,2,3,4

Success, right? No, of course not. This naive code doesn’t handle over/underflow, and doesn’t even remotely work for floating point data. For that, we need special SIMD instructions.

x86 and ARM have gone the way of fixed sized registers. In 1997 Intel introduced MMX, to great Continue reading

Rebuilding FRR with pim6d

Short post today.

Turns out that Debian, in its infinite wisdom, disables pim6d in frr. Here’s a short howto on how to build it fixed.

$ sudo apt build-dep frr
[…]
$ apt source frr
[…]
$ cd frr-8*
$ DEB_BUILD_PROFILES=pkg.frr.pim6d dpkg-buildpackage -us -uc -b
$ sudo dpkg -i ../frr_*.deb

Then you can enable pim6d in /etc/frr/daemons and restart frr.

Not that I managed to get IPv6 multicast routing to to work over wireguard interfaces anyway. Not sure what’s wrong. Though it didn’t fix it, here’s an interesting command that made stuff like ip -6 mroute look like it should work:

$ sudo smcroutectl  add LAN ff38:40:fd11:222:3333:44:0:1122 wg-foo

Pike is wrong on bloat

This is my response to Rob Pike’s words On Bloat.

I’m not surprised to see this from Pike. He’s a NIH extremist. And yes, in this aspect he’s my spirit animal when coding for fun. I’ll avoid using a framework or a dependency because it’s not the way that I would have done it, and it doesn’t do it quite right… for me.

And he correctly recognizes the technical debt that an added dependency involves.

But I would say that he has two big blind spots.

He doesn’t recognize that not using the available dependency is also adding huge technical debt. Every line of code you write is code that you have to maintain, forever.
The option for most software isn’t “use the dependency” vs “implement it yourself”. It’s “use the dependency” vs “don’t do it at all”. If the latter means adding 10 human years to the product, then most of the time the trade-off makes it not worth doing at all.

He shows a dependency graph of Kubernetes. Great. So are you going to write your own Kubernetes now?

Pike is a good enough coder that he can write his own editor (wikipedia: “Pike has written many text Continue reading

Connection coalescing breaks the Internet

Connection coalescing is the dumbest idea to ever reach RFC status. I can’t believe nobody stopped it before it got this far.

It breaks everything.

Thus starts my latest opinion post.

What is connection coalescing?

It’s specified in the RFC for HTTP/2 as connection reuse, but tl;dr: If the IP address of host A and B overlap, and host A presents a TLS cert that also includes B (via explicit CN/SAN or wildcard cert), then the client is allowed to send HTTP requests directed to B on the connection that was established to A.

Why did they do that?

To save roundtrips and TLS handshakes. It seems like a good idea if you don’t think about it too much.

Why does it break everything?

I’ll resist just yelling “layering violation”, because that’s not helpful. Instead I’ll be more concrete.

Performing connection coalescing is a client side (e.g. browser) decision. But it implicitly mandates a very strict server architecture. It assumes that ALL affected hostnames are configured exactly the same in many regards, and indeed that the HTTP server even has the config for all hostnames.

Concrete things that this breaks:

The server can’t have a freestanding TLS termination layer, Continue reading

An AX.25 implementation in Rust

After having written a user space AX.25 stack in C++, I got bitten by the Rust bug. So this is the third time I’ve written an AX.25 stack, and I’ve become exceedingly efficient at it.

Here it is:

The reason for a user space stack remains from last time, but this time:

It’s written in Rust. Yay! I know people say Rust has a honeymoon period, but I guess that’s where I am, still.
It’s a normal library first. The previous C++ implementation started off as microservices, which in retrospect was needlessly complex and put the cart before the horse.

I’ve added almost an excessive amount of comments to the code, to cross reference with the specs. The specs that have a few bugs, by the way.

Rust

I’m not an expert in Rust, but it allows for so much more confidence in your code than any other language I’ve tried.

I think I know enough Rust to know what I don’t fully know. Sure, I’ve successfully added lifetime annotations, created macros, and built async code, but I’m not fluent in those yet.

Interestingly, Continue reading

Is your TLS resuming?

There are two main ways that a TLS handshake can go: Full handshake, or resume.

There are two benefits to resumption:

it can save a round trip between the client and server.
it saves CPU cost of a public key operation.

Round trip

Saving a round trip is important for latency. Some websites don’t use a CDN, so a roundtrip could take a while. And even those on a CDN can be tens of milliseconds away. Maybe won’t matter much for a human, but roundtrips can kill the performance of something that needs to do sequential connections.

E.g. Australia is far away:

$ ping -c 1 -n www.treasury.gov.au
PING treasury.gov.au (3.104.80.4) 56(84) bytes of data.
64 bytes from 3.104.80.4: icmp_seq=1 ttl=39 time=369 ms

That’s about a third of a second. Certainly noticeable to a human. Especially since rendering a web page usually requires many connections to different hosts.

For TCP based web requests (in other words: not QUIC), there’s usually four roundtrips involved (slightly simplified):

TCP connection establishment.
ClientHello & ServerHello.
Client & Server ChangeCipherSpec.
HTTP request & response.

So from the UK to Australia, that’s about Continue reading

Rust is faster than C, even before I added SIMD

I found some old C code of mine from around 2001 or so. I vaguely remember trying to make it as optimized as possible. Sure, I was still a teenager, so it’s not state of the art. But it’s not half bad. I vaguely suspect I could do better with better optimization for cache lines, but it’s pretty good.

On my current laptop it does about 12 million passwords per second, single threaded.

Because I’m learning Rust, I decided to port it, and see how fast rust is.

Imagine my surprise when even the first version in Rust was faster. (Yes, I rebuilt the old C code with a modern compiler and its optimizations)

The first Rust version was about 13 million passwords per second.

Why is that? It’s basically the same as the C code. Maybe Rust can take advantage of knowing there’s no pointer aliasing (the reason usually quoted for why Fortran can be faster than C)? Or maybe the memory layout just happened to become more cache friendly?

In any case, I think we can already say that Rust is at least as fast as C.

The code is on github.

SIMD (with Rust)

I realized, of Continue reading

Cross compiling Rust — Fixed

Set up build environment

rustup toolchain install nightly
rustup component add rust-src --toolchain nightly
apt install {binutils,gcc}-mips-linux-gnu

Create test project

cargo new foo
cd foo

Configure linker

mkdir .cargo
cat > .cargo/config.toml
[target.mips-unknown-linux-gnu]
linker = "mips-linux-gnu-gcc"
^D

Build

cargo +nightly build --release -Zbuild-std --target mips-unknown-linux-gnu

Change the “interpreter” to what the Ubiquiti system expects

cd target/mips-unknown-linux-gnu/release
patchelf --remove-needed ld.so.1 foo
patchelf --set-interpreter /lib/ld-musl-mips-sf.so.1 foo

Does it work?

$ ./foo
Hello, world!

Yay!

Cross compiling Rust to Ubiquiti access point

This is not the right way to do it, as will become abundantly clear. But it works.

Set up build environment

rustup toolchain install nightly
rustup component add rust-src --toolchain nightly
apt install {binutils,gcc}-mips-linux-gnu

Create test project

cargo new foo
cd foo

Build most of it

This will build for a while, then fail.

cargo +nightly build --release -Zbuild-std --target mips-unknown-linux-gnu

For some reason it’s trying to use cc to link. I tried putting this in Cargo.toml, but it does nothing:

[target.mips-unknown-linux-gnu]
linker = "mips-linux-gnu-gcc"

But I found a workaround.

Temporarily change `/usr/bin/cc` to point to the mips gcc

It does not work if you do this before the previous step.

PREV="$(readlink -v /usr/bin/cc)"
sudo rm /usr/bin/cc
sudo ln -s /usr/bin/mips-linux-gnu-gcc /usr/bin/cc

Link the program

Same command again

cargo +nightly build --release -Zbuild-std --target mips-unknown-linux-gnu

It should succeed. Yay.

Restore `/usr/bin/cc`

sudo rm /usr/bin/cc
sudo ln -s "${PREV?}" /usr/bin/cc

Change the “interpreter” to what the Ubiquiti system expects

cd target/mips-unknown-linux-gnu/release
patchelf --remove-needed ld.so.1 foo
patchelf --set-interpreter /lib/ld-musl-mips-sf.so.1 foo

Building it again

Probably easiest to rm -fr target, and go back to the step “Build most of it”.

Does it work?

$ ./foo
Hello, world!

Yay!

Use AGW for packet radio applications

When creating packet radio applications, there are several options on how to get the packets “out there”, and get them back. That is, how to interface with the modem.

Sure, you can write your own modem, and have the interface to the outside world be plain audio and PTT (push to talk, i.e. trigger transmit). But now you’re writing a modem, not an application. You should probably split the two, and have an interface between them.

KISS

You can use KISS, but it’s very limited. You can only send individual packets, so it’s only really good for sending unconnected (think UDP) packets like APRS. It’s not good for querying metadata, such as port information and outstanding transmit queue.

Think of KISS like a lower layer that applications shouldn’t think about. Like ethernet. Sure, as a good engineer you should know about KISS, but it’s not what your application should be interfacing with.

Linux kernel implementation

On Linux you can use AF_AX25 sockets, and program exactly like you do for regular internet/IP programs. SOCK_DGRAM for UI frames (UDP-like), and SOCK_STREAM for connected mode (TCP-like).

But the Linux kernel implementation is way too buggy. SOCK_STREAM works kinda OK, but does Continue reading

Meshtastic quick setup

I wanted some nice offline mid range chat app, for when I don’t have data, or data roaming is too expensive. I also want it to work for people who are not amateur radio licensed, since my girlfriend stubbornly refuses to be interested in that.

Looks like the answer I’m looking for is Meshtastic, preferably with LoRalora]. I bought a couple of Heltec V3 ESP32 LoRa OLED and the matching case.

Maybe I’ll buy a battery, but I’m fine just powering it from a USB power bank.

The documentation makes a fair bit of assumptions about the user knowing the name for what they want, and what firmware provides what.

In short, what I think I want is to ignore the Heltec firmware, and instead just treat the Heltec V3 as the hardware that Meshtastic runs on.

The recommended way to flash, and for some cases even use, is the Meshtastic Web UI. It uses browser integration for serial ports and bluetooth. A nice idea, but it was extremely unreliable for me. The flasher worked for one device, but not the other. The chat client never worked at all.

Here’s what worked reliably for me:

Download “stable” firmware Continue reading

Apollo 11 notes

I was re-reading the Apollo 11 mission reports, as one does, and decided to take some notes along the way.

If you’re interested in these things, I also highly recommend curiousmarc’s series on the Apollo comms hardware.

Notes

First time I’ve seen the word “doff”. Can’t wait to use it in daily conversation.

The rocket equation is a beast. The LM descent stage had 8’210kg of propellant. The ascent stage only 2’365kg.
– Volume 1, Page 50

In total 10’849kg out of 15’061 (72%) of the LM was propellant. (excluding the astronauts themselves)

The LM flown on Apollo 10 did not have the landing program in its computer. To prevent the temptation to land?
– Volume 1, Page 62

Armstrong’s parents were “Mr. and Mrs. Stephen Armstrong”. Michael Collins’ mother is mentioned, but her name is also lost to history, as she’s referred to as “Mrs. James L. Collins”. Only Buzz Aldrin’s mother is named (and what a name!), as Marion Moon Aldrin.

All three were born in 1930, making them turn 39 in 1969.
– Volume 1, Page 76-78

“High speed” data mode is 2400bps, divided into 240 bit blocks.
– Volume 1, Page 93

Aside from the Continue reading

RustRadio improved API 0.4

Since last time, I’ve improved the API a bit. That last post was about API version 0.3. Now it’s on 0.4, and I think it’s getting pretty decent.

0.3 could never have worked very well. The API was VecDeque-based, which means it could not provide a linear view (a slice) of all the data in the buffer.

The 0.4 API is simpler. You get a typed slice, and you read or write to, it as appropriate. Because all streams are currently single writer, single reader, the code is simple, and requires minimal amount of locking.

It’s simpler, but I switched to using memory mapped circular buffers, with a slice as the stream interface. This means that the buffer is allocated only once, yet both reader and writer can use all space available to them, linearly, without having to worry about wrapping around.

The code is still at https://github.com/ThomasHabets/rustradio. I registered the github org rustyradio, too. rustradio was taken. I sent a message to the owner, since it seems to not have any real content, but have not heard back.

Unsafe code

To make this multiuser stream I did have to write some Continue reading

SDR transmit and clean signals

If you have a transmit capable SDR, you may have heard that you need to filter its output, before transmitting to the world. Certainly before amplifying the signal.

I have a TinySA Ultra spectrum analyzer, and will here show you some screenshots about just how true that is.

I tested this with my USRP B200, transmitting a pure carrier around 145MHz and 435MHz.

Oh, and a word of caution: If you want to replicate this, make sure to add an inline attenuator, to not damage your spectrum analyzer. I had a cheap 40dB one, but the values in the graphs have been adjusted to show the real signal strength, as if I hadn’t.

tl;dr

Harmonics can be almost as strong as the fundamental. You need to filter these.
Transmitting at maximum output gain may cause lots of unwanted signals right around your fundamental. You cannot filter these. You need to not generate them.

Harmonics

Reducing the output gain did not meaningfully fix the problem. The best I saw from using half output gain was to make the strongest harmonic 9dB less than the fundamental. That’s way too strong.

I added a cheap band pass filter (FBP-144), which made Continue reading

AX.25 and 9600bps G3RUH decoding

I’ve been coding more on my rust SDR framework, and want to improve my ability to send/receive data packets efficiently and reliably.

There are two main ways I use learn to do this better: designing a new protocol, and making the best implementation possible for an existing one. This post is about refining the latter.

AX.25 and APRS

First a detour, or background.

AX.25 is the standard amateur radio data protocol. It’s mostly an OSI layer 2-4 protocol, mashing the layers together into one. Contrast this with IP, which just encapsulates the next layer.

Layer 3 (IP stack equivalent: IP itself) consists of the ability to add, in addition to source and destination, a variable number of intermediate repeaters. This allows limited source routing. In APRS the repeaters are usually not named, but instead uses “virtual” hops like WIDE1-1.

Layer 4 (IP stack equivalent: TCP and UDP) allows both connected and disconnected communication channels. In my experience connected AX.25 works better over slow simplex radio than TCP. If TCP was ever optimized for high delay low bandwidth, it’s not anymore.

For the physical layer, there are three main “modems”:

300 baud bell 103, used Continue reading

Setting up secure wifi

If you don’t set a password on your wifi, then not only can anyone connect, but it’s not even encrypted. This means that even when an open network gives you a captive portal, that could actually be an attacker giving you a fake portal. Even if the portal is HTTPS, because you may be connected to https://evil-fake-portal.com.

That is solved in WPA3, where even open networks become encrypted.

Of course, the attacker can just set up a fake access point, and you’ll connect, none the wiser. Even if the network has a password, the attacker only needs to know that password in order to fake it.

Before WPA3, passwords can easily be brute forced offline. A few years ago I calculated that it would cost about $70 to crack the default generated 8 character random passwords used by a popular ISP here in London, using some GPUs in Google Cloud. I’m sure it’s cheaper now.

That’s potentially years of free use of your neighbours wifi, for just the cost of a couple of months of paying for your own.

But that’s illegal, of course. This post is about protecting you against these attacks, not performing them.

If you Continue reading

RustRadio, and Roast My Rust

I’m learning Rust. And I like playing with software defined radio (SDR). So the natural project to take on to learn Rust is to write a crate for making SDR applications. I call it RustRadio.

I have something that works, and seems pretty OK. But before marking a 1.0.0 release I want to see if I can get some opinions on my use of the Rust language. Both in terms of design, and more clippy-like suggestions.

Hence: Roast My Rust. File a github issue, email me, or tweet at me. Tell me I’m doing it wrong.

RustRadio code: https://github.com/ThomasHabets/rustradio
RustRadio docs: https://docs.rs/rustradio/latest/rustradio/
The first application: https://github.com/ThomasHabets/sparslog

What my priorities are

There are two API surfaces in RustRadio; the Block API (for writing blocks), and the Application API (for writing applications that use blocks). I want them to be good, and future proof, so that I don’t have to change every block and every application, after adding a feature or improving the API.

The blocks will need to be thread safe, even though the scheduler is currently single threaded.

For the streams between blocks I’ll eventually want to make a more fancy, but unsafe circular Continue reading

Downloading web resources

Last time I went to the dentist, they offered to use a fancy scanner to better be able to show me my teeth.

Who can say no to that? I already for fun got a 3D scan of my brain, so why not teeth too?

I requested the data, and got a link to a web UI. Unfortunately it was just a user friendly 3D viewer, without any download button.

Here’s how I extracted the 3D data:

Open Chrome developer console, e.g. by pressing Ctrl-Shift-C (I hate it that Chrome hijacked this. Every single day I press Ctrl-Shift-C to copy, and it throws up this thing)
Close the stupid “what’s new” spam, that nobody in the history of ever has wanted to see.
Go to the ‘Network’ tab.
Reload the page.
Right click on any item in the list, and choose “Save all as HAR with content”. No, I don’t know why I can’t just save that one resource.

A HAR file is a JSON file archive, essentially.

$ jq '.log | keys' foo.har
[
  "creator",
  "entries",
  "pages",
  "version"
]
$ jq '.log | .entries[0].request | keys' foo.har
[
  "bodySize",
  "cookies",
  "headers",
  "headersSize",
   Continue reading

The unreasonable effectiveness of radio

Light and radio transmissions are the same thing, just using different frequencies.

How is it reasonable that I can transmit with a radio using the energy of a low energy light bulb (10 watts), and easily chat with someone 1800km away?

Even if you imagine a perfectly dark world, where the only light bulb is a 10W bulb in Sweden; How is it even possible that this could be seen in Italy? It’s not even a spotlight! It’s only vaguely aimed away from east-west, in favour of up, north, and south.

My little light bulb (radio) could be seen in (approximately) all of Europe. ~750 million people potentially could have received it at the same time. With 10 watts.

“Blink blink” — All of Europe can see my little lightbulb.

And this isn’t some specialized super duper antenna, nor was it set up by an expert, fine tuning everything. I just put up the antenna in a PVC pipe and connected it. We can’t even credit fancy computers digging signals out of the noise. This was not FT8, this was PSK31. I’m sure voice would also have worked.

I also used FT8 with these same 10W to check off Continue reading

1 2 3 … 5 Next »

SIMD background

What is connection coalescing?

Why did they do that?

Why does it break everything?

Rust

Round trip

SIMD (with Rust)

Set up build environment

Create test project

Configure linker

Build

Change the “interpreter” to what the Ubiquiti system expects

Does it work?

Links

Set up build environment

Create test project

Build most of it

Temporarily change /usr/bin/cc to point to the mips gcc

Link the program

Restore /usr/bin/cc

Change the “interpreter” to what the Ubiquiti system expects

Building it again

Does it work?

Links

KISS

Linux kernel implementation

Notes

Unsafe code

tl;dr

Harmonics

AX.25 and APRS

What my priorities are

Temporarily change `/usr/bin/cc` to point to the mips gcc

Restore `/usr/bin/cc`