Tracing function calls

Sometimes you want to see functions of a library, as they’re called. I know of two ways of doing this.

Let’s have a super simple test program:

#include<iostream>
#include<unistd.h>

void func1() {}
void func2() {}

int main()
{
  std::cout << "Hello world\n";
  func1();
  func2();
  func1();

  // Wait a bit for bpftrace to be able to aquire the function name.
  // Not applicable for something that doesn't exist.
  sleep(1);
}

bpftrace

Start a bpftrace in one terminal, and run the program in another.

$ sudo bpftrace -e 'uprobe:./a.out:func* { print(func); }'
Attaching 2 probes...
func1()
func2()
func1()

GDB

$ gdb a.out
[…]
(gdb) rbreak func.*
[…]
(gdb) commands
Type commands for breakpoint(s) 1-3, one per line.
End with a line saying just "end".
>silent
>bt 1
>cont
>end
(gdb) r
Starting program: […]/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
#0  0x0000555555555215 in _GLOBAL__sub_I__Z5func1v ()
Hello world
#0  0x000055555555516d in func1() ()
#0  0x0000555555555174 in func2() ()
#0  0x000055555555516d in func1() ()
[Inferior 1 (process 424744) exited normally]
(gdb)

Which to use?

bpftrace is lower (but Continue reading

RISC-V optimization and -mtune

I’ve been getting into RISC-V optimization recently. Partly because I got my SiFive VisionFive 2, and partly because unlike x86 the number of RISC-V instructions is so managable that I may actually have a chance at beating the compiler.

I’m optimizing the inner loops of GNURadio, or in other words the volk library. I’ve been getting up to a about a doubling of the speed compared to the compiled C code, depending on the function.

But it got me thinking how far I could tweak the compiler and its options, too.

Yes, I should have done this much sooner.

Many years ago now I built some data processing thing in C++, and thought it ran too slowly. Sure, I did a debug build, but how much slower could that be? Half speed? Nope. 20x slower.

Of course this time I never compared to a debug build, so don’t expect that kind of difference. Don’t expect that it’ll reach my hand optimized assembly either, imperfect as it may be.

The test code

This may look like a synthetic benchmark, in simplified C++:

complex volk_32fc_x2_dot_prod_32fc_generic(const vector<complex> &in1,
                                           const vector<complex> &in2)
{
  complex res;
  for (unsigned int i = 0; i  Continue reading

VisionFive 2 quickstart

RISC-V small computer

For a long time I’ve wanted something Raspberry-pi-like but with RISC-V. And finally there is one, and a defensible price! Especially with the Raspberry Pi 4 shortage this seemed like a good idea.

This post is my first impressions and setup steps.

It’s just like when I was a kid!

When I was in my late teens I was playing with different architectures, mostly using discarded university computers. It was fun to have such different types of computers. Back then it was SPARC (And UltraSparc), Alpha, and x86. Maybe access to some HPPA. I even had a MIPS (SGI Indigo 2).

Nowadays instead of SPARC, Alpha, and x86 it’s ARM, RISC-V, and x64.

Luckily they can be smaller nowadays. Before I left home my room had more towers of computers than it had furniture. In my first flat I had a full size rack!

Write SD card

pv starfive-jh7110-VF2_515_v2.5.0-69-minimal-desktop.img \
   | sudo dd of=/dev/sda

Repartition SD card

We need to repartition, because the boot partition is way too small. It only fits one kernel/initrd, which became a problem I ran into.

Unfortunately gparted doesn’t seem to work on disk images. It Continue reading

Learning Rust, assisted by ChatGPT

I finally got around to learn Rust. Well, starting to.

It’s amazing.

I’m comparing this to learning Basic, C, C++, Erlang, Fortran, Go, Javascript, Pascal, Perl, PHP, Prolog and Python. I wouldn’t say I know all these languages well, but I do know C, C++, Go, and Python pretty well.

I can critique all these languages, but I’ve not found anything frustrating or stupid in Rust yet.

Rust is like taking C++, but all the tricky parts are now the default, and checked at compile time.

Copies

With C++11 we got move semantics, so we have to carefully disable the (usually) default-created copy constructor and copy assignments, or if you have to allow copies then every single use of the type has to be very careful to not trigger copies unless absolutely necessary.

And you have to consider exactly when RVO kicks in. And even the best of us will sometimes get it wrong, especially with refactors. E.g. who would have thought that adding this optimization would suddenly trigger a copy of the potentially very heavy Obj object, a copy that was not there before?

--- before.cc   2022-12-28 10:32:50.969273274 +0000
+++ after.cc    2022-12-28 10:32:50.969273274 +0000
 Continue reading

Better pipes

In my blog post The uselessness of bash I made a tool to improve pipes in shell, to assemble a better pipeline.

It solves the problem, but it’s a bit too different, with its own language.

While complaining with some people at work that one of the main features of shell (the pipe operator) is broken, someone joked that it should be replaced by a protobuf based protocol.

But on second thought it’s not really a joke.

How about instead of this:

$ goodpipe <<EOF
[
  ["gsutil", "cat", "gs://example/input-unsorted.txt"],
  ["sort", "-S300M", "-n"],
  ["gzip", "-9"],
  ["gsutil", "cp", "-", "gs://example/input-sorted-numerically.txt.gz"]
]
EOF

how about this:

$ wp -o gsutil cat gs://example/input-unsorted.txt \
  | wp -io sort -S300M -n \
  | wp -io gzip -9 \
  | wp -i gsutil cp - gs://example/input-sorted-numerically.txt.gz

It doesn’t use protobufs, but a simpler regular protocol. This in order to avoid well known bugs types. Before implementing any protocol also see formal theory and science of insecurity.

First I hacked it together in Go, but I think the main implementation I’ll maintain is the one I made while porting it to Rust, as a way to learn Rust. The Continue reading

Can ChatGPT debug my code? Can it write better code?

Everyone’s talking about it, so why should I not add something too.

Interview question

ChatGPT is pretty amazing. I asked it an interview question, and it just plain solved it. Well, there was one real world imperfection. I asked it to fix that problem, and then it went off the rails. The more I hinted, the worse the code got.

I won’t give the details here, because I want to keep using the question / topic.

Finding bugs

So I wondered if it could find bugs. First an easy one:

Q:

What's the problem with this C++ code:
int* foo() {
  auto ret = new int;
  int r2 = *ret * 2;
  return &r2;
}

A:
There are several problems with the code you provided:

1. The function foo is returning a pointer to a local variable r2,
   which is not safe because the memory for r2 is deallocated when the
   function returns. This can cause undefined behavior when you try to
   access the returned pointer after the function has returned.
2. The function foo is allocating memory for an integer with new int,
   but it is not deallocating that memory before the function
   returns. This can lead to a  Continue reading

Terminator 1 is the best Terminator movie

And now for something completely different.

I’ve off and on thought about this for years, so it needed to be written down.

Terminator 1 is the best Terminator movie

Obviously SPOILERS, for basically all Terminator movies.

Summary of reasons

• The robot is really not human.
• It’s a proper time loop, with a bonus that none of the players in the movie know it.

I’m aware of The Terminator Wiki, but I don’t care about it. My opinions are on the movies as movies.

The behavior of the terminator

In Terminator 1 (T1) Arnold is clearly a robot in human skin. At no point do you believe it’s a human. The only reason people don’t stop and scream and point, is that “I’m being silly, that’s clearly impossible”. But Arnold spends the whole movie in the uncanny valley, the kind in 2022 reserved for realistically generated CGI characters.

It’s very nearly a perfect movie. Just take his first dialog. “Nice night for a walk”, the punks say. They are saying this to a machine that has never talked to a human before, so its response is complete nonsense. It just repeats the words back to them.

It’s a Continue reading

Fast zero copy static web server with KTLS

I’m writing a webserver optimized for serving a static site with as high performance as possible. Counting every syscall, and every copy between userspace and kernel space.

It’s called “tarweb”, because it serves a website entirely from a tar file.

I’m optimizing for latency of requests, throughput of the server, and scalability over number of active connections.

I won’t go so far as to implement a user space network driver to bypass the kernel, because I want to be able to just run it in normal setups, even as non-root.

I’m not even close to done, and the code is a mess, but here are some thoughts for now.

First optimize syscall count

Every syscall costs performance, so we want to minimize those.

The minimum set of syscalls for a webserver handling a request is:

1. accept() to acquire the new connection.
2. epoll_ctl() to add the fd.
3. epoll_wait() & read() or similar. (ideally getting the whole request in one read() call)
4. epoll_wait() & write() or similar. (again ideally in one call)
5. close() the connection.

There’s not much to do about accept() and read(), as far as I can see. You need to accept the connection, and you need to Continue reading

Integer handling is broken

Floating point can be tricky. You can’t really check for equality, and with IEEE 754 you have a bunch of fun things like values of not a number, infinities, and positive and negative zero.

But integers are simple, right? Nope.

I’ll use “integers” to refer to all integer types. E.g. C’s int, unsigned int, gid_t, size_t, ssize_t, unsigned long long, and Java’s int, Integer, etc…

Let’s list some problems:

• You can’t parse unsigned integers in C/C++ with standard functions.
• Casting between types silently discards information (C, Java, Go, Rust, Haskell).
• Casting between some other types changes the semantic meaning of the number, even when technically no information is lost.
• The state of integers in Javascript is its own mess.

What’s wrong with casting?

Casting an integer from one type to another changes three things:

1. The type in the language’s type system.
2. Crops values that don’t fit.
3. May change the semantic value, by changing sign.

The first is obvious, and is even safe for the language to do implicitly. Why even bother telling the human that a conversion was done?

But think about the other two for a minute. Is there any reason that you want your Continue reading

No way to parse integers in C

There are a few ways to attempt to parse a string into a number in the C standard library. They are ALL broken.

Leaving aside the wide character versions, and staying with long (skipping int, long long or intmax_t, these variants all having the same problem) there are three ways I can think of:

1. atol()
2. strtol() / strtoul()
3. sscanf()

They are all broken.

What is the correct behavior, anyway?

I’ll start by claiming a common sense “I know it when I see it”. The number that I see in the string with my eyeballs must be the numerical value stored in the appropriate data type. “123” must be turned into the number 123.

Another criteria is that the WHOLE number must be parsed. It is not OK to stop at the first sign of trouble, and return whatever maybe is right. “123timmy” is not a number, nor is the empty string.

Failing to provide the above must be an error. Or at least as the user of the parser I must have the option to know if it happened.

First up: atol()

No. All Continue reading

Java — A fractal of bad experiments

The title of this post is clearly a reference to the classic article PHP a fractal of bad design. I’m not saying Java is as bad as that, but that it has its own problems.

Do note that this post is mostly opinion.

And I’m not saying any language is perfect, so I’m not inviting “but what about C++’s so-and-so?”.

What I mean by “bad experiments” is that I don’t think the decisions the creators of Java were bad with the information they had at the time, but that with the benefit of hindsight they have proven to be ideas and experiments that turned out to be bad.

Ok, one more disclaimer: In some parts here I’m not being precise. I feel like I have to say that I know that, to try to reduce the anger from Java fans being upset about me critiqueing their language.

Don’t identify with a language. You are not your tool.

Too much OOP

A lot of Java’s problems come from the fact that it’s too object oriented. It behaves as if everything is axiomatically an object.

No free-standing functions allowed. So code is full of public static functions, in classes with no Continue reading

Another way MPLS breaks traceroute

I recently got fiber to my house. Yay! So after getting hooked up I started measuring that everything looked sane and performant.

I encountered two issues. Normal people would not notice or be bothered by either of them. But I’m not normal people.

I’m still working on one of the issues (and may not be able to disclose the details anyway, as the root cause may be confidential), so today’s issue is traceroute.

In summary: A bad MPLS config can break traceroute outside of the MPLS network.

What’s wrong with this picture?

$ traceroute -q 1 seattle.gov
traceroute to seattle.gov (156.74.251.21), 30 hops max, 60 byte packets
 1  192.168.x.x (192.168.x.x)  0.302 ms     <-- my router
 2  194.6.x.x.g.network (194.6.x.x)  3.347 ms
 3  10.102.3.45 (10.102.3.45)  3.391 ms
 4  10.102.2.29 (10.102.2.29)  2.841 ms
 5  10.102.2.25 (10.102.2.25)  2.321 ms
 6  10.102.1.0 (10.102.1.0)  3.454 ms
 7  10.200.200.4 (10.200.200.4)  2. Continue reading

Dropping privileges

If you’re writing a tool that takes untrusted input, and you should treat almost all input as untrusted, then it’s a good idea to add a layer of defense against bugs in your code.

What good is a buffer overflow, if the process is fully sandboxed?

This applies to both processes running as root, and as normal users. Though there are some differences.

Standard POSIX

In POSIX you can only sandbox if you are root. The filesystem can be hidden with chroot(), and you can then change user to be non-root using setuid() and setgid().

There have been ways to break out of a chroot() jail, but if you make sure to drop root privileges then chroot() is pretty effective at preventing opening new files and running any new programs.

But which directory? Ideally you want it to be:

• read-only by the process (after dropping root)
• empty
• not shared by any other process that might write to it

The best way no ensure this is probably to create a temporary directory yourself, owned by root.

This is pretty tricky to do, though:

// Return 0 on success.
int do_chroot()
{
  const char* tmpdir = getenv("TMPDIR");
  if (tmpdir == NULL)  Continue reading

seccomp — Unsafe at any speed

I’ll just assert that there’s no way to use seccomp() correctly. Just like how there’s no way to use gets() correctly, causing it to eventually be removed from the C and C++ standards.

seccomp, briefly

seccomp allows you to filter syscalls with a ruleset.

The obvious thing is to filter anything your program isn’t supposed to be doing. If it doesn’t do file IO, don’t let it open files. If it’s not supposed to execute anything, don’t let it do that.

But whether you use a whitelist (e.g. only allow working with already open file descriptors), or a blacklist (e.g. don’t allow it to open these files), it’s fundamentally flawed.

1. Syscalls change. Sometimes without even recompiling

open() in your code actually becomes the openat syscall. Maybe. At least today. At least on my machine, today.

select() actually becomes pselect6. At least on Fridays.

If you upgrade libc or distribute a binary to other systems, this may start to fail.

2. Surprising syscalls

Calling printf() will call the syscall newfstatat, a syscall hard to even parse into words. But only the first time you call it! So after your first printf() you can block newfstatat.

Maybe Continue reading

AX.25 over D-Star

Setting up AX.25 over 1200bps was easy enough. For 9600 I got kernel panics on the raspberry pi, so I wrote my own AX.25 stack.

But I also want to try to run AX.25 over D-Star. Why? Because then I can use radios not capable of 9600 AX.25, and because it’s fun.

It seems that radios (at least the two I’ve been working with) expose the D-Star data channel as a byte stream coming over a serial connection. Unlike working with a TNC you don’t have to talk KISS to turn the byte stream into packets, and vice versa.

IC9700 setup

The first hurdle to overcome, because we want to send binary data, is to escape the XON/XOFF flow control characters that the IC9700 mandates. Otherwise we won’t be able to send 0x13 or 0x11. Other bytes seem to go through just fine.

So I wrote a wrapper for that, taking /dev/ttyUSB1 on one side, and turning it into (e.g.) /dev/pts/20 for use with kissattach.

$ ./dsax /dev/ttyUSB1
/dev/pts/20

$ kissattach /dev/pts/20 radio
$ kissattach -p radio -c 2     # See below

Set Menu>Set>DV/DD Set>DV Data TX to Auto, for “automatic PTT”. As Continue reading

Localisation isn’t translation

If you only have your app in English then you’ll still be understood[1] by the new market whose official language isn’t English.

If you show farenheit (a word I can’t even spell), then 96% of the world cannot understand your app. At all.

For most of the west I would argue that translation doesn’t even matter at all, but you cannot have your app start your weeks on Sunday, you cannot show fahrenheit, or feet, or furlongs, or cubits or whatever US-only units exist. And you cannot use MM/DD/YY.

NONE of these things are tied to language. Most users of English don’t want any of this US-only failure to communicate.

[1] While most of the world doesn’t speak English fluently, they may know words. And they can look up words. You cannot “look up” understanding fahrenheit or US-only date formats.

Go programs are not portable

A while ago I was asked why I wrote Sim in C++ instead of Go. I stumbled upon my answer again and realized it could be a blog post.

So here’s what I wrote then. I think I stand by it still, and I don’t think the situation has improved.

Why not write portable system tools in Go

My previous experience with “low level” things in Go (being very careful about which syscalls are used, and in which order) has had some frustrations in Go. Especially with portability. E.g. different definitions of syscall.Select between BSDs and Linux, making me have to use reflection at some points. (e.g. see this Go bug.

And to work around those things Go unfortunately uses the antipattern of (essentially) #ifdef __OpenBSD__, which we’ve known for decades is vastly inferior to checking for specific capabilities.

To me the Go proverb “Syscall must always be guarded with build tags” essentially means “Go is not an option for any program that needs to be portable and may at some point in the future require the syscalls package”. And since this tool is meant to be portable, and calls what would be syscall.Setresuid, Continue reading

SSH over bluetooth – cleanly

In my previous two posts I set up a login prompt on a bluetooth serial port and then switched to running SSH on it.

I explicitly did not set up an IP network over bluetooth as I want to minimize the number of configurations (e.g. IP address) and increase the chance of it working when needed.

E.g. firewall misconfiguration or Linux’s various “clever” network managers that tend to wipe out network interface configs would have more of a shared fate with the primary access method (SSH over normal network).

This post is about how to accomplish this more properly.

The problems now being solved are:

• It wasn’t entirely reliable. The rfcomm tool is pretty buggy.

• There was no authentication of the Bluetooth channel. Not as much a problem when doing SSH, but if there are passwords then there could be a man-in-the-middle attack.

• The server side had to remain discoverable forever. So anyone who scans for nearby bluetooth devices would see your servers, and would be able to connect, possibly brute forcing passwords. Not as much of a problem if running SSH with password authentication turned off, but why broadcast the name of a server if you don’t Continue reading

SSH over bluetooth

Yesterday I set up a simple serial console over bluetooth as a backup console.

Today I’m running SSH over bluetooth. Raw SSH, no IP. I only use IP on the two ends to talk to the SSH client and server. It doesn’t actually go over the bluetooth.

This fixes the security aspects with the previous solution. As long as you make sure to check the host key signature it’ll be perfectly secure.

No need for one-time passwords. You can even use SSH pubkey auth.

Connect to the system SSH

Server:

rfcomm watch hci0 2 socat TCP:127.0.0.1:22 file:/proc/self/fd/6,b115200,raw,echo=0

Client:

sudo rfcomm bind rfcomm2 AA:BB:CC:XX:YY:ZZ 2
ssh -oProxyCommand="socat - file:/dev/rfcomm2,b115200,raw,echo=0" dummy-hostname

A backup SSH

If you’re messing around with an OpenSSH config then it may be a good idea to set up a minimal config on another port. Maybe port 23. Not like that port is used for anything else anymore.

Raspberry Pi bluetooth console

Sometimes you want to connect to a bluetooth on the console. Likely because you screwed something up with the network or filewall settings.

You could plug in a screen and keyboard, but that’s a hassle. And maybe you didn’t prepare the Pi to force the monitor to be on even if it’s not connected at boot. Then it just doesn’t work.

Even more of a hassle is to plug in a serial console cable into the GPIO pins.

But modern Raspberry Pi’s have bluetooth. So let’s use that!

Setting up the service on the raspberry pi

Create /etc/systemd/system/bluetooth-console.service with this content:

[Unit]
Description=Bluetooth console
After=bluetooth.service
Requires=bluetooth.service

[Service]
ExecStart=/usr/bin/rfcomm watch hci0 1 getty rfcomm0 115200 vt100
Restart=always
RestartSec=10
StartLimitIntervalSec=0

[Install]
WantedBy=multi-user.target

This sets up a console on bluetooth channel 1 with a login prompt. But it doesn’t work yet. Apparently setting After, Required, and even Requisite doesn’t prevent systemd from running this before setting up bluetooth (timestamps in the logs don’t lie). Hence the restart stuff.

I also tried setting ExecStartPre / ExecStartPost there to enable Bluetooth discoverability, since something else in the boot process seems to turn it back off if I set it Continue reading