Adventures in Open Source

In the past couple weeks I contributed to a bunch of different open source projects in different ways and I thought I'd write about some of them.

Syncthing

I switched from Dropbox to Syncthing a while ago and so far it's been pretty great. I run it on my macOS server in the basement which mirrors everything on its large disks, and also on my various laptops where I selectively sync certain directories that I need.

Dropbox has a feature in its iOS app where it can upload photos taken to Dropbox, which would then sync to my Mac. I've mostly replicated this functionality with Syncthing by using the PhotoSync app which has SFTP support. It automatically uploads new photos to my macOS server via SFTP, which Syncthing picks up and can sync to my laptops.

Recently Syncthing gained kqueue support so it doesn't have to walk the directory structure every n minutes looking for changes. When I opened the Syncthing web interface to add synced photos to my OpenBSD laptop, it prompted me with a "you can use kqueue now, do you want to?" dialog so I enabled it.

Unfortunately a few minutes later, my laptop started getting hot and I noticed that the syncthing processes were using 100% CPU. After doing a ktrace I noticed it was caught in a tight loop trying to monitor a new directory, failing because it was out of file descriptors, and then retrying with no delay, causing the 100% CPU.

This class of problem was actually one that OpenBSD audited for not too long ago. System daemons that spun when running out of file descriptors (such as from an accept) were fixed not to do that as it could cause a remotely triggered DoS.

I opened an issue on the Syncthing issue tracker complaining about the problem, and a fix was committed 5 days later. These kinds of contributions are the easiest to do; just complain about a problem with the proper details, to the right people via the right channel, and they fix it for you.

Crystal

While listening to an episode of the Bike Shed podcast the other week, I was reminded to look into the Crystal programming language once again.

Crystal gained OpenBSD support long ago, but there is no port for it in the OpenBSD ports tree. I found a work-in-progress and tried to get it to compile, but on my 6.3-current laptop, it would crash while trying to compile its compiler.

After the OpenBSD 6.3 release, the OpenBSD kernel was changed to require that allocations from mmap used for thread stacks be done with a new MAP_STACK flag. This allows the kernel to do enforcement of the use of the allocation and kill any program that doesn't do things properly.

This required changes in OpenBSD's ports of Go, sbcl, and some others. I patched Crystal to pass the MAP_STACK flag on OpenBSD, and submitted a pull request to the author of the work-in-progress port. Since Crystal should continue working on OpenBSD out of the box, I submitted a pull request to the upstream Crystal project for formal MAP_STACK support, which was accepted for their 0.25.1 release. Now that the port would no longer require patches, I again updated the work-in-progress port for the new 0.25.1 release.

After running Crystal's std_spec test suite, I noticed some garbage printed when running on OpenBSD because it uses an alternative method to determine the number of CPUs. I submitted another pull request to clean this up, which was quickly merged.

These kinds of contributions are time consuming and technical, but as an OpenBSD developer I find it rewarding to get proper OpenBSD support in upstream projects.

bitwarden-ruby

At some point during this work, someone submitted a pull request to my bitwarden-ruby project to update its Sinatra dependency to fix a security issue in Sinatra. After merging it, another user reported breakage because the Sinatra update pulled in ActiveSupport which somehow broke the .try() functionality I was already using (which doesn't make sense because ActiveSupport includes a .try() implementation). Not wanting to fight things in the future or spend much time on it, I just removed my use of .try() .

Maintaining public projects and accepting contributions from others is not one of my strong points (which is partly why I gave up maintaining Lobsters). Users want to take the project in a different direction and add things that I don't want or care about, and I feel bad telling them "no" just because I don't like it, not really for any technical reason.

OpenBSD

Back to running Crystal's std_spec test suite, I had noticed one test failed on OpenBSD where it was expecting realpath(3) on a dangling symlink to return ENOENT.

After looking into the OpenBSD libc code and seeing how other OSes handled it, I realized it was actually an incompatibility in OpenBSD's libc. I posted about it on our tech@ mailing list and another developer helped come up with a fix.

One of our system tools, installboot(8) was actually relying on the incompatible behavior and would need to be changed before realpath(3) could be fixed in libc. Martijn and I worked together to develop a fix and I committed it a few days ago. Our ports team is doing a full package build of our entire ports tree with the proposed libc patch to determine any fallout. Since OpenBSD is behaving differently than most other OSes, hopefully there won't be anything else relying on our incompatible behavior.

OpenBSD's structure shines here; an issue in our libc was identified, we could easily see which of our own software relied on the behavior and fix it, and we could then build the thousands of 3rd party software packages that work on OpenBSD to identify any fallout before it's committed.

VNC

While testing the realpath behavior on other OSes, I needed to access my macOS server through VNC. OpenBSD's current VNC port is ssvnc which is a modified version of TightVNC. I noticed that when connecting to my Mac even over my fast network, the window would draw very slowly and mouse movements would take a few seconds to respond on the other end.

I had chalked it up to some weird incompatibility between TightVNC's viewer and macOS's native VNC server, but since I was on a streak of fixing things that bothered me this week, I spent some time digging into why it was so slow.

After connecting, vncviewer spit out a warning that it was unable to create a shared-memory image, but continued on its way. I traced this error in the ssvnc code and thought it may have been some problem in the MIT-SHM Xorg extension on OpenBSD. Some hours later debugging Xorg's XShm* functions and OpenBSD's kernel shm functionality, it all turned out to be for nothing. The ssvnc package was simply not being compiled with SHM support because its authors only enabled it on Linux (as Theo sarcastically said a decade ago, "all the world is an i386 running Linux").

Once I built ssvnc with -DMITSHM, the window would draw nearly instantly and my mouse movements reflected in near real-time.

I sent a patch to the ssvnc OpenBSD port maintainer to properly enable -DMITSHM on OpenBSD, who quickly ok'd it and I committed it.

Since we'd prefer to have as few patches as possible in our ports tree, I should really send this patch upstream but it seems like the kind of project where the patch will languish for years because nobody is maintaining it anymore.

cmus

While doing most of this work, I was using the console-based cmus audio player to play music (which is synced from my Mac via Syncthing, see how this is all connected?). One thing that has always bothered me about cmus was that it didn't support album shuffle. Not many music players do, but iPods and iTunes have always supported it and I've gotten used to it.

I don't like listening to songs on normal shuffle mode and prefer to listen to an album all the way through from its first track to the last, then skip to another random album by a different artist and continue playing from its first track.

Continuing my streak of digging in and fixing things, I spent an hour or so reading code and implementing album shuffle in cmus. I'm not sure whether it should be implemented the way I did it, so I'm waiting for the project maintainer to chime in before submitting a pull request for merging.

I like these kinds of contributions where I implement something for myself but it's also something a bunch of other people were waiting for someone to come along and do. I did this when adding configurable margins to iTerm2 and it seemed well received (hi /r/unixporn).