OpenBSD in Stereo with Linux VFIO

I use a Huawei Matebook X as my primary OpenBSD laptop and one aspect of its hardware support has always been lacking: audio never played out of the right-side speaker. The speaker did actually work, but only in Windows and only after the Realtek Dolby Atmos audio driver from Huawei was installed. Under OpenBSD and Linux, and even Windows with the default Intel sound driver, audio only ever played out of the left speaker.

Now, after some extensive reverse engineering and debugging with the help of VFIO on Linux, I finally have audio playing out of both speakers on OpenBSD.

VFIO

The Linux kernel has functionality called VFIO which enables direct access to a physical device (like a PCI card) from userspace, usually passing it to an emulator like QEMU.

To my surprise, it seems to be primarily used these days by gamers who boot Linux, then use QEMU to run a game in Windows and use VFIO to pass the computer's GPU device through to Windows.

By using Linux and VFIO, I was able to boot Windows 10 inside of QEMU and pass my laptop's PCI audio device through to Windows, allowing the Realtek audio drivers to natively control the audio device. Combined with QEMU's tracing functionality, I was able to get a log of all PCI I/O between Windows and the PCI audio device.

Using VFIO

To use VFIO to pass-through a PCI device, it first needs to be stubbed out so the Linux kernel's default drivers don't attach to it. GRUB can be configured to instruct the kernel to ignore the PCI audio device (8086:9d71) and explicitly enable the Intel IOMMU driver by adding the following to /etc/default/grub and running update-grub:

GRUB_CMDLINE_LINUX_DEFAULT="text pci-stub.ids=8086:9d71 iommu=pt intel_iommu=on"

With the audio device stubbed out, a new VFIO device can be created from it:

$ sudo modprobe pci-stub
$ sudo modprobe vfio-pci

$ echo 0000:00:1f.3 | sudo tee /sys/bus/pci/devices/0000:00:1f.3/driver/unbind
$ echo 0x8086 0x9d71 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

Then the VFIO device (00:1f.3) can be passed to QEMU:

$ sudo ../qemu/x86_64-softmmu/qemu-system-x86_64 \
	-M q35 -m 2G -cpu host,kvm=off \
	-enable-kvm \
	-device vfio-pci,host=00:1f.3,multifunction=on,x-no-mmap \
	-hda win10-tmp.img \
	-trace events=events.txt 2>&1 | tee debug-output

I was using my own build of QEMU for this, due to some custom logging I needed (more on that later), but the default QEMU package should work fine. The events.txt was a file of all VFIO events I wanted logged (which was all of them).

Since I was frequently killing QEMU and restarting it, Windows 10 wanted to go through its unexpected shutdown routine each time (and would sometimes just fail to boot again). To avoid this and to get a consistent set of logs each time, I used qemu-img to take a snapshot of a base image first, then boot QEMU with that snapshot. The snapshot just gets thrown away the next time qemu-img is run and Windows always starts from a consistent state.

$ qemu-img create -f qcow2 -b win10.img win10-tmp.img

QEMU will now log each VFIO event which gets saved to a debug-output file.

[...]
9645@1541992466.382461:vfio_pci_read_config  (0000:00:1f.3, @0x2e, len=0x2) 0x3200
9645@1541992466.395726:vfio_region_read  (0000:00:1f.3:region0+0xc, 2) = 0x0
9645@1541992466.395792:vfio_region_read  (0000:00:1f.3:region0+0xe, 2) = 0x1
9645@1541992466.396021:vfio_region_write  (0000:00:1f.3:region0+0xc, 0x0, 2)
[...]

With a full log of all PCI I/O activity from Windows, I compared it to the output from OpenBSD and tried to find the magic register writes that enabled the second speaker. After days of combing through the logs and annotating them by looking up hex values in the documentation, diffing runtime register values, and even brute-forcing it by mechanically duplicating all PCI I/O activity in the OpenBSD driver, nothing would activate the right speaker.

One strange thing that I noticed was if I booted Windows 10 in QEMU and it activated the speaker, then booted OpenBSD in QEMU without resetting the PCI device's power in-between (as a normal system reboot would do), both speakers worked in OpenBSD and the configuration that the HDA controller presented was different, even without any changes in OpenBSD.

A Primer on Intel HDA

Most modern computers with integrated sound chips use an Intel High Definition Audio (HDA) Controller device, with one or more codecs (like the Realtek ALC269) hanging off of it. These codecs do the actual audio processing and communicate with DACs and ADCs to send digital audio to the connected speakers, or read analog audio from a microphone and convert it to a digital input stream. In my Huawei Matebook X, this is done through a Realtek ALC298 codec.

On OpenBSD, these HDA controllers are supported by the azalia(4) driver, with all of the per-codec details in the lengthy azalia_codec.c file. This file has grown quite large with lots of codec- and machine-specific quirks to route things properly, toggle various GPIO pins, and unmute speakers that are for some reason muted by default.

azalia0 at pci0 dev 31 function 3 "Intel 200 Series HD Audio" rev 0x21: msi
azalia0: host: High Definition Audio rev. 1.0
azalia0: host: 9 output, 7 input, and 0 bidi streams
azalia0: found a codec at #0
azalia0: found a codec at #2
azalia_init_corb: CORB allocation succeeded.
azalia_init_corb: CORBWP=0; size=256
azalia_init_rirb: RIRB allocation succeeded.
azalia_init_rirb: RIRBRP=0, size=256
azalia0: codec[0] vid 0x10ec0298, subid 0x320019e5, rev. 1.3, HDA version 1.0
azalia_codec_init: There are 36 widgets in the audio function.
[...]
azalia0: codecs: Realtek ALC298, Intel/0x280b, using Realtek ALC298

The azalia driver talks to the HDA controller and sets up various buffers and then walks the list of codecs. Each codec supports a number of widget nodes which can be interconnected in various ways. Some of these nodes can be reconfigured on the fly to do things like turning a microphone port into a headphone port.

The newer Huawei Matebook X Pro released a few months ago is also plagued with this speaker problem, although it has four speakers and only two work by default. A fix is being proposed for the Linux kernel which just reconfigures those widget pins in the Intel HDA driver. Unfortunately no pin reconfiguration is enough to fix my Matebook X with its two speakers.

While reading more documentation on the HDA, I realized there was a lot more activity going on than I was able to see through the PCI tracing.

For speed and efficiency, HDA controllers use a DMA engine to transfer audio streams as well as the commands from the OS driver to the codecs. In the output above, the CORBWP=0; size=256 and RIRBRP=0, size=256 indicate the setup of the CORB (Command Output Ring Buffer) and RIRB (Response Input Ring Buffer) each with 256 entries. The HDA driver allocates a DMA address and then writes it to the two CORBLBASE and CORBUBASE registers, and again for the RIRB.

When the driver wants to send a command to a codec, such as CORB_GET_PARAMETER with a parameter of COP_VOLUME_KNOB_CAPABILITIES, it encodes the codec address, the node index, the command verb, and the parameter, and then writes that value to the CORB ring at the address it set up with the controller at initialization time (CORBLBASE/CORBUBASE) plus the offset of the ring index. Once the command is on the ring, it does a PCI write to the CORBWP register, advancing it by one. This lets the controller know a new command is queued, which it then acts on and writes the response value on the RIRB ring at the same position as the command (but at the RIRB's DMA address). It then generates an interrupt, telling the driver to read the new RIRBWP value and process the new results.

Side note: if you've ever had a kernel panic or blue screen while playing music and had your audio device repeat the same second of audio over and over until you power it off, this is why. The audio driver put the last samples of audio onto the ring buffer of the HDA device and the hardware played them as it walked the ring buffer, but because the kernel panicked and can't put new samples onto the ring nor tell the hardware to stop consuming it, the HDA device just keeps walking around the ring playing that last sample of audio.

Since the actual command contents and responses are handled through DMA writes and reads, these important values weren't showing up in the VFIO PCI trace output that I had gathered. Time to hack QEMU.

Logging DMA Memory Values in QEMU

Since DMA activity wouldn't show up through QEMU's VFIO tracing and I obviously couldn't get Windows to dump these values like I could in OpenBSD, I could make QEMU recognize the PCI write to the CORBWP register as an indication that a command has just been written to the CORB ring.

My custom hack in QEMU adds some HDA awareness to remember the CORB and RIRB DMA addresses as they get programmed in the controller. Then any time a PCI write to the CORBWP register is done, QEMU fetches the new CORB command from DMA memory, decodes it into the codec address, node address, command, and parameter, and prints it out. When a PCI read of the RIRBWP register is requested, QEMU reads the response and prints the corresponding CORB command that it stored earlier.

With this hack in place, I now had a full log of all CORB commands and RIRB responses sent to and read from the codec:

9645@1541992466.588081:vfio_region_read  (0000:00:1f.3:region0+0x48, 2) = 0xdb
CORBWP advance to 220, last WP 219
CORB[220] = 0x21f0800 (caddr:0x0 nid:0x21 control:0xf08 param:0x0)
9645@1541992466.588109:vfio_region_write  (0000:00:1f.3:region0+0x48, 0xdc, 2)
[...]
9645@1541992466.588386:vfio_region_write  (0000:00:1f.3:region0+0x5d, 0x1, 1)
RIRBWP advance to 220, last WP 219
CORB caddr:0x0 nid:0x21 control:0xf08 param:0x0 response:0x82 (ex 0x0)
9645@1541992466.588431:vfio_region_read  (0000:00:1f.3:region0+0x58, 2) = 0xdc
[...]

An early version of this patch left me stumped for a few days because, even after submitting all of the same CORB commands in OpenBSD, the second speaker still didn't work. It wasn't until re-reading the HDA spec that I realized the Windows driver was submitting more than one command at a time, writing multiple CORB entries and writing a CORBWP value that was advanced by two. This required turning my CORB/RIRB reading into a for loop, reading each new command and response between the new CORBWP/RIRBWP value and the one previously seen.

Sure enough, the magic commands to enable the second speaker were sent in these periods where it submitted more than one command at a time.

Minimizing the Magic

The full log of VFIO PCI activity from the Windows driver was over 65,000 lines and contained 3,150 CORB commands, which was a lot to sort through. It took me a couple more days to reduce that down to a small subset that was actually required to activate the second speaker, and that could only be done through trial and error:

• Boot OpenBSD with the full list of CORB commands in the azalia driver
• Comment out a group of them
• Compile kernel and install it, halt the QEMU guest
• Suspend and wake the laptop, resetting PCI power to the audio device to reset the speaker/Dolby initialization and ensure the previous run isn't influencing the current test (I'm guessing there is an easier to way to reset PCI power than suspending the laptop, but oh well)
• Start QEMU, boot OpenBSD with the new kernel
• Play an MP3 with mpg123 which has alternating left- and right-channel audio and listen for both channels to play

This required a dozen or so iterations because sometimes I'd comment out too many commands and the right speaker would stop working. Other times the combination of commands would hang the controller and it wouldn't process any further commands. At one point the combination of commands actually flipped the channels around so the right channel audio was playing through the left speaker.

The Result

After about a week of this routine, I ended up with a list of 662 CORB commands that are needed to get the second speaker working.

The stereo sound from OpenBSD is wonderful now and I can finally stop downmixing everything to mono to play from the left speaker. In case one ever needs to do this, sndiod can be run with -c 0:0 to reduce the channels to one.

Update (2019-03-24): This list of CORB commands was optimized by Thomas Espeleta for inclusion in the Linux kernel and then Stefan Sperling (stsp@) implemented that logic back in the OpenBSD azalia driver. Coming full circle, I committed Stefan's implementation and OpenBSD now fully supports the audio on the Huawei Matebook X.

Thanks to rjc for proofreading and feedback.