BSD Kernel Hacking - Part 2 (character pseudo-devices)

Continuing on from the last post, I now tackle character devices. It turns out that OpenBSD is quite a lot different than FreeBSD in the way that devices are registered. To begin, I need to set up a cdevsw entry for the new device. Rather than defining the struct directly in the implementation code, it is implemented via a macro which is called in the cdevsw struct definition in sys/arch/amd64/amd64/conf.c

As opposed to FreeBSD, the cdevsw structs are built out by a macro, this macro will take the number of devices being initialized, and the name of the module passed to it (e.g. "cdev_example_init(1, example)") and fill out struct cdevsw to point to the appropriate functions via name concatenation (e.g. the open function will be registered as exampleopen).

I begin by adding a cdev_example_init macro to sys/sys/conf.h, then call the macro within the cdevsw[] array in sys/arch/amd64/amd64/conf.c. Also, while I'm in here, I add a call to the cdev_decl() macro. This should take care of the cdevsw setup portion of the code.

From there, I add example.c under sys/dev/ and build out basic open, close, read, and write functions for it, and add it to files.conf. Much to my surprise, after fixing a few compile errors for missing headers, the kernel compiles and installs fine. Once the machine had booted into my new kernel, I ran a "doas mknod /dev/example c 98 0" to create a node for the device in /dev. In this case, 98 was the position in the cdevsw[] array where my call to cdev_exampl_init landed. In OpenBSD, the major node number is simply an index into the cdevsw array.

I had to switch from using copystr to uiomove in order to get I/O working properly, but once I did, the basic interface code worked fine.

Finally, I decided to work with the MAKEDEV script to ensure that my device would be present in a release, if I so chose to make one. The script required me to add a new case for "example)" with a call to the M function and a few parameters to set the device node name, Major/Minor numbers, permissions, and ownership. In order for etc/etc.amd64/MAKEDEV to be copied to /dev by sysmerge, I had to do a full build of the system. This is a bit time consuming, and there might be a better way, but it worked for my immediate purposes.

BSD Kernel Hacking - Part 1 (Intro/Syscalls)

Lately I've been reading through Designing BSD Rootkits by Joseph Kong. The book was released back in 2007, and is a treasure trove of information about BSD kernel hacking.

Unfortunately, much has changed since it was released, and the book was written with FreeBSD in mind, while I am more of an OpenBSD fan. I am hoping to start another multipart series based on exercises in this book. First off I will be going through the code in this book as I read, and updating it to run on FreeBSD 11.1. After that, I intend to work on hacking this code into the OpenBSD kernel, which is a little more difficult, as OpenBSD does not support Loadable Kernel Modules anymore.

The first chapter begins with a simple module that will print "Hello World!" on load, and "Goodbye, Cruel World!" on unload. This little experiment works just find out of the box on FreeBSD 11.1. Since OpenBSD doesn't have a LKM system, and this module does nothing without being loaded, porting it over isn't worthwhile.

One lesson of note from this experiment, is that uprintf() is used for printing to userspace, while regular ole' printf() will print to the system console (viewable with dmesg).

The second example in the book expands on the first to create a new syscall. This is where things start to go sideways -- my first attempts using code from the book fail to compile:

root@freebsd-dev:~/syscall # make
Warning: Object directory not changed from original /root/syscall
cc -O2 -pipe  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc   -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer  -MD  -MF.depend.sc_example.o -MTsc_example.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__  -Wmissing-include-dirs -fdiagnostics-show-option  -Wno-unknown-pragmas  -Wno-error-tautological-compare -Wno-error-empty-body  -Wno-error-parentheses-equality -Wno-error-unused-function  -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c sc_example.c -o sc_example.o
sc_example.c:48:1: error: use of undeclared identifier 'AUE_NULL'
SYSCALL_MODULE(sc_example, &offset, &sc_example_sysent, load, NULL);
^
/usr/src/sys/sys/sysent.h:204:43: note: expanded from macro 'SYSCALL_MODULE'
        evh, arg, offset, new_sysent, { 0, NULL, AUE_NULL }     \
                                                 ^
1 error generated.
*** Error code 1

Stop.
make: stopped in /root/syscall

This issue turned out to be caused by missing definitions in sys/sysproto.h. Once the missing header is included, the kernel module builds and loads fine, but the example interface code refuses to find the module. I modified the interface a bit to add error handling and check errno on the call to modfind() and found that it was returning an ENOENT. A bit of experimentation later, I discovered (via kldstat -v) that the actual name of sc_example was "sys/sc_example", not just "sc_example".

Next, I make an attempt to install a syscall into OpenBSD. This is a little more tricky due to a lack of LKMs. First, I create a file with a syscall function in it at sys/kern/sys_example.c. This syscall function is following the same prototype as all other OpenBSD syscalls, which is similar, but slightly different than FreeBSD. The syscall function simply outputs "proto wuz here" to the system console. 

From there, I found an unimplemented syscall number in sys/kern/syscalls.master (241), and updated the master list to include my sys_example syscall and called "make syscalls" in sys/kern, then added kern/sys_example.c to sys/conf/files to make sure it is included in the build.

This operation led to chasing compiler errors around missing includes for a while, but after all of the required headers were included, I was able to do a standard kernel build following the directions in release(8). After reboot, I was able to call syscall 241 from Perl and have the kernel display my message in the system console:

[peter@dev ~]$ perl -e "syscall(241);"
[peter@dev ~]$ dmesg | tail -1

proto wuz here

Here is the code from sys/kern/sys_example.c:


My final challenge for this post is to make the function behave the same as the example in the BSD Rootkit book, by accepting a string as an argument to the syscall and printing that to the console. I had to update my entry in syscalls.master to include the new argument, and include sys/syscallargs.h as well as a few other various headers. From there, the operation is almost identical to FreeBSD, except that a macro is used to access the arguments from their structure:



And the output from calling the syscall:

[peter@dev ~]$ perl -e '$str = "this is an OpenBSD syscall";' -e 'syscall(241, $str);'
[peter@dev ~]$ dmesg | tail -1

proto says: this is an OpenBSD syscall.

I obviously have some learning to do regarding code management in CVS, as my changes broke a bunch of the change/copyright information inside the files that were modified. I'm not too worried about it for the time being, as I don't intend to submit these changes upstream.

I did a little more code cleanup, and here is the final result:

If anyone finds anything that I've done wrong here, please feel free to yell at me about it in the comments.


References:

Designing BSD Rootkits: An Introduction to Kernel Hacking
by Joseph Kong
http://a.co/eSG5wWX

FAQ 5 - Building the System From Source
https://www.openbsd.org/faq/faq5.html

release(9)
https://man.openbsd.org/release

syscall(9)
https://man.openbsd.org/man9/syscall.9