BSD Kernel Hacking - Part 1 (Intro/Syscalls)

Lately I've been reading through Designing BSD Rootkits by Joseph Kong. The book was released back in 2007, and is a treasure trove of information about BSD kernel hacking.

Unfortunately, much has changed since it was released, and the book was written with FreeBSD in mind, while I am more of an OpenBSD fan. I am hoping to start another multipart series based on exercises in this book. First off I will be going through the code in this book as I read, and updating it to run on FreeBSD 11.1. After that, I intend to work on hacking this code into the OpenBSD kernel, which is a little more difficult, as OpenBSD does not support Loadable Kernel Modules anymore.

The first chapter begins with a simple module that will print "Hello World!" on load, and "Goodbye, Cruel World!" on unload. This little experiment works just find out of the box on FreeBSD 11.1. Since OpenBSD doesn't have a LKM system, and this module does nothing without being loaded, porting it over isn't worthwhile.

One lesson of note from this experiment, is that uprintf() is used for printing to userspace, while regular ole' printf() will print to the system console (viewable with dmesg).

The second example in the book expands on the first to create a new syscall. This is where things start to go sideways -- my first attempts using code from the book fail to compile:

root@freebsd-dev:~/syscall # make
Warning: Object directory not changed from original /root/syscall
cc -O2 -pipe  -fno-strict-aliasing -Werror -D_KERNEL -DKLD_MODULE -nostdinc   -I. -I/usr/src/sys -fno-common  -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer  -MD  -MF.depend.sc_example.o -MTsc_example.o -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float  -fno-asynchronous-unwind-tables -ffreestanding -fwrapv -fstack-protector -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes  -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual  -Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__  -Wmissing-include-dirs -fdiagnostics-show-option  -Wno-unknown-pragmas  -Wno-error-tautological-compare -Wno-error-empty-body  -Wno-error-parentheses-equality -Wno-error-unused-function  -Wno-error-pointer-sign -Wno-error-shift-negative-value -Wno-error-address-of-packed-member  -mno-aes -mno-avx  -std=iso9899:1999 -c sc_example.c -o sc_example.o
sc_example.c:48:1: error: use of undeclared identifier 'AUE_NULL'
SYSCALL_MODULE(sc_example, &offset, &sc_example_sysent, load, NULL);
^
/usr/src/sys/sys/sysent.h:204:43: note: expanded from macro 'SYSCALL_MODULE'
        evh, arg, offset, new_sysent, { 0, NULL, AUE_NULL }     \
                                                 ^
1 error generated.
*** Error code 1

Stop.
make: stopped in /root/syscall

This issue turned out to be caused by missing definitions in sys/sysproto.h. Once the missing header is included, the kernel module builds and loads fine, but the example interface code refuses to find the module. I modified the interface a bit to add error handling and check errno on the call to modfind() and found that it was returning an ENOENT. A bit of experimentation later, I discovered (via kldstat -v) that the actual name of sc_example was "sys/sc_example", not just "sc_example".

Next, I make an attempt to install a syscall into OpenBSD. This is a little more tricky due to a lack of LKMs. First, I create a file with a syscall function in it at sys/kern/sys_example.c. This syscall function is following the same prototype as all other OpenBSD syscalls, which is similar, but slightly different than FreeBSD. The syscall function simply outputs "proto wuz here" to the system console. 

From there, I found an unimplemented syscall number in sys/kern/syscalls.master (241), and updated the master list to include my sys_example syscall and called "make syscalls" in sys/kern, then added kern/sys_example.c to sys/conf/files to make sure it is included in the build.

This operation led to chasing compiler errors around missing includes for a while, but after all of the required headers were included, I was able to do a standard kernel build following the directions in release(8). After reboot, I was able to call syscall 241 from Perl and have the kernel display my message in the system console:

[peter@dev ~]$ perl -e "syscall(241);"
[peter@dev ~]$ dmesg | tail -1

proto wuz here

Here is the code from sys/kern/sys_example.c:


My final challenge for this post is to make the function behave the same as the example in the BSD Rootkit book, by accepting a string as an argument to the syscall and printing that to the console. I had to update my entry in syscalls.master to include the new argument, and include sys/syscallargs.h as well as a few other various headers. From there, the operation is almost identical to FreeBSD, except that a macro is used to access the arguments from their structure:



And the output from calling the syscall:

[peter@dev ~]$ perl -e '$str = "this is an OpenBSD syscall";' -e 'syscall(241, $str);'
[peter@dev ~]$ dmesg | tail -1

proto says: this is an OpenBSD syscall.

I obviously have some learning to do regarding code management in CVS, as my changes broke a bunch of the change/copyright information inside the files that were modified. I'm not too worried about it for the time being, as I don't intend to submit these changes upstream.

I did a little more code cleanup, and here is the final result:

If anyone finds anything that I've done wrong here, please feel free to yell at me about it in the comments.


References:

Designing BSD Rootkits: An Introduction to Kernel Hacking
by Joseph Kong
http://a.co/eSG5wWX

FAQ 5 - Building the System From Source
https://www.openbsd.org/faq/faq5.html

release(9)
https://man.openbsd.org/release

syscall(9)
https://man.openbsd.org/man9/syscall.9

Observium On OpenBSD

Note:  The following is a blow by blow of the installation, told in story form.  If you intend to follow this as a guide to the installation of Observium in OpenBSD, please read through to the end before starting the installation.  I will be show config snippets at varying stages of broken-ness.

Always being one to enjoy a technological challenge, I was presented with the task of building out a new management network.  I firmly believe that management of all devices on a network needs to be out of band from the rest of the network operation, so I was quite happy to take on this task.  After installing the new switches, and getting all of the monitor ports cabled up, I stopped to contemplate how I would set up my network monitoring suite, and how I would access this OOB network remotely.

Being a project that requires security (someone finding their way onto this network could be catastrophic), I decided that the best course of action was to set up a machine with a NIC in both my internal and my management networks, and run OpenBSD 5.3 on it.  A few months ago, the friendly folks over at #juniper on freenode recommended Observium as a network monitoring suite, and I have found it to be a fantastic project so far.  The Observium installation documentation is very clear that it supports Ubuntu/Debian only.  There is a second set of installation instructions for RHEL/CentOS.

So off I went with my OpenBSD installation, knowing full well that the Observium installation was probably going to be a painful one, but the machine that spans the gap between the two networks is a prime location for my monitoring utilities, so I decided to do it anyway.  Starting off with the OS installation, I accepted the defaults for the most part, configured my NICs for my network, and allowed OBSD to work its own disklabel magic (use the whole disk, autopartition).  When it came time to choose installation sets, I simply issued a -x* -games*, and went along my merry way (not fully appreciating the implications of my actions at the time).  Once the install was done, and I was presented with a prompt, the first order of business was to get the package mirror set up so I could begin installing the prerequisite software for Observium.  I chose to use the mirror in Denver, CO, USA.  This involved adding one line, and changing one line in .profile (changes in bold):


PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/sbin:/usr/local/bin
PKG_PATH=ftp://ftp3.usa.openbsd.org/pub/OpenBSD/5.3/packages/amd64/
export PATH PKG_PATH

Next, we need to get a proper editor, and a few tools:

# pkg_add -vi vim nmap svn

After everything is done installing, I picked up the prereqs for Observium from their install documentation:

# pkg_add -vi fping mysql-server net-snmp rrdtool graphviz php

This will chug along for a while, and occasionally ask a question.  When prompted for a php version, I chose the latest 5.3 release (per the prereqs), in the non ap2 build (this is for Apache 2, but Apache 1.3 ships with OBSD out of the box).  But...  the damn thing breaks on rrdtool...  After a bit of Googling, I discovered that the required libraries are part of xcore53...  one of the packages that I opted to not install earlier.  SOOOOO, off to the closest mirror to grab xcore53.tgz, and run a cd / && tar -zxvpf ~/xcore53.tgz, start the pkg_add over, and we're done.

I brought up the mysql database with /usr/local/bin/mysql_install_db, then set the root password, and created a database for Observium according to the installation instructions.  I set the database credentials in /var/www/observium/config.php.

Next, I decided to install observium into /usr/local/observium, as I didn't like the idea of using /opt.  I pulled the code down via SVN per the instructions on the site, and dropped the config snippet from the RHEL instructions into /var/www/conf/httpd.conf, with some minor modifications:



       ServerAdmin webmaster@localhost
       DocumentRoot /usr/local/observium/html
       
               Options FollowSymLinks
               AllowOverride None
       
       
               Options Indexes FollowSymLinks MultiViews
               AllowOverride All
               Order allow,deny
               allow from all
       
       ErrorLog  /usr/local/observium/logs/error.log
       LogLevel warn
       CustomLog  /usr/local/observium/logs/access.log combined


I wound up chasing my tail for an hour or two messing with the httpd configs, before I realized that httpd had chrooted itself into /var/www (hence the funny config location).  I quickly moved the Observium install over to /var/www/observium, and changed the httpd config again:



       ServerAdmin webmaster@localhost
       DocumentRoot /var/www/observium/html
       
               Options FollowSymLinks
               AllowOverride None
       
       
               Options Indexes FollowSymLinks MultiViews
               AllowOverride All
               Order allow,deny
               allow from all
       
       ErrorLog  /var/www/observium/logs/error.log
       LogLevel warn
       CustomLog  /var/www/observium/logs/access.log combined


Much better!  Now we get an error about the database.  pkg_add -vi php-mysql 

Now getting an error about not being able to bind to the MySQL socket.  Wow, wonderful.  More Googling revealed that, yet again, the chroot was biting me:

# mkdir -p /var/www/var/run/mysql
# ln /var/run/mysql/mysql.sock /var/www/var/run/mysql/mysql.sock

Now that we're past that, OH LOOK, It's a 500 error!  After a few hours of sticking die()'s into the code, I figured out that since I was in a chroot, the install_dir variable would be relative to /var/www.  Once this variable was set from /var/www/observium to /observium, the web UI loaded right up.  When I attempted to add a user, I found that adduser.php was now throwing errors about not being able to find functions that should have been included with common.php.  Turns out that even though Apache is chrooted, the php cli, is not.  DOH!  ln -s /var/www/observium /observium  It's a hack, but hey, it works now... Kind of...  In reality, it just fails differently.  Turns out I missed a step in the install documentation, and didn't initialize the database...

After I got the DB initialized, and a user created, I realized that addhost.php wasn't working.  After more tinkering, I realized that all of the default paths for the utilities were wrong, because the software was designed to run in Linux.  I grabbed the whole block of utility path define()'s out of the defaults file, and gave them the correct paths, and lo and behold!  OBSERVIUM IN OBSD.

I will most likely be revisiting this topic in the not-too-distant future, as I find more things that I need to hack around, but for the time being, It looks like I have monitoring running on the machine I wanted it on.  Now it's time for pf!