Observium On OpenBSD

Note:  The following is a blow by blow of the installation, told in story form.  If you intend to follow this as a guide to the installation of Observium in OpenBSD, please read through to the end before starting the installation.  I will be show config snippets at varying stages of broken-ness.

Always being one to enjoy a technological challenge, I was presented with the task of building out a new management network.  I firmly believe that management of all devices on a network needs to be out of band from the rest of the network operation, so I was quite happy to take on this task.  After installing the new switches, and getting all of the monitor ports cabled up, I stopped to contemplate how I would set up my network monitoring suite, and how I would access this OOB network remotely.

Being a project that requires security (someone finding their way onto this network could be catastrophic), I decided that the best course of action was to set up a machine with a NIC in both my internal and my management networks, and run OpenBSD 5.3 on it.  A few months ago, the friendly folks over at #juniper on freenode recommended Observium as a network monitoring suite, and I have found it to be a fantastic project so far.  The Observium installation documentation is very clear that it supports Ubuntu/Debian only.  There is a second set of installation instructions for RHEL/CentOS.

So off I went with my OpenBSD installation, knowing full well that the Observium installation was probably going to be a painful one, but the machine that spans the gap between the two networks is a prime location for my monitoring utilities, so I decided to do it anyway.  Starting off with the OS installation, I accepted the defaults for the most part, configured my NICs for my network, and allowed OBSD to work its own disklabel magic (use the whole disk, autopartition).  When it came time to choose installation sets, I simply issued a -x* -games*, and went along my merry way (not fully appreciating the implications of my actions at the time).  Once the install was done, and I was presented with a prompt, the first order of business was to get the package mirror set up so I could begin installing the prerequisite software for Observium.  I chose to use the mirror in Denver, CO, USA.  This involved adding one line, and changing one line in .profile (changes in bold):


PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/sbin:/usr/local/bin
PKG_PATH=ftp://ftp3.usa.openbsd.org/pub/OpenBSD/5.3/packages/amd64/
export PATH PKG_PATH

Next, we need to get a proper editor, and a few tools:

# pkg_add -vi vim nmap svn

After everything is done installing, I picked up the prereqs for Observium from their install documentation:

# pkg_add -vi fping mysql-server net-snmp rrdtool graphviz php

This will chug along for a while, and occasionally ask a question.  When prompted for a php version, I chose the latest 5.3 release (per the prereqs), in the non ap2 build (this is for Apache 2, but Apache 1.3 ships with OBSD out of the box).  But...  the damn thing breaks on rrdtool...  After a bit of Googling, I discovered that the required libraries are part of xcore53...  one of the packages that I opted to not install earlier.  SOOOOO, off to the closest mirror to grab xcore53.tgz, and run a cd / && tar -zxvpf ~/xcore53.tgz, start the pkg_add over, and we're done.

I brought up the mysql database with /usr/local/bin/mysql_install_db, then set the root password, and created a database for Observium according to the installation instructions.  I set the database credentials in /var/www/observium/config.php.

Next, I decided to install observium into /usr/local/observium, as I didn't like the idea of using /opt.  I pulled the code down via SVN per the instructions on the site, and dropped the config snippet from the RHEL instructions into /var/www/conf/httpd.conf, with some minor modifications:



       ServerAdmin webmaster@localhost
       DocumentRoot /usr/local/observium/html
       
               Options FollowSymLinks
               AllowOverride None
       
       
               Options Indexes FollowSymLinks MultiViews
               AllowOverride All
               Order allow,deny
               allow from all
       
       ErrorLog  /usr/local/observium/logs/error.log
       LogLevel warn
       CustomLog  /usr/local/observium/logs/access.log combined


I wound up chasing my tail for an hour or two messing with the httpd configs, before I realized that httpd had chrooted itself into /var/www (hence the funny config location).  I quickly moved the Observium install over to /var/www/observium, and changed the httpd config again:



       ServerAdmin webmaster@localhost
       DocumentRoot /var/www/observium/html
       
               Options FollowSymLinks
               AllowOverride None
       
       
               Options Indexes FollowSymLinks MultiViews
               AllowOverride All
               Order allow,deny
               allow from all
       
       ErrorLog  /var/www/observium/logs/error.log
       LogLevel warn
       CustomLog  /var/www/observium/logs/access.log combined


Much better!  Now we get an error about the database.  pkg_add -vi php-mysql 

Now getting an error about not being able to bind to the MySQL socket.  Wow, wonderful.  More Googling revealed that, yet again, the chroot was biting me:

# mkdir -p /var/www/var/run/mysql
# ln /var/run/mysql/mysql.sock /var/www/var/run/mysql/mysql.sock

Now that we're past that, OH LOOK, It's a 500 error!  After a few hours of sticking die()'s into the code, I figured out that since I was in a chroot, the install_dir variable would be relative to /var/www.  Once this variable was set from /var/www/observium to /observium, the web UI loaded right up.  When I attempted to add a user, I found that adduser.php was now throwing errors about not being able to find functions that should have been included with common.php.  Turns out that even though Apache is chrooted, the php cli, is not.  DOH!  ln -s /var/www/observium /observium  It's a hack, but hey, it works now... Kind of...  In reality, it just fails differently.  Turns out I missed a step in the install documentation, and didn't initialize the database...

After I got the DB initialized, and a user created, I realized that addhost.php wasn't working.  After more tinkering, I realized that all of the default paths for the utilities were wrong, because the software was designed to run in Linux.  I grabbed the whole block of utility path define()'s out of the defaults file, and gave them the correct paths, and lo and behold!  OBSERVIUM IN OBSD.

I will most likely be revisiting this topic in the not-too-distant future, as I find more things that I need to hack around, but for the time being, It looks like I have monitoring running on the machine I wanted it on.  Now it's time for pf!

Book Review: The Absolute Beginner's Guide to Binary

At one of the Defcon 101 talks given by 1o57, a complaint was made.  1o57's talk was called Hacking the Hacker, and was a rundown of skills that anyone who considers themselves to be a hacker should have, or be striving towards.  1o57 was concerned at the state of the hacker scene, in that many people he talked to couldn't even do a 4-bin binary count-up.

The talk was great, but left me concerned.  I picked up a copy of The Absolute Beginner's Guide to Binary by Greg Perry, as a refresher, and found it to be quite an enjoyable read.  Greg explains Binary, Bits, Bytes, and Hex in enough detail as to not leave anything out, yet at a level of readability that I would recommend it even to complete novices of base-2 math, and computers in general as a great text to fully explain the topic to them.

Mr. Perry also gives several topics throughout the book that he leaves to the reader as areas for farther research, which is something that I, personally, find to be very attractive.   It lets the reader branch out and expand their horizons in an organic way.  This is a fantastic learning style for hackers.  Thus, I would definitely recommend this book to anyone who is interested in computers, electronics, or digital communications, but needs a bit of work at the bits and bytes level.

Finding Unused IPs

I recently had a need to fit a new host onto a crowded network, without well documented address assignments.  This led me to several obvious answers, all of which required reading long lists of IP addresses while looking for gaps here and there.  Since reading through DNS zone files, or filtering Nmap ping sweep output didn't sound like much fun, I threw this little guy together.

#!/usr/bin/perl
#############################
## ipchecker.pl            ##
## (c)2012 Peter H. Ezetta ##
#############################


use strict;
use warnings;
use 5.010;

my @results;

foreach (`nmap -sP -v 192.168.0.*`) {
    chomp;
    push( @results, $_ ) if (/^Host/);
}

foreach (@results) {
    say if (/down/);
}

Change the address in the foreach loop to match the network you're scanning, and Nmap will run a ping scan like normal, with a nice little Perl wrapper to only give you the hosts that are down, instead of  up.

"Borrowing" A Screen Session

GNU Screen is a tool that most everyone reading this will be aware of.  In a nutshell, it's a terminal multiplexer, that is, it allows multiple virtual terminals to be brought up in one screen session.  Very useful when you're on the terminal.

Another purpose that is used quite often is to allow long running scripts to run on a remote server, without having to worry about your SSH session getting disconnected.  One would simply start a screen session, run their script, and detach from the screen session, leaving the script to do it's work.

This can occasionally be a problem when one system administrator has started a long running process, and for whatever reason, another system administrator needs to get into the screen session, when it's owner isn't present.  Attempting to su user, then attach to the screen session will lead to the following error:


[14:43:06] [peter@server ~]$ sudo su user
[sudo] password for peter: 
[14:43:45] [user@server /home/peter]$ screen -l
Cannot open your terminal '/dev/pts/2' - please check.
[14:43:53] [user@server /home/peter]$

A quick check of the permissions of /dev/pts/2 reveals the problem:


[14:43:53] [user@server /home/peter]$ cd /dev/pts/
[14:51:42] [user@server /dev/pts]$ ls -lsa 2
0 crw--w---- 1 peter tty 136, 2 2012-11-04 14:51 2

As we can see, the terminal I'm connected to is owned by my user, not effective user that we assumed with su user.  One solution would be to chmod +w /dev/pts/2, but allowing other users to connect to our pts device poses a security risk.  Enter script.

Script is used to create a typescript recording of everything that happens in a terminal, as long as script is running.  It will connect to a new tty device when it is called:

[15:00:20] [user@server ~]$ script test

Script started, file is test

[15:00:25] [user@server ~]$ tty

/dev/pts/3

[15:00:32] [user@server ~]$ ls -lsa /dev/pts/3

0 crw--w---- 1 user tty 136, 3 2012-11-04 15:00 /dev/pts/3

As you can see, by calling script, we have obtained a new pts device, and it's owned by user, not by peter.  This means that we can now screen -d -r to attach to user's screens, even in his absence.

One last note:  If you don't have a purpose for the typescript file that is generated by script, just give it /dev/null as an output file:  script /dev/null, which will throw away the typescript.

Lines of Code

This is going to be a short one.  The other day, one of my friends asked me for a quick way to tell how many lines of code were in a project directory.  Here was my solution (thank you, Perl!)

$ find . -name '*.pl' | xargs perl -e \
'my $x = 0; foreach (<>) { $x++ }; print "$x\n"'

Of course the *.pl can be changed to whatever extension you would like, and TIMTOWTDI, but I happen to be fond of foreach loops and the <> operator, so...

Loopback Filesystems Revisited

In a post quite some time ago, I discussed loopback filesystems.  Well, lo and behold, a few weeks ago, I had the opportunity to play with them again.  Xen, a Linux VM Hypervisor (http://www.xen.org) mounts up its disk images as loopbacks.

I found out rather quickly, that the default number of loopback devices available to the Kernel is 8.  When you have one image for disk, and one for swap on each domU (virtual guest), that means you can only cram 4 machines onto a single hypervisor by default...  That's just unacceptable.

I found two methods that worked, although one is a bit more painful than the other:

Hypervisor with Guests running in Production:

When you can't reboot the machine for whatever reason (in my case, it was running VM's that I didn't have any space to migrate, and couldn't shut down), you can modprobe the loop module again and pass it a parameter:

# modprobe loop max_loop=128

This is all fine and good, really, except for the fact that you don't get nodes created for the loop devices in /dev automatically, so you have to do some lovely work with mknod before you can use your new devices, even though the kernel is aware of them:

# mknod -m660 /dev/loop8 b 7 8

Loop devices have a major number of 7, so make sure to change both 8's to the appropriate value for your system.  The defaults on the systems I've been working with (Debian Lenny, Debian Squeeze, and Ubuntu Precise (12.04 LTS)) are to have 8 loop devices, loop0 - loop7.

Of course making 120 new loop devices by hand isn't exactly fun... Time for bash-fu!

# for x in `seq 8 128`; do mknod -m660 /dev/loop$x b 7 $x; done;

This gives us our 128 loopback devices (assuming you had 0-7 OOTB), with the nodes created in /dev.  If you reboot, they're gone, so make sure to add the options to /etc/modules.conf or wherever your flavor decided to keep the module options.

Hypervisor without Guests (or can be rebooted):

If the machine can be rebooted, the situation is much easier.  My solution was to pass the parameter to the kernel at boot time via GRUB.

Simply set the following in /etc/default/grub (for Debian based distros):

GRUB_CMDLINE_LINUX_DEFAULT="max_loop=128 quiet"

Then rebuild your GRUB config and reboot:

# update-grub && shutdown -r now

When the machine comes back up, GRUB will have passed the max_loop option to the kernel at boot, and the kernel will have automatically created all of your loopback devices for you.

Obviously, this is a significantly simpler operation than creating the nodes in place, and if you have the option to reboot, I would say go for it.

Part I

protosho v1.0 (almost)

Admittedly, I haven't been playing with Dettu[xX] too much over the last month, but I did come back to a stagnant code project that I started in late September or early October, and did a bit of general cleanup and prep work towards being something thats fit for (very adventurous) human consumption...

It doesn't have a working installer (I'd advise leaving Makefile.PL well alone unless you know what you're doing... it's not complete), it doesn't have any documentation other than the comments in the code, but if you can get it to run, it does it's job.

So, for the moment that no-one has been waiting for, I give you protosho:

https://github.com/protoCall7/protosho

This little baby is a CLI::Framework based SHODAN interface for the commandline.  It's written 100% in Perl 5.14, it's licensed under the BSD 3-clause, and at the moment, it kind of sucks.  But I've had a ton of fun playing with the results that it cranks out, so for anyone else out there who likes to tinker with Perl, or has a need for some SHODAN on the CLI, or is just feeling techno-masochistic, grab a git clone of it, and leave me a comment here, or on the Github issues board, and I'll help you through getting it installed.

In the meantime, you should also check out SHODAN @ http://www.shodanhq.com

proto