Suppose your friend wanted to install a video camera on your car. The video camera would only be able to see the treads of your tires as part of a study. He could only tell from this feed the depth of the treads of your tires. Nothing more, nothing less. Would you be particularly offended by this? Odds are probably not, this information is pretty benign. But what if this was a complete stranger who wanted to put the camera on your car? You’d probably be very wary of it, and inspect the camera to make sure it did exactly what he said it does. Fair enough, as the information is still pretty benign and he was pretty honest about it to begin with.

Now consider you walked outside, and one day found a video camera watching your tires just sitting there. You’ve had the car for years, you don’t know where the information is going, and you have no idea how the camera got there. And to boot, it looks like someone attempted to hide the camera — poorly. Perhaps by covering it with some clear tape or something. You’d be pretty pissed that someone did this, right?

I’ve been having issues from time to time with this blog. Randomly, the sidebars on the right load very slowly as compared to the rest of the document. I got really curious as to why the other day when it was happening again, and so I decided to figure it out. I had a suspicion that it was the mysql database, but I wanted to make sure. So I started an strace on the http daemon, and refreshed the page. I really was quite pissed at what I saw next. There was an outgoing http connection I’d never seen to a website I’d never heard of before. The connection was made moments after a gigantic glob of data was read from a php file on the filesystem. It started with eval(gzinflate(string_rot13(base64_decode(…………)))) and at this point, I knew I was in trouble. It was hidden code I wasn’t supposed to see running on my website.

See, when I first got this blog, I started by finding a nice theme. I did eventually come across the one you see now, and I liked it. A tad bit land-of-the-rising-sunny for me, but whatever, it did look nice. So I installed it. There were some plugs down the bottom of the page, one my friend even asked me about. He said, “dude, what’s with the plug for Burt’s Bee’s?”. I said, “I dunno, but the author of the document put it there, and he did ask that I not remove it, so I left it.” I swear I did the honest thing!

But then I see the camera pointing at my tires that I’d never authorized. The outgoing HTTP connection, that is. I quickly edited /etc/hosts and changed the hostname it was looking for to address 127.0.0.1. I refreshed the page. What happened? The footer and all the plugs (Burt’s Bees included) disappeared. The outgoing connection was actually quite legit. I wrote a quick php script myself to parse the output of the several layers of evaluations of random blocks of data. SEVENTY-ONE EVALUATIONS LATER, I attained the source code. It is a large user-agent tracking system. I’ve left the source code [right here].

So, I hate to inform you, but if you’ve visited this site in the past month or so, you have had some of your more public information read in by some stranger on the internet. No worries, the information was completely benign, simple stuff like your browser version and operating system type, hence the reference to tire tread. It’s stuff that I really wouldn’t have cared too much about the author collecting… if only he had just ASKED!. So now I found out that the entire footer of the page was actually coming from his server, which could mean simply one thing – Burt’s Bees is PAYING for this kind of shady advertising. That’s a horrible business practice!

In response, I’ve removed the offending code and published it in the link above. I’ve also blackhole routed any traffic from the offending website. The name of the theme, by the way, is SoulVision, and I actually did get it from a reputable repository of quality wordpress themes. As someone in the security field, I recognize that this could have been much worse than it was, and I realize just how lucky I am that it wasn’t. I guess my moral for the day is to always verify code that’s going into production on a locked down server. Oh, that, and I’ll probably never buy Burt’s Bees. Ever. End ‘o discussion on ethics.

Keri Rand has written a good article that has been published in the Advertiser News, one of the regional papers where I live. She sent out a questionnaire via facebook message to a bunch of people, and I was one who responded. She decided to use my input, and now I’m featured on their website.

I may sound like an absolute tool, but I do like the one quote that was made. They highlighted it in the actual paper printing, but not online. Quote:

“This recession has sort of made me go ‘back to the basics’ (by American standards), and appreciate the simpler things in life a whole lot more. I think it’s also impacted my health positively as well – I cook now instead of buying prepared foods, I exercise more for lack of anything ‘more exciting’ to do. Honestly, I kind of like the new lifestyle I’ve adopted because of it.”

So what does everyone else here think, has the latest economic recession affected you enough to make some lifestyle changes? It is an interesting thing to consider.

Link: [Teens and 20-somethings reflect on the state of economy]

The nagios plugin that you find in the package nagios-plugins-snmp was insufficient for my needs in a new nagios deployment. The biggest reason that it was insufficient was that it gets integer values, and then can only issue a warning or alert if that integer value is GREATER than what value you gave it. The deployment I’m setting up required that the values could be checked against a range, which is necessary when receiving SNMP data from a thermometer or hygrometer. This plugin supports receiving a range of values to check against, and then it returns the appropriate exit code.

I’ve written a tad bit of documentation in the top of the file, but here it is again, in block quotes! Download link is just below the block quote area.

# This script written with haste by Benjamin Rose, July 8th 2009 @ 11:45:42 AM
# It was written because the check_snmp plugin provided by the nagios package
# does not support range matching. It can check if the snmp value is greater than
# a given number but not less than, nor a range consisting of either a high value
# or a low value. Hence, this script, given a mode of 1 and a good range with
# which to work, will report back appropriately.
#
# Modes:
# 1 = Number comparison, reports on a given range. Argument order given in
# the usage statement.
# 2 = String comparison, which for now is just “Open” or otherwise.
#
# TODO:
# 1) Change the order of the arguments, putting mode in front of the
# variables, and then change the usage based on the given mode.
# 2) Allow the user to configure which strings are “good” and which are “bad”.

Plugin link: [snmp_plugin_wrapper]

It exists! It’s not in the Adobe yum repos yet, as it’s still an alpha build, but you can download it here:

[Native Linux 64-bit Flash Player from Adobe Labs]

It works very well. No longer do you need to get nspluginwrapper working, and then configure pulseaudio (if that’s your distro) to listen to 32-bit apps, blah blah blah. Simply unpackage, throw the .so file into the firefox plugins directory, and restart firefox. Great stuff. I’ve also noticed there’s a performance gain to be seen, my flash videos used to lag and flicker on youtube when I made them fullscreen, and this doesn’t happen anymore. Sound worked out of the box with pulse.

And the best part, although it’s supposedly an alpha build, it crashes a heck of a lot less than it used to. No more npviewer.bin segfaults in the /var/log/messages file! My guess is that this is because of the lack of a hacked wrapper. But, that being said, if it crashes on you a lot after trying it, don’t be too upset. At least they’re trying.

I’m the fool. I took the plunge this week and installed a fresh clean copy of Fedora 11, but I made my root partition btrfs instead of something more… well, for lack of better words, stable and mature. This should serve as a little documentation of my findings, in the hopes that others don’t make the hasty call to dedicate their root partition to the new filesystem.

From the dawn of my days with Linux, I have been told to heed warnings that this new system gives. This isn’t Windows any more, the warnings that you receive are no longer benign. My first experience with this was when I had a hard drive that was dying, so I decided to do a fsck. The only problem was that I did a fsck on the root partition – while it was still mounted — read/write. In all fairness, it was my own arrogance that made me add the flag to manually override that protection system. I figured, “what could get worse? The drive is already dying”. I did this with no backups. Three days and 20-something manual superblock hex-edits later, I finally got my data back. The valuable thing from that incident is the experience, though. I now knew how to do manual superblock recreations, which always comes in super handy. I also had learned to heed the warnings of the core developers very well.

Until Saturday. My friend had told me before about how funny the new anaconda argument is: “icantbelieveitsnotbtr”. And now who says these hardcore developers don’t have a sense of humor? Well, I’ve learned a heck of a lot over these last few years, and sometimes I even feel guilty that I don’t give back as much as I should to the open source community. So I figured I’d give their hard work a shot. I’ve heard nothing but glowing reviews about ZFS, and btrfs is supposed to be the up-and-coming big thing in the filesystem world. I figured, at the very least, I could do some benchmarks or something.

So here goes nothing…

I fired up the installer, adding the appropriate flags. I had /dev/sda1 already formatted 50 gig ntfs for Windows. This was to be a dual-booting system. I added a 512-meg /boot, sda2, and a 2-gig swap, sda3. sda4 was to be a pointer to an extended partition table, where sda5 resides, filling up all of the remaining space of 59 gigs. I picked my packages and started the installer. Uh oh, the format of sda5 as btrfs failed. Oh well. Reboot, repeat. Same results, but this time it just died with some non-helpful error code 1. Uhhh. So I tried to boot again. My partitioning table was simply gone. WTF.

I remade the partition table, and figured, “third time’s the charm!”. I selected btrfs as the filesystem for /. If this didn’t work, I was going to give up and use the (now-default) ext4 filesystem. Alas, it worked. I was actually looking forward to using the new filesystem. The install finished normally, and I booted into my new Linux system.

First thing’s first, I ran a full yum upgrade. I noticed something just then. The seek times with btrfs were amazing, read speeds were okay, but the write speeds were flat-out terrible. I’m guessing that’s simply because of all of the metadata btrfs stores, but I honestly have no idea why that is. Oh well, I do far more reads than writes, so this works out well. I then proceeded to use the system for a while. After my used disk space started to get above 25%, I started to get really unexpected results – kernel failures, corrupted filesystem blocks, half-missing extent entries, system bus errors, the whole nine yards.

I did the next logical thing, I strace’d the processes that were dying unexpectedly on me. Worst of all was yum, it would just die with a “Bus Error”. Strace revealed it was hitting massive ENOSPC (no space left) errors trying to open files on the hard drive. df reported the filesystem as exactly 27% full. A google search revealed this as a known bug, but no one else has really had the problem until their filesystem reached over 75% full. So why was mine being the opposite? I theorized that it may have been because of loads and loads of useless metadata across the hard drive. I rebooted (the only thing that fixed the ENOSPC errors) and installed the btrfs-progs package. This package includes userspace utilities needed to create, check, modify and correct any inconsistencies in the btrfs filesystem. I ran a btrfsck, which reported no problems. I read that it may be useful to re-examine my metadata by forcing a balance of it across all drives, even though it only existed on one partition. This is done with the command “btrfs-vol -b /”. Kicker – this operation, after about 5 minutes of heavily accessing the drive, segfaulted. I was left with a half-balanced btrfs system, ENOSPC errors everywhere, and a feeling like this was the beginning of the end.

I didn’t quite want to give up yet, but I also wanted a usable machine. I had read somewhere that kernel 2.6.30 had an insane number of btrfs improvements. On a personal wish, I downloaded the kernel source for 2.6.31 and compiled. Booting into the new kernel was like a breath of fresh air.

SIDE NOTE: kernel > 2.6.30 is awesome. readahead is now merged into the kernel source for MUCH faster booting. There have been a ludicrous amount of improvements, check the changelog on kernel.org for a good list of these changes.

Anyway, the new kernel loaded up just fine. The errors held off for a bit, long enough to make me think they were gone. But then they came back full-force. The balance segfaulted again. The fsck said nothing was wrong. No one had any idea what was going on. I do like to help the open source community, but I simply don’t have time at the present to dig through hundreds of thousands of C code source lines in order to fix a bug this serious. After filing 4 bug reports, submitting probably close to 50 kernel errors, and genuinely giving it my best college try, I was done with btrfs. I rebooted into anaconda and formatted my system as ext4 and went along my way. Yes, I know you can *theoretically* convert a btrfs volume to ext4, but with the structure so far destroyed, I didn’t even bother wasting my time.

Moral? I guess just know what you’re getting into if you decide to give btrfs a try. When the kernel developers say that a feature is experimental, take that literally. There is no warning here, only some sound advice – proceed at your own risk, make plenty of backups, and just generally don’t expect too much of the filesystem at this early stage of it’s development.