I’m the fool. I took the plunge this week and installed a fresh clean copy of Fedora 11, but I made my root partition btrfs instead of something more… well, for lack of better words, stable and mature. This should serve as a little documentation of my findings, in the hopes that others don’t make the hasty call to dedicate their root partition to the new filesystem.

From the dawn of my days with Linux, I have been told to heed warnings that this new system gives. This isn’t Windows any more, the warnings that you receive are no longer benign. My first experience with this was when I had a hard drive that was dying, so I decided to do a fsck. The only problem was that I did a fsck on the root partition – while it was still mounted — read/write. In all fairness, it was my own arrogance that made me add the flag to manually override that protection system. I figured, “what could get worse? The drive is already dying”. I did this with no backups. Three days and 20-something manual superblock hex-edits later, I finally got my data back. The valuable thing from that incident is the experience, though. I now knew how to do manual superblock recreations, which always comes in super handy. I also had learned to heed the warnings of the core developers very well.

Until Saturday. My friend had told me before about how funny the new anaconda argument is: “icantbelieveitsnotbtr”. And now who says these hardcore developers don’t have a sense of humor? Well, I’ve learned a heck of a lot over these last few years, and sometimes I even feel guilty that I don’t give back as much as I should to the open source community. So I figured I’d give their hard work a shot. I’ve heard nothing but glowing reviews about ZFS, and btrfs is supposed to be the up-and-coming big thing in the filesystem world. I figured, at the very least, I could do some benchmarks or something.

So here goes nothing…

I fired up the installer, adding the appropriate flags. I had /dev/sda1 already formatted 50 gig ntfs for Windows. This was to be a dual-booting system. I added a 512-meg /boot, sda2, and a 2-gig swap, sda3. sda4 was to be a pointer to an extended partition table, where sda5 resides, filling up all of the remaining space of 59 gigs. I picked my packages and started the installer. Uh oh, the format of sda5 as btrfs failed. Oh well. Reboot, repeat. Same results, but this time it just died with some non-helpful error code 1. Uhhh. So I tried to boot again. My partitioning table was simply gone. WTF.

I remade the partition table, and figured, “third time’s the charm!”. I selected btrfs as the filesystem for /. If this didn’t work, I was going to give up and use the (now-default) ext4 filesystem. Alas, it worked. I was actually looking forward to using the new filesystem. The install finished normally, and I booted into my new Linux system.

First thing’s first, I ran a full yum upgrade. I noticed something just then. The seek times with btrfs were amazing, read speeds were okay, but the write speeds were flat-out terrible. I’m guessing that’s simply because of all of the metadata btrfs stores, but I honestly have no idea why that is. Oh well, I do far more reads than writes, so this works out well. I then proceeded to use the system for a while. After my used disk space started to get above 25%, I started to get really unexpected results – kernel failures, corrupted filesystem blocks, half-missing extent entries, system bus errors, the whole nine yards.

I did the next logical thing, I strace’d the processes that were dying unexpectedly on me. Worst of all was yum, it would just die with a “Bus Error”. Strace revealed it was hitting massive ENOSPC (no space left) errors trying to open files on the hard drive. df reported the filesystem as exactly 27% full. A google search revealed this as a known bug, but no one else has really had the problem until their filesystem reached over 75% full. So why was mine being the opposite? I theorized that it may have been because of loads and loads of useless metadata across the hard drive. I rebooted (the only thing that fixed the ENOSPC errors) and installed the btrfs-progs package. This package includes userspace utilities needed to create, check, modify and correct any inconsistencies in the btrfs filesystem. I ran a btrfsck, which reported no problems. I read that it may be useful to re-examine my metadata by forcing a balance of it across all drives, even though it only existed on one partition. This is done with the command “btrfs-vol -b /”. Kicker – this operation, after about 5 minutes of heavily accessing the drive, segfaulted. I was left with a half-balanced btrfs system, ENOSPC errors everywhere, and a feeling like this was the beginning of the end.

I didn’t quite want to give up yet, but I also wanted a usable machine. I had read somewhere that kernel 2.6.30 had an insane number of btrfs improvements. On a personal wish, I downloaded the kernel source for 2.6.31 and compiled. Booting into the new kernel was like a breath of fresh air.

SIDE NOTE: kernel > 2.6.30 is awesome. readahead is now merged into the kernel source for MUCH faster booting. There have been a ludicrous amount of improvements, check the changelog on kernel.org for a good list of these changes.

Anyway, the new kernel loaded up just fine. The errors held off for a bit, long enough to make me think they were gone. But then they came back full-force. The balance segfaulted again. The fsck said nothing was wrong. No one had any idea what was going on. I do like to help the open source community, but I simply don’t have time at the present to dig through hundreds of thousands of C code source lines in order to fix a bug this serious. After filing 4 bug reports, submitting probably close to 50 kernel errors, and genuinely giving it my best college try, I was done with btrfs. I rebooted into anaconda and formatted my system as ext4 and went along my way. Yes, I know you can *theoretically* convert a btrfs volume to ext4, but with the structure so far destroyed, I didn’t even bother wasting my time.

Moral? I guess just know what you’re getting into if you decide to give btrfs a try. When the kernel developers say that a feature is experimental, take that literally. There is no warning here, only some sound advice – proceed at your own risk, make plenty of backups, and just generally don’t expect too much of the filesystem at this early stage of it’s development.

My coworker just pointed me in the direction of this document. This is, I believe, the MOST USELESS tutorial of all time. It highlights using the system’s RAM as a swap partition, by zeroing a swapfile in /dev/shm and then mkswap’ing it.

I ask, what’s the point of swap? Well, it’s an “overflow reservoir” if you will, storing memory pages on disk when RAM is full. It’s stored on disk because disk space is cheap, but memory space is not. So swap only really gets used when RAM is full. So if you put a swap file in RAM… what the hell did you accomplish? There’s no performance gain to be seen whatsoever! The only effect is that now you have the overhead that results from storing files, which means LESS RAM available for applications.

The really funny part is how the author claims he saw significant performance gains. The number of comments saying, “great idea!!!” is just unbelievable as well.

If you’re thinking to yourself that it may be useful since really old pages get swapped even though they could potentially fit, I point you to the kernel proc parameters. Running “sysctl vm.swappiness=0” will make the kernel swap as infrequently as possible without causing issues with your running applications. The default value of this parameter is (usually) 60.

Anyway, here’s the link. Read it for the facepalm… you should have a handprint on your forehead afterwards if you read it correctly.

Link: http://kerneltrap.org/node/3660

So maybe the title of this article is really a bit overkill. Or maybe It’s true. Maybe you should decide….

See, Bruce runs the fancy email newsletter known as “Cryptogram”. It’s actually a very good newsletter, so props to Bruce for that. It’s really just a monthly conglomeration of stories that relate to security, be it software, hardware, social, or otherwise. I have enjoyed the newsletter thoroughly for a while now, so I’m not just complaining about something after my first experience with it.

And congress, well, they seem to run the country. Which is funny, but we’ll get to that later.

Bruce made an interesting claim in his email newsletter a little bit ago – that the Chinese Government was developing some sort of secret and powerful operating system hellbent on attacking the critical computing infrastructure of the United States of America. To be fair, however, he did put this bit of text in quotation marks. Apparently this is not his doing. The backstory seems to follow that a briefing was given to congress not too long ago about this very subject. Apparently it’s called “Kylin”, and as a member of the IT community, we’re supposed to fear it.

Well lo and behold, the other day at work, I was setting up a snort Intrusion Detection System, and what did I see? Some very weird packets bouncing to-and-fro around the network. I mean _weird_ packets. Combinations of SYN/ACK/RST/FIN flags, sometimes there was an URG/PSH flag set just to throw off any sensors that might be listening. Fortunately, I log ALL packets to pcap data files, so I fired up tcpdump and extracted the relevant data. Insert this data into wireshark for some nice analysis, and the weirdest patterns were showing up. Scans like I really have not seen ever, but ones that make logical sense. Some ACK packets were being sent across the wire in the hopes to solicit an RST packet from a listening port. There were just strange packets everywhere, highly anomalous packets, some were even SYN/ACK/FIN packets. Not quite a christmas tree, but something that will never occur in nature. So I did what I could, and traced the packets back to their source. The funny thing was, they were bouncing all around the world. Africa, Asia, South America — nearly every country was seemingly scanning the network. And as quick as it started, it was over. I realized what was going on… A massive distributed reconnaissance mission just took place on our network. Combinations of weird packets and different sources all came together in harmony to do a full 65536-port scan on the network, and almost all without sending off a single alert from snort. Properly collated, this raw data could be put into a map of every open service on the network, and if it weren’t for a single seemingly benign alert, I would have not even known something just happened. The most chilling thing is, the possibility certainly exists that this has happened before and will happen again, but I just happened to be lucky enough to catch it this time as it happened.

I needed a lead. This was too strange. So I started analysing the packets that were sent across the wire. The ICMP packets looked very familiar, and upon further investigation, I noticed that these ICMP packets were somewhat verbatim from the FreeBSD kernel. The actual data portion of the packets vary from OS to OS… don’t believe me, then look it up! Suddenly I noticed that some packets were coming from China. I don’t know why I noticed this, but it just seemed like a pertinent detail. I think it was my subconscious telling me exactly what I feared – I was facing this “new and secret” Chinese operating system. I google’d “Kylin” and, rather unsurprisingly, the first page of results were all the same fear-mongering based on the briefing that was given to congress. But what if I changed the search a little bit maybe someone has seen this before? In this light, I google’d “Kylin FreeBSD”.

WELL WHAT HAVE WE HERE?

Kylin is not some secret Chinese government project to destroy America. It’s not even a secret. It’s a state-sponsored project to secure a common open source product. It’s absolutely no different than the US NSA’s SELinux. For crying out loud, YOU CAN DOWNLOAD THE ISO’S! It’s almost disappointing – I was secretly hoping deep down inside that I was the first person to encounter Kylin’s doings first-hand. Not so much. So I gave Bruce’s newsletter a once-over again. Looks like I was wrong all along, Bruce didn’t believe it either. He even made the conjecture that it was just over-hyped nonsense. But why even write about it as if it COULD be legit if a simple google search shows it’s all just overhyped garbage reporting? Like I said, Bruce needs to learn to use the Interwebs a little better!

And as for congress, well, they do too. I’m tired of the fear-mongerting that occurs so very, very often. I’m tired of the irresponsibility of it all. What does it accomplish? Maybe it pushes your own agenda a little faster? Why not do the right thing and report the facts, then maybe we can have some time to plan things out and do it right the first time. I’m all for a national Cybersecurity initiative, in both the public and private sectors. It’s truly a great thing that this is happening. I would just really hate to see it all go as wasted effort because proper time and resources were not allocated ahead to keep the project alive. Perhaps I’m just an idealist this way, but it’s the only way that (IMHO) works.

Now, in the light of sharing and freedom, here’s the Kylin ISO’s. Use responsibly, and above all else, use them to learn! This is no different than SELinux, and maybe the Chinese government will continue to sponsor open source projects like this, something from which we surely can ALL benefit. And don’t believe everything you read, I know I’ve learned my lesson this time around…

And as for the scans I was seeing? Who knows, it could have been one person all along, or it could have just been coincidence. It’s really hard to say, but that’s the nature of the beast we call the Internet. Surely, nothing that was done is out of the reach of some clever SSH’ing and NMapping. If there’s more to the story, I’ll make an update. But for now, I’m content with the facts that I have, no more, no less.

DOWNLOAD LINKS:
DISC ONE: [KYLIN-2.1-1A.iso]
DISC TWO: [KYLIN-2.1-1B.iso]
MD5SUMS: [KYLIN-checksum-2.1-1.md5]

Server is now running much faster, if you can’t already tell. The images may still kinda load slow, but that’s not necessarily the fault of the machine! The upgrade actually went as planned, just plugged in the network and power cables, powered her on, and it all worked first shot. On second thought, nothing ever works perfectly like that the very first time, so please contact me if anything strange is found on this server. Let’s hope all is well with it, though!

It’s now pushing 3 GHz with a good hunk of RAM and the newest version of my favorite Linux distribution.

Look for some good blog articles coming soon. 🙂

Ok so the server this website sits on? It’s terrible. 550 MHz. I think 384 Megs of RAM. It’s time for an upgrade, to say the least!

That’s why shortly a new server will be taking this one’s place. A much better server, without such things as a slow processor…

Anyway, expect some downtime this weekend (if you actually follow this blog) so that the server can be replaced.

I have a lot of good ideas for blog entries to write, but I’m really sick and tired of this site taking literally minutes to load.

See you soon!