I took several classes in college that were all about corporate buzzwords, but all of them failed to mention a trend that is huge in corporate culture – that of the “best practice” document. In theory, these documents are great, and help you get the most out of your product, methodology, or technology. However, there are a few problems that these documents tend to create, and it is up to the reader to take them with a grain of salt. Listed below are a few of the pitfalls of adhering directly to one of these documents.

1) No one is perfect
“Best practice” documents are generally written by an idealist. This person has the responsibility to make the product or technique about which they are writing perform optimally. This means there is very little consideration for outside variables. Points in best practice documents tend to be very black and white – Do this, do not do this other thing, etc. Yet, how often is it the case that a reader is concerned with making only one piece of a puzzle work flawlessly? The answer is a resounding “never”, it takes the other 499 pieces of the puzzle working in tandem as well to produce something worthwhile and appealing.

2) Context is key
While we’re on the subject of black and white bullet points, we should discuss context. While adhering directly to a “best practices” document, it becomes easy to miss apparent pitfalls right in front of you. The document might say, “XYZ should look like ABC.” It becomes increasingly easy to see only the fact that XYZ != ABC, and even easier to miss the circumstances that made XYZ a little different than ABC. As mentioned in point #1, the writer of the “best practices” document is concerned only with making XYZ the best it can be. Often it is the case, however, that the overall quality of a product can be increased by decreasing the effectiveness or simplicity of a single component in the equation. Hence, context really is everything.

3) Stagnation is an enemy
“Best practice” documents literally define the status quo. Anyone who uses the same product or technology that you use will have access to this document. Therefore, it follows that they will have the same setup, the same features, and the same problems as you. At this point, what sets you apart? Where is the innovation, the fresh ideas to shake up the ordinary? As I remember reading elsewhere, what would have happened if Google followed the search engine “best practices” set forth by Yahoo and Alta Vista? Where would Apple be today if they took pages from IBM’s book about personal computers? This all lends itself very nicely to the next point.

4) Opinions become squelched
Now suppose you or a member of your team has an idea that challenges something stated in the “best practices” document. Which do you follow? Who do you trust? Odds are that you or your team member has much more insight into the issue than a document. However, when adhering directly to one of these documents, it becomes way too easy to ignore the opposing opinions altogether and do what is recommended in the document.

Now, I’m not saying that “best practice” documents are complete garbage. Often they contain many helpful tips and tricks for getting the most out of whatever they are written about. Yet, it is important to remember they are not the bible. To summarize my feelings on the subject, context is everything, and if there’s one thing “best practice” documents lack, it is context.

It’s finally been done, users with an @allmybase.com email address can now send email to users @hotmail.com, @mail.usp.edu, and other addresses that use Microsoft Network (MSN) Live services. Some of you may not even be aware that you couldn’t send email to these users, so let me Tarantino this story for you a bit and go back to the beginning. The short of this story is that email services are now restored with minimal continuing effects, and if you are receiving the Microsoft 550 SC-001 bounce, [File a ticket with them by clicking right here]. But, if you want the good details of what it takes to get a result from Microsoft, read on. It should be mildly entertaining at best.

It all started when a user of mine complained that she could not send email to users @eden.rutgers.edu and users @mail.usp.edu (college friends). I checked my maillog, and found lines like these ones:

maillog-20091025: Oct 20 11:06:11 Darkflame postfix/smtp[12627]: 7107F4230: to=, relay=mx.eden.rutgers.edu[128.6.68.25]:25, delay=1, delays=0.39/0.01/0.52/0.08, dsn=5.7.1, status=bounced (host mx.eden.rutgers.edu[128.6.68.25] said: 550 5.7.1 Client host rejected: cannot find your hostname, [64.247.29.70] (in reply to RCPT TO command))

and:

maillog:Nov 15 14:29:09 Darkflame postfix/smtp[24890]: AE0E1466A: to=, relay=pamx1.hotmail.com[65.54.188.109]:25, delay=1.1, delays=0.27/0.01/0.75/0.11, dsn=5.0.0, status=bounced (host pamx1.hotmail.com[65.54.188.109] said: 550 SC-001 Mail rejected by Windows Live Hotmail for policy reasons. Reasons for rejection may be related to content with spam-like characteristics or IP/domain reputation problems. If you are not an email/network admin please contact your E-mail/Internet Service Provider for help. Email/network admins, please visit http://postmaster.live.com for email delivery information and support (in reply to MAIL FROM command))

Fair enough, I figure being a good sysadmin I should now contact the parties involved to see what the problem was. I sent two emails that night, one to the Rutgers helpdesk:

From: Ben Rose
To: helpdesk@nbcs.rutgers.edu, abuse@hotmail.com
Date: Sun, Oct 25, 2009 at 6:46 PM
Subject: emails being bounced

Greetings:

I own and administer the domain allmybase.com. I have several
constituents who are trying to send email to people on your campus.
They are getting their emails bounced with the messages outlined
below.

I figured I’d contact you since 64.247.29.70 does indeed resolve to
darkflame.allmybase.com and vice versa. Please let me know if there is
anything else I can help with.

Thanks!
–Benjamin Rose

Note, though, that that email had two recipients, Rutgers helpdesk and the abuse department at hotmail. Why abuse? Well, simply because in the actual error message in my maillog, the URL http://postmaster.live.com was given as a resource center. I checked this site, and it has absolutely NOTHING except some links to sell your Microsoft products. Awesome. The only email address I could find was, in fact, the one I sent to, which is actually supposed to be for reporting spam and account abuse. I figured something was better than nothing, though, so whatever. I was wrong, as I nearly instantly got a bounce:

From: MSN Hotmail
To: OMITTED
Date: Sun, Oct 25, 2009 at 7:30 PM
Subject: emails being bounced

Thank you for reporting spam to the Windows Live Hotmail Support Team. This is an auto-generated response to inform you that we have received your submission. Please note that you will not receive a reply if you respond directly to this message.

Unfortunately, in order to process your request, Windows Live Hotmail Support needs additional information to validate and confirm the abuse.

The easiest way to report spam to Hotmail is to click on the “Report Spam” or “Junk” button provided by your ISP. Hotmail has systems set up with most major ISPs so that when their users click on “Report Spam” or “Junk” buttons, we automatically receive a notification. Check the link below to find out if your ISP is included. If you cannot find your ISP in the list, please forward the spam or abusive mail to us as an attachment. Simply start a new mail message, attach the spam to that mail, and send it.

Windows Live Hotmail processes complaints received in the Abuse Reporting Format (ARF) format. ARF is the industry standard for reporting spam complaints. Using the ARF format helps us ensure that someone can only report complaints about mail actually generated by a Windows Live Hotmail user. A valid ARF formatted complaint is a message containing the entire original spam or abusive message (including all message headers) as an attachment. To learn more about ARF, review the draft RFC at:
http://www.mipassoc.org/arf/specs/draft-shafranovich-feedback-report-05.txt.

For additional information about reporting abuse to Windows Live Hotmail, please visit:
https://windowslivehelp.com/solutions/safety/archive/2009/03/23/how-to-report-abuse-or-spam-in-windows-live-hotmail.aspx

For more information on how to protect your email, please visit:

http://www.microsoft.com/protect/yourself/email/default.mspx.

For more information about Window Live Hotmail’s efforts and technologies used to fight spam and abusive e-mails please visit:

http://postmaster.live.com/FightingJunk.aspx

Oh how helpful, more links to products Microsoft would love to sell me, and a quote from an RFC for spam reporting. Lovely. Thank you Microsoft for the RFC reference, I know you’re all about open standards and compliance after all! Rutgers, on the other hand, could not have been MORE helpful:

From: NBCS Help Desk on line
To: Ben Rose
Date: Mon, Oct 26, 2009 at 2:10 PM
Subject: Re: emails being bounced

Ben:

We require that all incoming mail come from hosts that pass a few Internet standards tests. The error message below indicates that your’s is failing the first, a Reverse DNS lookup test. The core of the error reads:

mx.eden.rutgers.edu[128.6.68.25] said: 550 5.7.1 Client host rejected:
cannot find your hostname, [64.247.29.70] (in reply to RCPT TO
command))

When I do a UNIX host check on that IP address I get:

59 toolbox!koft> host 64.247.29.70
70.29.247.64.IN-ADDR.ARPA domain name pointer darkflame.allmybase.com

When I check that name I get:

60 toolbox!koft> host darkflame.allmybase.com
Host not found.

Your options are to send from a different, properly configured host or to get darkflame.allmybase.com pass a reverse DNS lookup test,

Dan

Oh, well looks like I messed up… here’s the rest of that email chain:

From: Benjamin Rose
To: NBCS Help Desk on line
Date: Mon, Oct 26, 2009 at 2:19 PM
Subject: Re: emails being bounced

Dan-

Hmmm okay this is pretty specific, I’ve only ever encountered the need
for reverse DNS to only return something, not the specific domain from
which the information is being sent.

Thank you for your help, I have updated my DNS and if this is indeed
the problem, it should be rectified once the changes propagate. You
can tentatively close this ticket as resolved, I’ll be in contact
again to confirm shortly.

In the meantime, are there any other tests your servers do that I
should be aware of? My server requires SSL, SASL auth, and does a
virus scan of the incoming and outgoing messages, so it should be good
on most tests.

Thanks for your time,
–Ben

From: NBCS Help Desk on line
To: Benjamin Rose
Date: Mon, Oct 26, 2009 at 3:11 PM
Subject: Re: emails being bounced

Ben:

The other main test is HELO/EHLO which requires a fully qualified domain name in communication. From what I can see in the below error message, I think that you should pass that test.

If you do not, please send email from the system that is failing that contains the error message you get to the address:

help@spam.rutgers.edu

That address is NOT requiring those tests to be successful (and hence gets MORE spam) and thus you should be able to reach it from sites that cannot reach this or other addresses.

Dan

I waited another day for DNS changed to propagate. Checking the next day whether or not I could send any email to users @eden.rutgers.edu, I was quite happy to find that the problem had been fixed. I reported the ticket as resolved and thanked Dan for his work in bringing my issue to a speedy resolve. I had hoped that this would also be the root cause to why Microsoft was blocking my outgoing emails. Wait, let’s actually examine that use of words there. BLOCKED. BOUNCED. Microsoft is not just filtering your incoming mail and maybe tagging some mail as spam, no, they are ACTIVELY making the choice for their users as to who gets to email them and who does not. What can this mean for you, the end user? Your best friend from high school trying to get back in touch with you? Could be blocked. Your company asking you some very important question? Kaput. For crying out loud, [Microsoft has actually once blocked all of email from Verizon users!] This to me seems like bad policy. I hoped that this issue may resolve itself in due time, so I gave it a couple of weeks. Two weeks or so later, my users were still unable to send email to the users on MSN live accounts (still includes hotmail and some companies/schools that rely on Microsoft for their email). I happened to casually mention something to my brother, [Jonathan Rose of Farious Net Solutions] over a lunchtime conversation. Turns out he’s having the same problems that I was with Microsoft, same EXACT error message, same UNHELPFUL link to a solution. What do our servers have in common? IP space. His mail server is 64.247.29.67, mine is 64.247.29.70. On a hunch, I nmap’d port 25 of 64.247.29.0/24 and found two more mail servers open for smtp connections in this IP space. Now, Jon only owns 64.247.29.64-127, and both these other mail servers were outside of this IP space. Running a blacklist check against our two mail servers, everything turned up squeaky clean, as expected. However, running a check against those other two IP’s turned out to be probably the root cause of this issue, they were on several different blacklists. It became clear to me why Microsoft should not be trusted with the decision as to what mail gets delivered to a user’s inbox and what doesn’t, as IT BLOCKED AN ENTIRE 24-BLOCK WHEN TWO IP’S IN IT, OWNED BY DIFFERENT PEOPLE THAN US, HAD BAD SENDER REPUTATIONS!

I was pissed at this point. I decided I would give Microsoft a call to discuss their policies and let them know just how stupid an action like this is. I checked Microsoft’s website for a phone number for MSN tech support. They couldn’t have hid it better, because I searched for a good while and didn’t even find one. Guess what number I could find, though? Microsoft sales, of course. Oh well, the journey of 1000 miles starts with a single footstep. So I called the sales department and pounded the 0 key until I got a person. He was the only polite person I’d encounter on this journey to an answer. Here’s the conversation’s transcript, word for word.

Him: Thank you for calling Microsoft, my name is Tom, how may I help you today?
Me: Hi, yeah, I’m in the totally wrong department. I have an email server that I administer that is having it’s outgoing email blocked by hotmail’s incoming servers, and I was wondering if you could transfer me to the tech support department that might cover something like that.
Him: I’d be happy to. I think that would fall under the category of MSN support, so I’ll transfer you over to them. Please hold.
Me: Thank you.

== 6 minutes later ==

Him: Hello and thank you for calling MSN tech support, can I please have your account number?
Me: Ummm I don’t actually have an account, I have an email server that I administer that is having it’s outgoing email blocked by hotmail’s incoming servers, and I was wondering if you could help me rectify the situation.
Him: What was your name again?
Me: Ben Rose
Him: Ok, and what is your account number?
Me: No, no, you’re not listening to me, I don’t have an account, I just need some SMTP support for outgoing emails.
Him: We can’t help you if you don’t have an account with us.
Me: So you want me to register for a free hotmail account before you’ll help me?
Him: No, this line is only for paid subscribers of the MSN Live services. I can’t help you if you don’t have a paid-for account.
Me: So to everyone else, you pretty much just say ‘screw you, not our problem?’
Him: No sir, we do our best to help.
Me: Ok, so please do your best to help.
Him: Well, we can’t help you if you don’t have an account. May I have your account number?
Me: OH. MY. GOD. Just transfer me to your manager.
Him: I’m sorry, I can’t do that.
Me: You can’t transfer me to your manager?
Him: Correct.
Me: You CAN’T, or you WON’T?
Him: I can’t.
Me: This is unbelievable. What do you do for people who don’t pay for an account, but still have issues with your services?
Him: Please visit microsoftlivehelp.com and check the knowledge base. You will find our knowledge base there. Have a good day, sir.
*click*

I’ve never encountered a more rude tech support person in my life. What kind of company policy prevents a support person from transferring a person to their manager unless they have a paid account? It doesn’t even make any sense! Ok well I have another lead, I’ll have to check out that site the guy just mentioned to me. I visited it, and there was a search field in the corner. Alright, I put the error message from the maillog into the box and hit enter. Results actually came back from other users, at least 50, with the same problem. [They reported it on a forum and a Windows Live rep responded saying he’d file a ticket for these people himself.] This isn’t exactly optimal, but I suppose it will do. I needed to sign up for an account on Windows Live before I could post a message asking for help, though, so I guess it comes full circle. I set up an alias on my server so as to be able to drop the email address immediately after this problem was fixed, and registered for an account. What happened next even I couldn’t have forseen. The site, when attempting to log me in, went into an infinite redirect loop between the forum and the login page, each time with no difference and no pause. Well, I am on Firefox on Linux, and this is a Microsoft website/product… so I borrowed my rommate’s laptop and fired up internet explorer. SAME. EXACT. PROBLEM. Infinite redirects. On the support website of a multi-billion dollar company. Using their browser. HOW?!?

I decided I’d just forget getting help from Microsoft at this point, I was to the end of my rope. I fired up the amazing google machine to see what the user community had to say about this error code. After about a half hour of googling, I came across this link: [Sender Information for Hotmail Delivery.] Even the person who posted the link on the 3rd party forum said he doesn’t believe this form is actually linked to anywhere on the Microsoft websites. How this was supposed to be found, I have no idea, but it looked like I might finally get some help. I filled out the form as completely as possible, making sure to give all the requested information lest my cries fall on deaf ears. I filled out the CAPTCHA at the bottom and hit submit. And then I waited. And waited. Annnnnd waited. Then I got the most amazing page ever: “The ticketing system is currently unavailable, please try again later.” WHAT?!? ARE YOU KIDDING ME?!? Of course, pushing the back button in the browser resulted in all of the form being reset, so I’d have to fill in all the information again. Defeated, I retired to bed for the night.

The next day, I filled out the form again and hit submit. I waited and I waited, and eventually I GOT CONFIRMATION OF A TICKET BEING FILED SUCCESSFULLY!!! Oh happy days!!! But then the best was, of course, yet to come in the email chain that followed. Below is the email chain with minimal interjection posted between emails, just so you get the full effect of my frustrations.

From: Microsoft Customer Support
To: OMITTED
Date: Fri, Nov 13, 2009 at 4:03 AM
Subject: RE: OMITTED – Sender Information for Hotmail Delivery

Hello Benjamin,

The deliverability issues on IP(s) (64.247.29.70, 64.247.29.67) were based on negative filter verdicts or other IP reputation issues that caused some (or) all of your mail to be deleted and/or potentially blocked.

In order to further investigate your concern, please create SPF records for your domain and enroll your IPs to the Junk Mail Reporting Program.

Create SPF record: http://www.microsoft.com/senderid/wizard

Junk Mail Reporting Program: https://support.msn.com/eform.aspx?productKey=edfsjmrpp&ct=eformts

Thank you,
Wilson
Windows Live Hotmail Domain Support

From: Benjamin Rose
To: Microsoft Customer Support
Date: Fri, Nov 13, 2009 at 7:54 AM
Subject: Re: OMITTED – Sender Information for Hotmail Delivery

To whom it may concern:

My IPs are on no blacklists and are blocked SOLELY by your company.

I also heard the other day your service has done fun things like block
all email from Verizon, and less recently, all email from yahoo. How
is this a good idea?

I looked into it, and it seems 64.247.29.2 and 64.247.29.164 have IP
reputation problems. It wouldn’t surprise me if your service simply
blocked all of 64.247.29.*, even though I own and administer
64.247.29.64-127 and none of these IPs have reputation problems.

Please don’t make me jump through hoops just to have mail work for
your service and your service alone. I have over 100 domain names on
this IP space, and email worked before about a week ago to users on
your service. Please just whitelist my IP space again and we can just
go back to not having any problems.

If this is impossible for you to do, please escalate this ticket to
someone who can.

Thank you,
–Benjamin Rose

From: Microsoft Customer Support
To: OMITTED
Date: Fri, Nov 13, 2009 at 11:46 AM
Subject: OMITTED – Sender Information for Hotmail Delivery

Hi,
Thank you for contacting Windows Live Hotmail Domain Support. My name is Manny and I will be glad to help you.

Your IP 64.247.29.70 and 64.247.29.67 were blocked by Windows Live Hotmail because the majority of all the email that you send to Hotmail has been judged to be spam by Windows Live Hotmail’s internal filtering system. We have conducted an investigation into the emails originating from your IP space and we have implemented a fix for your deliverability problem. This fix may take 24 – 48 hours to replicate completely throughout our system.

Please note that lifting the block does not guarantee that your email will be delivered to a user’s inbox. However, enrollment in our JMR program and having your IPs registered with Sender ID will help with your mail delivery to your recipient’s inbox, thereby improving your IP’s reputation as well.

. Please ensure that you have published SPF records for your sending domains and register with Sender ID. You can find additional information and submit your domain for inclusion into the Sender ID program at http://www.microsoft.com/senderID. Please note that technical standards (RFC 4408) discourage use of “ptr” for performance and reliability reasons.

. Monitor user complaints. Hotmail also has a sender complaint feedback loop program called the Junk Email Reporting Program (JMRP). Enrollment in this free program will benefit you as a sender as it will keep your email lists updated and populated with interested Windows Live Hotmail Customers. This program will help you to remove those Windows Live Hotmail Customers who do not want to receive emails from your company. If you are interested in joining this program, please visit https://support.msn.com/eform.aspx?productKey=edfsjmrpp&ct=eformts

. Hotmail has created the Smart Network Data Services program. This is a service that helps legitimate email senders work with their customers and partners to reduce spam originating from their IP. To register, please go to http://postmaster.msn.com/snds/. This program allows a sender to monitor the ‘health’ of their IPs.

While using the SNDS tool, enrollment in the JMRP or having your IPs registered with Sender ID will not allow emails from your mail servers to bypass our filters, these are in place to help legitimate companies deliver their emails to Hotmail Customers.

. SenderScore Certified Mail Program. Many legitimate mailers and marketers have qualified and joined this “white listing” program to improve mail deliverability and decrease email from being filtered to the Junk E-mail Folder. Sender Score is a third party program administered by Return Path. Sender Score (www.senderscorecertified.com) is the only white listing service to which we subscribe.

The troubleshooting steps in this email are recommendations only. Microsoft makes no guarantees that following these steps will guarantee deliverability to MSN, Windows Live Hotmail, or Live.com customers.

Thank you,
Manny
Windows Live Hotmail Domain Support

Wait, did you read that? Different tech support rep, and check out that part in bold. But, it looks like he lifted my block. I waited 3 days for the issue to go away, which worked out perfectly since that was actually my 21st birthday. Turns out, though, that the issue never went away…

from Benjamin Rose
To: Microsoft Customer Support
Date: Tue, Nov 17, 2009 at 1:21 PM
Subject: Re: OMITTED – Sender Information for Hotmail Delivery

Manny:

This issue is still not resolved. Neither 64.247.29.67 nor
64.247.29.70 can send emails to users on your system. Specifically, I
have been testing sending email to an account @mail.usp.edu which has
their email service through you.

Please take another look into the issue and see why this is occurring.
It has been well over the 48-hour grace period you gave me, and I am
still not seeing any results.

Thanks,
–Ben

Not even sure why I’m still saying please or thank you anymore at this point…

From: Microsoft Customer Support
To: OMITTED
Date: Tue, Nov 17, 2009 at 7:55 PM
Subject: RE: OMITTED – Sender Information for Hotmail Delivery

Hello Ben,

This is Christine with Windows Live Hotmail Domain Support. We appreciate your patience while we are investigating your deliverability issue

I can see that we lifted the block where your IP was previously listed. However, one of our filters is actively blocking your messages because of its poor reputation within our system.

Please note that lifting the block does not guarantee that your email will be delivered to a user’s inbox. However, enrollment in our JMR program and having your IPs registered with Sender ID will help with your mail delivery to your recipient’s inbox, thereby improving your IP’s reputation as well.

· Please ensure that you have published SPF records for your sending domains and register with Sender ID. You can find additional information and submit your domain for inclusion into the Sender ID program at http://www.microsoft.com/senderID. Please note that technical standards (RFC 4408) discourage use of “ptr” for performance and reliability reasons.

· Monitor user complaints. Hotmail also has a sender complaint feedback loop program called the Junk Email Reporting Program (JMRP). Enrollment in this free program will benefit you as a sender as it will keep your email lists updated and populated with interested Windows Live Hotmail Customers. This program will help you to remove those Windows Live Hotmail Customers who do not want to receive emails from your company. If you are interested in joining this program, please visit https://support.msn.com/eform.aspx?productKey=edfsjmrpp&ct=eformts

After you have taken steps to enroll in the JMRP and Sender ID, please contact us again and we will further investigate the issue. Please include the SRX number you were given when beginning your JMRP enrollment.

Best regards,
Christine C.
Windows Live Hotmail Domain

Time to get angry. I looked up Microsoft on the better business bureau and got a phone number, and got ready to make as much noise as a pebble can make against the ocean that is Microsoft…

From: Benjamin Rose
To: Microsoft Customer Support
Date: Tue, Nov 17, 2009 at 9:10 PM
Subject: Re: OMITTED – Sender Information for Hotmail Delivery

Christine:

I guess Manny and Wilson were unable to resolve my issue, I assume you
are a higher level tech than they are and that you will be able to
resolve my issue.

Here’s what I am telling you:
1) 64.247.29.67 and 64.247.29.70 need to be able to send email to
users on your system, be it hotmail or any other live service.
2) I actively refuse to enroll in any microsoft programs. I will not,
and in fact CANNOT, modify DNS for all domains hosted on these
servers.

Here is what I am asking you:
1) Please, if this hasn’t already been done, remove me from any and
all blacklists.
2) Please reset the reputations for these two IP addresses and any/all
domain names registered thereto.
3) Please ensure that you don’t block these two IPs, or the IPs of
other MSP’s such as Verizon and Yahoo erroneously again.

You are costing me and my clients TIME AND MONEY! Sales have now
officially been lost because of this deliverability problem, one that
only exists with your service. I actively refuse to do any work for
you and your service that has caused me nothing but problems. I just
want your system to reset my IP’s reputations back to whatever they
were before you started this campaign against working email systems.

I hate to be one to resort to angry emails and threats, but I refuse
to do any work for you when you are causing me headaches. I will not
let this ticket close for as long as I have deliverability problems
with your service. If this ticket gets closed without my consent or
knowledge, I will continue filing tickets with your system until such
time as I am happy with your work in eliminating this bug in your
system. I will be forced to report you to the better business bureau
and go through all proper channels in Microsoft to file a complaint
with your department, several complaints if I must, I have the phone
number for one Ms. Kathy Cole sitting right here.

I just want you to know, I want this issue fixed in a reasonable
timeperiod, I’ll give you 48 hours, and I’m really not kidding around
about this issue. Please just stop with the canned responses and
telling me to do additional work. If this is not possible for you,
please escalate this ticket even higher until it comes across the eyes
of someone who can do what I am asking.

Thanks,
–Benjamin Rose

Threats, my dear Watson. That’s what it takes to get a response from Microsoft.

From: Microsoft Customer Support
To: OMITTED
Date: Tue, Nov 17, 2009 at 10:56 PM
Subject: RE: OMITTED – Sender Information for Hotmail Delivery

Hello Benjamin,

My name is Arc; I work on the Windows Live Hotmail Sender Support Team, helping to support Hotmail’s anti-spam efforts. I apologize for the delay in responding to your email. We have received a large number of support requests lately.

I understand the importance of you being able to send email to hotmail on the affected IP’s. This is why we are working towards a solution to your delivery issue. Please understand that we have guidelines to follow, and to lift the block on your IP’s, we need you enroll in our programs. We need you to participate in JMRP and SenderID. After which, we will be implementing a fix and mitigation.

Thank you,
Arc,
Windows Live Hotmail Sender Support Team

These people are just NOT getting it. Of course, all the while, checks of sendability come back negatively…

From: Benjamin Rose
To: Microsoft Customer Support
Date: Tue, Nov 17, 2009 at 11:48 PM
Subject: Re: OMITTED – Sender Information for Hotmail Delivery

Arc:

I have well over 50 domain names on this IP space. If you’d like to
come over to my place and configure SPF records for all these domain
names individually, please let me know.

If not, please escalate this ticket to a higher tech, maybe one who
can actually solve my problem this time. I figure if I keep requesting
this, eventually it’ll happen.

–Benjamin Rose

The kicker… note that this one actually got sent pretty much 48 hours after my threatening email saying they have 48 hours to fix the problem….

From: Microsoft Customer Support
To: OMITTED
Date: Thu, Nov 19, 2009 at 11:46 PM
Subject: RE: OMITTED – Sender Information for Hotmail Delivery

Hello,
Thank you for writing us back.

My name is Marianne and I will be assisting you today.

Our history shows that we blocked your IP (64.247.29.70 and 64.247.29.67) in the past; however, we do not have a record of any active blocks against your IP. Please confirm you are still receiving the error – 550 “Blocked due to policy reasons.”

I hope that the information that I have provided to you has been helpful. You may also be able to find additional information on common delivery questions at the Hotmail Postmaster Site found at http://postmaster.msn.com/.

Best Regards,
Marianne
Windows Live Hotmail Domain Support

IT WORKS!! Users can now send email as they wish. Almost surprising really. The bad news is, a vast majority of your email may get tagged as spam when sending to Microsoft Live users. You should encourage them to switch. If you actually made it to this point in reading, I know you’re thinking exactly the same thing I am… I hope Dan from Rutgers gets a very lucrative job offer from Microsoft email tech support and decides to take the job…

Thanks for reading.

Suppose your friend wanted to install a video camera on your car. The video camera would only be able to see the treads of your tires as part of a study. He could only tell from this feed the depth of the treads of your tires. Nothing more, nothing less. Would you be particularly offended by this? Odds are probably not, this information is pretty benign. But what if this was a complete stranger who wanted to put the camera on your car? You’d probably be very wary of it, and inspect the camera to make sure it did exactly what he said it does. Fair enough, as the information is still pretty benign and he was pretty honest about it to begin with.

Now consider you walked outside, and one day found a video camera watching your tires just sitting there. You’ve had the car for years, you don’t know where the information is going, and you have no idea how the camera got there. And to boot, it looks like someone attempted to hide the camera — poorly. Perhaps by covering it with some clear tape or something. You’d be pretty pissed that someone did this, right?

I’ve been having issues from time to time with this blog. Randomly, the sidebars on the right load very slowly as compared to the rest of the document. I got really curious as to why the other day when it was happening again, and so I decided to figure it out. I had a suspicion that it was the mysql database, but I wanted to make sure. So I started an strace on the http daemon, and refreshed the page. I really was quite pissed at what I saw next. There was an outgoing http connection I’d never seen to a website I’d never heard of before. The connection was made moments after a gigantic glob of data was read from a php file on the filesystem. It started with eval(gzinflate(string_rot13(base64_decode(…………)))) and at this point, I knew I was in trouble. It was hidden code I wasn’t supposed to see running on my website.

See, when I first got this blog, I started by finding a nice theme. I did eventually come across the one you see now, and I liked it. A tad bit land-of-the-rising-sunny for me, but whatever, it did look nice. So I installed it. There were some plugs down the bottom of the page, one my friend even asked me about. He said, “dude, what’s with the plug for Burt’s Bee’s?”. I said, “I dunno, but the author of the document put it there, and he did ask that I not remove it, so I left it.” I swear I did the honest thing!

But then I see the camera pointing at my tires that I’d never authorized. The outgoing HTTP connection, that is. I quickly edited /etc/hosts and changed the hostname it was looking for to address 127.0.0.1. I refreshed the page. What happened? The footer and all the plugs (Burt’s Bees included) disappeared. The outgoing connection was actually quite legit. I wrote a quick php script myself to parse the output of the several layers of evaluations of random blocks of data. SEVENTY-ONE EVALUATIONS LATER, I attained the source code. It is a large user-agent tracking system. I’ve left the source code [right here].

So, I hate to inform you, but if you’ve visited this site in the past month or so, you have had some of your more public information read in by some stranger on the internet. No worries, the information was completely benign, simple stuff like your browser version and operating system type, hence the reference to tire tread. It’s stuff that I really wouldn’t have cared too much about the author collecting… if only he had just ASKED!. So now I found out that the entire footer of the page was actually coming from his server, which could mean simply one thing – Burt’s Bees is PAYING for this kind of shady advertising. That’s a horrible business practice!

In response, I’ve removed the offending code and published it in the link above. I’ve also blackhole routed any traffic from the offending website. The name of the theme, by the way, is SoulVision, and I actually did get it from a reputable repository of quality wordpress themes. As someone in the security field, I recognize that this could have been much worse than it was, and I realize just how lucky I am that it wasn’t. I guess my moral for the day is to always verify code that’s going into production on a locked down server. Oh, that, and I’ll probably never buy Burt’s Bees. Ever. End ‘o discussion on ethics.

So maybe the title of this article is really a bit overkill. Or maybe It’s true. Maybe you should decide….

See, Bruce runs the fancy email newsletter known as “Cryptogram”. It’s actually a very good newsletter, so props to Bruce for that. It’s really just a monthly conglomeration of stories that relate to security, be it software, hardware, social, or otherwise. I have enjoyed the newsletter thoroughly for a while now, so I’m not just complaining about something after my first experience with it.

And congress, well, they seem to run the country. Which is funny, but we’ll get to that later.

Bruce made an interesting claim in his email newsletter a little bit ago – that the Chinese Government was developing some sort of secret and powerful operating system hellbent on attacking the critical computing infrastructure of the United States of America. To be fair, however, he did put this bit of text in quotation marks. Apparently this is not his doing. The backstory seems to follow that a briefing was given to congress not too long ago about this very subject. Apparently it’s called “Kylin”, and as a member of the IT community, we’re supposed to fear it.

Well lo and behold, the other day at work, I was setting up a snort Intrusion Detection System, and what did I see? Some very weird packets bouncing to-and-fro around the network. I mean _weird_ packets. Combinations of SYN/ACK/RST/FIN flags, sometimes there was an URG/PSH flag set just to throw off any sensors that might be listening. Fortunately, I log ALL packets to pcap data files, so I fired up tcpdump and extracted the relevant data. Insert this data into wireshark for some nice analysis, and the weirdest patterns were showing up. Scans like I really have not seen ever, but ones that make logical sense. Some ACK packets were being sent across the wire in the hopes to solicit an RST packet from a listening port. There were just strange packets everywhere, highly anomalous packets, some were even SYN/ACK/FIN packets. Not quite a christmas tree, but something that will never occur in nature. So I did what I could, and traced the packets back to their source. The funny thing was, they were bouncing all around the world. Africa, Asia, South America — nearly every country was seemingly scanning the network. And as quick as it started, it was over. I realized what was going on… A massive distributed reconnaissance mission just took place on our network. Combinations of weird packets and different sources all came together in harmony to do a full 65536-port scan on the network, and almost all without sending off a single alert from snort. Properly collated, this raw data could be put into a map of every open service on the network, and if it weren’t for a single seemingly benign alert, I would have not even known something just happened. The most chilling thing is, the possibility certainly exists that this has happened before and will happen again, but I just happened to be lucky enough to catch it this time as it happened.

I needed a lead. This was too strange. So I started analysing the packets that were sent across the wire. The ICMP packets looked very familiar, and upon further investigation, I noticed that these ICMP packets were somewhat verbatim from the FreeBSD kernel. The actual data portion of the packets vary from OS to OS… don’t believe me, then look it up! Suddenly I noticed that some packets were coming from China. I don’t know why I noticed this, but it just seemed like a pertinent detail. I think it was my subconscious telling me exactly what I feared – I was facing this “new and secret” Chinese operating system. I google’d “Kylin” and, rather unsurprisingly, the first page of results were all the same fear-mongering based on the briefing that was given to congress. But what if I changed the search a little bit maybe someone has seen this before? In this light, I google’d “Kylin FreeBSD”.

WELL WHAT HAVE WE HERE?

Kylin is not some secret Chinese government project to destroy America. It’s not even a secret. It’s a state-sponsored project to secure a common open source product. It’s absolutely no different than the US NSA’s SELinux. For crying out loud, YOU CAN DOWNLOAD THE ISO’S! It’s almost disappointing – I was secretly hoping deep down inside that I was the first person to encounter Kylin’s doings first-hand. Not so much. So I gave Bruce’s newsletter a once-over again. Looks like I was wrong all along, Bruce didn’t believe it either. He even made the conjecture that it was just over-hyped nonsense. But why even write about it as if it COULD be legit if a simple google search shows it’s all just overhyped garbage reporting? Like I said, Bruce needs to learn to use the Interwebs a little better!

And as for congress, well, they do too. I’m tired of the fear-mongerting that occurs so very, very often. I’m tired of the irresponsibility of it all. What does it accomplish? Maybe it pushes your own agenda a little faster? Why not do the right thing and report the facts, then maybe we can have some time to plan things out and do it right the first time. I’m all for a national Cybersecurity initiative, in both the public and private sectors. It’s truly a great thing that this is happening. I would just really hate to see it all go as wasted effort because proper time and resources were not allocated ahead to keep the project alive. Perhaps I’m just an idealist this way, but it’s the only way that (IMHO) works.

Now, in the light of sharing and freedom, here’s the Kylin ISO’s. Use responsibly, and above all else, use them to learn! This is no different than SELinux, and maybe the Chinese government will continue to sponsor open source projects like this, something from which we surely can ALL benefit. And don’t believe everything you read, I know I’ve learned my lesson this time around…

And as for the scans I was seeing? Who knows, it could have been one person all along, or it could have just been coincidence. It’s really hard to say, but that’s the nature of the beast we call the Internet. Surely, nothing that was done is out of the reach of some clever SSH’ing and NMapping. If there’s more to the story, I’ll make an update. But for now, I’m content with the facts that I have, no more, no less.

DOWNLOAD LINKS:
DISC ONE: [KYLIN-2.1-1A.iso]
DISC TWO: [KYLIN-2.1-1B.iso]
MD5SUMS: [KYLIN-checksum-2.1-1.md5]