The Great Malware Crisis of 2013 – As promised, here is the first of three journal entries to come before the end of this month. It is something I should have written a long time ago, and something that I did in fact start to write about a month ago. But I got bogged down and lost interest, and it is only through a supreme act of will that I managed to finish this. I can’t promise that it will necessarily be enjoyable to read, but it is something I had to do. Think of this as the part of dinner that you hate but have to get through to get to the tasty stuff.
This all began a shade over two months ago, when I received an email from Hahna, a reader and frequent commenter over at the Big Hominid’s place. She told me that Firefox on her Mac was blocking my site and giving her a malware warning. I was busy at the time, prepping for classes, so I wrote back and thanked her for the heads-up. I also mentioned that Firefox has always been a little paranoid in my experience (it was one of the reasons I stopped using the browser).
I did not think anything of it until the next day, when I tried to visit my site in Chrome and saw the dreaded malware warning. I began to feel sick in the pit of my stomach, and I started to look into the various links and resources provided by Google to combat malware attacks. Google suggested using Webmaster Tools to diagnose the problem, so I signed up for that and was able to get my site validated. Then I went into the malware section and saw the list of pages that Google claimed were infected—or, more specifically, had code injection issues. I checked all of the pages on my server and found that they were clean—there was no injected code. I then used the “Fetch as Google” service, which allows you to pull up the page as it is seen by Google’s tiny little spider-bots (which I have always envisioned as being like the spiders in Minority Report), but there as well the pages were all clean. Finding no problem, I submitted a request to Google for a review of the site, thinking that maybe they had gotten something wrong.
The next day I checked the results of the review: Google was claiming that my site was still infected. I was getting a little frustrated at this point; I wanted to solve the problem, but you can’t solve a problem until you can identify it. So I rechecked the pages that Google claimed were infected and once again found nothing. I went through every single folder on my server to make sure that there were no files that I hadn’t put there. Then I ran a PHP script that searched the text of every single file on the server for injected code. I came up with nothing.
I poked around on some sites dedicated to fighting malware, and on one site I posted a thread describing my problem. One of the users there replied saying that he (of course I’m assuming here, perhaps incorrectly) had pulled up one of the files in question (that is, one of the files Google said was infected) and had indeed found an iframe embedded in it. But the second time he pulled up the same file, the iframe was gone. He suggested that it was being inserted through .htaccess and possibly being “hidden by cookies.” I never did figure out what that last part meant, but I double-checked my .htaccess files to make sure that there was no suspicious code. (An .htaccess file, by the way, allows you to tell the server what to do, to a certain extent; so you could, for example, rewrite URLs or block certain bots and spiders from accessing your site.) I was not terribly surprised to find nothing suspicious, but I wasn’t comforted. Someone had indeed seen an instance of infection, but it could not be reproduced. To say that I was frustrated would be an understatement. Not knowing what to do, I submitted another request for review to Google. It was ignored.
I should mention that all of this happened over the course of several days, and at a time when I was very busy and really couldn’t focus all my attention on fixing the problem. It was extremely stressful, and I tried not to think about it too much, just doing what little I could when I could and hoping for the best. It soon became apparent that “hoping for the best,” though, was not going to be enough. One of the first things you are supposed to do when you are suffering from a malware attack is to contact your host, as they might either have some useful information or be able to help you with the problem. This was one of the first things that I did, firing off an email to Michal, the guy who runs my hosting company, Cornerhost.
I had been with Cornerhost for quite a few years at that point, and although there had been some rough patches, I had never really had any problems that couldn’t be fixed. Then again, Liminality is not a very demanding site—it’s not a business, nor is it a professional site, so I’ve never been worried about 100% uptime. If it goes down every once a while for a bit, no real harm done. Unbeknownst to me, though, about a year before the malware attack there had been a big panic going around regarding Cornerhost. Apparently the DNS records were going to expire for sabren.com (this is the domain for the Cornerhost DNS servers), and someone went around telling Cornerhost users that they were about to lose their hosting. I never got that memo, probably because I’m not a big enough fish to warrant much attention. But there was an article about it on Gizmodo, which in turn inspired a response from Michal on the Cornerhost blog. In this response he says that “Cornerhost isn’t going anywhere, and neither am I.” He also mentions that he has “been doing a really poor job of replying to emails lately,” but he promises that “it is going to change.”
Flash back with me now to the year 2013. As soon as I figured out what was going on with my site, I sent Michal an email. A day passed with no reply. I sent him another. Still no reply. I even tried sending him a few messages through the official Cornerhost website. I don’t know how many messages I sent in total, but when I submitted my second request for review to Google, I had still not received a reply. (In fact, to this day I have not received a single reply to any of the emails or messages I sent.)
I began to suspect that the problem might not be with Liminality at all, but without hearing anything from my host I had no way to prove that. Well, that’s not entirely true. I did plan out a series of steps that would allow me to narrow down the source of the problem. First I would wipe out every file I had on the server and then restore everything from backup. If that didn’t solve the problem, I would then wipe everything out and replace it with a single index file with nothing but a line of text. If the site was still flagged as being infected after all that, I would know that the problem was not with Liminality but with my host.
The problem, of course, was that doing this would take time, and I did not have much free time at all. And the more I thought about it—and the more deafening the silence from my host grew—the more I realized that those steps I had planned would most likely be an exercise in futility. I was already pretty sure that the problem lay with my host; in fact, I would say I was about 90% sure at the time. Even if I went through all those steps and boosted that figure to 100%, how would it get me any closer to solving the problem? The answer is, of course, that it wouldn’t. The only logical option was to abandon the Cornerhost ship, which appeared to be sinking—if it hadn’t sunk already.
I can’t say that I was sad to leave Cornerhost. Like I said above, the company (Michal, really, as it’s a one-man gig) served me well, and at one point I was happy enough with the service to buy a lifetime hosting plan for a one-time payment. This means that it wasn’t costing me anything to stay with Cornerhost, but at the same time my lifetime plan had long since paid for itself, so technically I wouldn’t really be losing anything by leaving, either. But it niggled, of course. In a post made on the Cornerhost blog in early August of last year, Michal wrote that after his February post saying that Cornerhost wasn’t going anywhere, he “plain stopped caring about cornerhost.” He went on to say: “I kind of decided that my customers were all grown ups, and if they weren’t getting what they needed here, then they knew how to take their business somewhere else.” True enough, but I was a lifetime customer, and when I made my one-time payment it was made with the understanding that I would receive a certain level of service as long as Cornerhost was in business. I suppose that, for Michal, me leaving is actually a good thing for Cornerhost, since I wasn’t making payments and was thus a drain on the system. For me, though, the idea of having to leave what was essentially free hosting to pay for hosting on a regular basis was somewhat of a bitter pill.
Of course, the equation is not complete. If you add in the fact that Liminality was likely to remain blocked by Google for as long as I stayed with Cornerhost, suddenly the math becomes easier to do: stay with Cornerhost and watch as Liminality is slowly strangled to death, or pay a small amount of money each month to see Liminality come back to life. In the end, it was an easy decision to make. In what is, at the time of writing, the last post on the Cornerhost blog, dated 17th August 2012, Michal wrote: “Even though my business is in shambles at the moment, the service is terrible and I’m working for free, every one of those lifetime accounts I sold is still active.” Well, not anymore—my account is effectively dead.
Once I made the decision to abandon the Cornerhost ship, I had to decide which of the myriad ships out there on the hosting seas I would board. I ended up choosing GoDaddy, having heard good things about them from my brother Brian (he uses them for his custom candles website) and some friends online. They happened to be having a special sale, so I went for the top-end hosting option; among other things, it listed a “malware scanner” as one of its features. Setting up the account itself wasn’t too much of a problem, and moving a backup of my site onto the GoDaddy servers was simply a matter of waiting for files to upload, but then I had to figure out how to reconfigure the name servers. This did not turn out to be that big of a deal, but you have to remember that it’s been ten years since I last did this, so I was a bit rusty. After contacting GoDaddy’s support staff (who were, as Brian promised, very timely with their response), I got that straightened out and Liminality was back online.
Well, sort of—even though everything was uploaded to the servers, the site refused to load, giving me a strange server error (I’ve since forgotten exactly what it was, but I think it was in the 500s). When I contacted GoDaddy’s support staff again, the guy who replied suggested that I take down my .htaccess file and see if that worked. I can’t do that, of course, because Liminality relies on the .htaccess file to redirect URL queries to a template that parses the URL and then serves up the requested content. When the support guy pointed out the .htaccess file as the culprit, though, I immediately realized what the problem was: it wasn’t the .htaccess file, it was the code I was using to parse the URL in my template. It was a bit silly because I knew the code was deprecated, but Michal had had a workaround in place that let me keep the deprecated code. The fix was simple and quick, though, and in a matter of minutes Liminality was truly back online. (With the exception of the contact page, which is also down because of deprecated code; I hope to get around to fixing this soon, but in the meantime there is an email address there if you don’t already have one for me and want to contact me.)
The malware scanner, by the way, turned out to be somewhat disappointing—it simply relies on Google’s own malware detection, so as soon as I set up my account and turned on the malware scanner, it started sending me daily warnings. It wasn’t until I uploaded the site to the GoDaddy servers and reconfigured the name servers that the warnings stopped. I suppose it’s not completely useless, though—if this ever happens again in the future, I will know right away and be able to deal with it a lot more quickly.
So there you have it. I can’t believe how long it took me to write this—not because it was particularly difficult, but after a while I just really did not want to relive the experience. Even just reading this over more time to tidy things up made me feel a little sick to my stomach. Now that this is out of the way, though, I can finally and truly put the whole mess behind me. At least now you know what happened.