Status

walrus

James was taken offline last week by my hosts, BlueHost, because something was spamming my database of posts incredibly rapidly with incredibly demanding data requests. It was using up so much CPU on the server that it was slowing down every other site it hosts.

I was in New York meeting a nervous walrus at the time, so I couldn’t do much about it. Now that I’m back I’ve looked into it, done some maintenance, taken some precautions, and asked them to put it back online. The upshot is that it seems to be fixed for the time being, but I’m going to have to keep a very close eye on it for a while. If it goes down again, I’ll post updates on Twitter here.

If you’re interested in the technical specifics, here’s what I found:

  • A few lines of malicious code have found their way back into my source code, despite my having changed my password and kept up to date with the latest WordPress updates since that last happened. I’ve removed them and changed my password again, to something ridiculous. Not sure if there’s anything I can do beyond that.
  • One visitor, presumably robotic, immediately started loading the same page dozens of times a second within a few minutes of James going back up. I’ve banned its IP, but I don’t know how to prevent that kind of thing in general, or why it would cause slow SQL database queries.
  • Something was accessing parts of the database that aren’t supposed to be public. After careful investigation I discovered that, on this occasion, it was myself: when I accessed the database directly to do some maintenance, it was taking 6 seconds or so just to display or select the tables. Further back, there are still calls to these private tables that I can’t explain. I choose to ignore both these problems.

Thanks for sticking around.

21 Replies to “Status”

  1. Cute picture! Man, dicks on the internet are so not in short supply. We should all write programs to scour every corner of this website for dicks! All day, every day!

  2. @ZomBuster
    I think it was the Eggman personally.

    Amusingly I came across about a dozen pieces published in the week or so you were offline, all linking to articles from here.

    Good to see your blog back Tom. Thanks for still taking the time to write interesting things for no real fiscal return on here.

  3. Tom, maybe it’s not WordPress. Maybe there’s a hole in some of the other files you have on the server, or maybe there’s a hole in BlueHost’s server. I wouldn’t be surprised, why the hell are you signed up with them anyway?

  4. @Iain: He’s definately aiding said lolrus. Otherwise why would the walrus be nervous? It would be Tom who would be terrified. I mean, isn’t several hundred (or whatever) pounds of blubber with GIANT TUSKS not the scariest thing in the history of forever when you are between it and it’s bucket?

    Also, what noise does a Walrus make anyway?

  5. Dangit. *If he were a suspect it would be Tom who would be terrified.

    Also curse my need to be coherent in my ramblings!

  6. Good job finding that bad code. I’ve had stuff like this happen to me several times before, and if I didn’t know a really competent and nice web developer, I would never be able to fix it when it happens. Who are these tossers, anyway?

    Looking forward to more posts out of you.

  7. I know, we make the server private to anyone who hasn’t personally been vetted by Tom himself. We send him a request and he comes round for a cup of Tea and certifies our profiles. Perhaps even biscuits and a scone.

  8. Try blocking the source ip subnet, not just the host. Find th subnet with a bgp looking glass. Might cause some collateral damage, but if the attacker has a dynamic ip from an ISP blocking one ip won’t work. That’s assuming they are attacking from their own connection…

  9. Glad to see the site back up. Don’t know if you got my comments on Twitter.

    As you’re running WordPress you really want to get yourself using the WP Super Cache plugin, it makes a huge difference to the CPU time and reducing the amount of SQL queries.

    Also, investigate possibly changing your web server (if your host allows it) to nginx or lighttpd. Both perform a lot better when serving the static content which WP Super Cache provides than Apache does.

    I assume you’ve always kept WordPress fully up-to-date? If not, do so and if so then considering using an .htaccess file to put another password on your /wp-admin/ folder, this helps prevent some of the bots which crawl the web looking for WordPress sites to attack.

    Hope this is of some use. If you want a hand with any of it then contact me and I’ll give you a hand. My contact details are on my website.

  10. Cheers Mike. I did see your Tweet, but thanks for reminding me now I’m back and able to follow up on it. Installed Supercache, don’t completely understand the options but I put it fully on and unchecked the thing for ‘VERY busy sites’.

    I’ll try that IP subnet mask banning thing if the problem recurs – so far, since the measures I took yesterday, there hasn’t been a single slow SQL query. Compare that with about fifty in the first five minutes of the site going back online, or the one day last week when there were so many that the log files of when they occured came to over 100MB of raw text. I may be in the clear.

  11. The problem with the subnet mask banning is if, for instance, someone in the same neighbourhood tried to hack your site then everyone on the same ISP in the same block will get blocked too.

    The Supercache options are fairly good by default as long as they are set to “on”. It basically means there won’t be any PHP run or database queries done.

    Looking at the footer of the page it looks like you might not have set up the .htaccess stuff properly yet. I could be wrong though.

    Hope you’re alright now though!

Comments are closed.