Status

 

walrus

James was taken offline last week by my hosts, BlueHost, because something was spamming my database of posts incredibly rapidly with incredibly demanding data requests. It was using up so much CPU on the server that it was slowing down every other site it hosts.

I was in New York meeting a nervous walrus at the time, so I couldn’t do much about it. Now that I’m back I’ve looked into it, done some maintenance, taken some precautions, and asked them to put it back online. The upshot is that it seems to be fixed for the time being, but I’m going to have to keep a very close eye on it for a while. If it goes down again, I’ll post updates on Twitter here.

If you’re interested in the technical specifics, here’s what I found:

  • A few lines of malicious code have found their way back into my source code, despite my having changed my password and kept up to date with the latest WordPress updates since that last happened. I’ve removed them and changed my password again, to something ridiculous. Not sure if there’s anything I can do beyond that.
  • One visitor, presumably robotic, immediately started loading the same page dozens of times a second within a few minutes of James going back up. I’ve banned its IP, but I don’t know how to prevent that kind of thing in general, or why it would cause slow SQL database queries.
  • Something was accessing parts of the database that aren’t supposed to be public. After careful investigation I discovered that, on this occasion, it was myself: when I accessed the database directly to do some maintenance, it was taking 6 seconds or so just to display or select the tables. Further back, there are still calls to these private tables that I can’t explain. I choose to ignore both these problems.

Thanks for sticking around.

Comment
 
 
Jazmeister: Cute picture! Man, dicks on the internet are so not in short supply. We should all write programs to scour every corner of this website for dicks! All day, every day!

ZomBuster: It was the walrus, he just wanted to check if you had his bucket.

EGTF: @ZomBuster
I think it was the Eggman personally.

Amusingly I came across about a dozen pieces published in the week or so you were offline, all linking to articles from here.

Good to see your blog back Tom. Thanks for still taking the time to write interesting things for no real fiscal return on here.

Iain "DDude" Dawson: So, were you aiding the lolrus bukkit search, or are you a suspect?

vladh: Tom, maybe it's not Wordpress. Maybe there's a hole in some of the other files you have on the server, or maybe there's a hole in BlueHost's server. I wouldn't be surprised, why the hell are you signed up with them anyway?

Dr. Nerfball: @Iain: He's definately aiding said lolrus. Otherwise why would the walrus be nervous? It would be Tom who would be terrified. I mean, isn't several hundred (or whatever) pounds of blubber with GIANT TUSKS not the scariest thing in the history of forever when you are between it and it's bucket?

Also, what noise does a Walrus make anyway?

Dr. Nerfball: Dangit. *If he were a suspect it would be Tom who would be terrified.

Also curse my need to be coherent in my ramblings!

Ludo: @Dr. Nerfball:

Coo coo ca choo!

Dr. ROCKZO: Great to see you back!

Jonas: Good job finding that bad code. I've had stuff like this happen to me several times before, and if I didn't know a really competent and nice web developer, I would never be able to fix it when it happens. Who are these tossers, anyway?

Looking forward to more posts out of you.

Lack_26: I know, we make the server private to anyone who hasn't personally been vetted by Tom himself. We send him a request and he comes round for a cup of Tea and certifies our profiles. Perhaps even biscuits and a scone.

Jethro: Nice to have you back on the things people call the webs. :D

J-Man: Has Tom gotten over his fear of large sea life?

Lb: Try blocking the source ip subnet, not just the host. Find th subnet with a bgp looking glass. Might cause some collateral damage, but if the attacker has a dynamic ip from an ISP blocking one ip won't work. That's assuming they are attacking from their own connection...

Aeneas: I just assumed you had died and not paid your bill. But good to see you back.

The Nervous Walrus: I found my bucket now, sorry for attacking your website manthing. Any future attacks I blame entirely on my friend the Carpenter.

Mike Arthur: Glad to see the site back up. Don't know if you got my comments on Twitter.

As you're running Wordpress you really want to get yourself using the WP Super Cache plugin, it makes a huge difference to the CPU time and reducing the amount of SQL queries.

Also, investigate possibly changing your web server (if your host allows it) to nginx or lighttpd. Both perform a lot better when serving the static content which WP Super Cache provides than Apache does.

I assume you've always kept Wordpress fully up-to-date? If not, do so and if so then considering using an .htaccess file to put another password on your /wp-admin/ folder, this helps prevent some of the bots which crawl the web looking for Wordpress sites to attack.

Hope this is of some use. If you want a hand with any of it then contact me and I'll give you a hand. My contact details are on my website.

 
Pentadact: Cheers Mike. I did see your Tweet, but thanks for reminding me now I'm back and able to follow up on it. Installed Supercache, don't completely understand the options but I put it fully on and unchecked the thing for 'VERY busy sites'.

I'll try that IP subnet mask banning thing if the problem recurs - so far, since the measures I took yesterday, there hasn't been a single slow SQL query. Compare that with about fifty in the first five minutes of the site going back online, or the one day last week when there were so many that the log files of when they occured came to over 100MB of raw text. I may be in the clear.
 

Mike Arthur: The problem with the subnet mask banning is if, for instance, someone in the same neighbourhood tried to hack your site then everyone on the same ISP in the same block will get blocked too.

The Supercache options are fairly good by default as long as they are set to "on". It basically means there won't be any PHP run or database queries done.

Looking at the footer of the page it looks like you might not have set up the .htaccess stuff properly yet. I could be wrong though.

Hope you're alright now though!

Rei Onryou: It's possible that some attacks were through SQL injections. By typing SQL commands into the comments box, an evil walrus could gain access to, or affect, the database.

XKCD does a better job of explaining the dangers here: http://xkcd.com/327/

Chris R: Glad you're back up. Was missing the great articles you write!
 
 

Comment

 
URLs get turned into links automatically. You can use <i>HTML</i> but not [b]forum[/b] code. If your comment doesn't show up, e-mail me - the spam filter's just detained it for questioning.






 
 

It’s about trusting another man with your life, and immediately feeling his confident expertise guide you both to glorious, spectacular victory in an airbourne vehicle you know from experience it’s extremely hard not to crash. It is deeply homoerotic. ...

 

 

About

 
A blog by award-winning asshole Tom Francis about:
Games
Personal stuff
Music
Television
Films
Everything
 

Status

 
 

PC Gamer

 
I write, furiously, for PC Gamer. So do these guys:
Tim Edwards
Craig Pearson
Graham Smith
Alec Meer
John Walker
Kieron Gillen
Jim Rossignol
Richard Cobbett
Jon Hicks
 

Favourites

 
Fluxblog
MP3 blog.

Adam and Joe
UK radio show.

Waxy Links
Tech links.

The Big Picture
News photography.

1st Person Shouter
Games.

Dinosaur Comics
Dinosaurs.

This American Life
Personal stories.

Mark Kermode
Film review podcast.
 

Images