Saving bandwidth by preventing spider access to your wordpress blog

Jun 21, 2012 by DelilloDress261

I have a couple of web sites, and just lately (this past month), I’ve noticed that one particular of them is employing what I would consider excessive bandwidth. Digging a small deeper, it appears that bots are making use of all the bandwidth.

This is just my private internet site – I have a travel weblog in the primary domain and a loved ones history site in a subdomain. In the past, I have utilized on typical 400mb per month of bandwidth. This month I’ve had to boost the bandwidth to 1.5GB, but it is probably going to go over (it is presently at 1.38GB). These two websites aren’t massive, and don’t get stacks of hits, mostly just close friends and household. The family history website has a couple of large pictures, but absolutely nothing excessive.

Hunting at awstats this month, the weblog site has used 150mb of bandwidth + 480mb of bot bandwidth (380mb of that is msnbot). The loved ones history site has utilized 55mb of bandwidth + 650mb of bot bandwidth (620mb of that is googlebot).

Most robots determine themselves by a custom user agent in the request headers. Which can simply be blocked with htaccess.

There are a number of great articles on this. Let me know if you have any problems, as it really is a matter of identifying the offending bots/crawlers and banning them as per your require.

I have utilised google tools to inform it to not frequent the website as considerably about a week ago, but it doesn’t seem to have made a distinct. Short of telling the robots to bugger off totally via the robots.txt file (it is just my individual internet site, but it is nevertheless good to be listed in google!), is there something else I can do?

I at the moment use this on all my sites , essentially it blocks all undesirable user agents , negative bots and scrappers, Not only can it save your content material from being mass harvested but will also save you a little bandwidth due to the fact of much less bots running about your internet site. Hope it helps

can inform you that Google drags in a fantastic several spiders due to advertizing, specifically if you are making use of Adsense on your web site along with ising the different advertisements from the Google ad network partners – these partners also send their bots to test your targeted traffic sources and what adverts to location on your internet site – Google has been hitting challenging lately because of the algorithm tweaks and Adsense having had a profitable month in the terms of the amount of new advertisers on board.

Even if you slow down the crawl rate, you will nonetheless see a significant chunk of bandwidth disappear. The bots are way also intermittent to make accurate adjustments unless you wish to block them.

There’s a new wordpress plugin that can help with this! I’ve gotten a couple of emails re an item being sold to remove or “spank” the poor spiders that are taking up lots of bandwidth and not adding value to your organization, freeing up space for real guests and not causing a difficulty with hosting limits. It’s known as Spyder spanker at that name .com if you want to see the sales page.

Anyway, I am not sure if this is one thing useful that I want or not. I do see a lot of spider activity in my stats, but I constantly thought that was sort of very good b/c it means they are crawling my sites and hopefully indexing them.

The large danger is stealing your bandwidth. Some of the spiders sent by spammers will hammer your site as quickly as they can, slowing down response for your human visitors.

To know more about it, please go to: spynderspanker review