How To Prevent Spy Bots From Snooping Around Your PPC Campaigns

As PPC affiliates, we spend countless hours researching our market, collecting keywords, building landing pages, and writing ads. After endless splits tests and tweaks, we finally find that grand slam “campaign” that seemingly deposits money into our bank, hand over fist. Month after month, the campaign is running strong, until one day for no apparent reason, the money dries up. What happened?

A number of things “could’ve” happened, but there is no time for speculation. There are “intelligence gathering” spiders out there, crawling and collecting all of our hard work into a centralized repository, for other affiliates to simply shortcut their way to your hard earned riches. Can anything be done to stop them?

Stopping these spy bots 100% of the time may or may not be possible, so prevention at the basic level is our best defense. We can construct a basic line of defense using our server’s .htaccess file.

If you aren’t familiar with .htaccess, you need to get educated quickly. Having a look at this article will get you up to speed with the rest of this tutorial: Apache Tutorial: .htaccess files

When I talk about “spy bots”, two that are widely in use among PPC affiliates comes to mind: KeywordSpy & SpyFu. Without giving you the whole breakdown of what these two services do, the reader’s digest version is: They steal your shit!

Luckily though, with a few .htaccess mods, we can block their spiders from ever visiting your site, therefore prevent the further analysis of your campaigns.

Here’s what you need to do:
1) Make a backup of your current .htaccess file. Modding existing .htaccess files can be tricky, especially if they already contain data. Save your ass and your site with a backup. Trust me.

2) Open the existing .htaccess or create a new one with notepad.

3) Add the following text on a new line:

<Limit GET HEAD POST>
order allow,deny
deny from 74.53.36.242
deny from 65.39.72.142
deny from 66.34.204.26
deny from 66.34.0.
deny from 66.34.255.
deny from keywordspy.com
deny from keywordspypro.com
deny from spyfu.com
deny from spyfoo.com
deny from foospy.com
deny from fuspy.com
allow from all
</LIMIT>

What these lines do is blacklist the specified IPs and domains from being served a website upon request to your server. Notice that not all of the “deny” lines include full IP addresses, but just the first 3 octets of the address. This specification prevents the entire range of IP addresses on the given network from accessing your site in case other servers not listed might actually be the culprit data mining servers.

4) We’re not done yet. Again, on a new line add the following:

RewriteCond %{HTTP_REFERER} keywordspy\.com [NC,OR]
RewriteCond %{HTTP_REFERER} keywordspypro\.com
RewriteCond %{HTTP_REFERER} spyfu\.com [NC,OR]
RewriteCond %{HTTP_REFERER} foospy\.com [NC,OR]
RewriteCond %{HTTP_REFERER} fuspy\.com [NC,OR]
RewriteCond %{HTTP_REFERER} spyfoo\.com [NC,OR]
RewriteRule .* - [F]

What we’re doing with these lines is returning a 403 Forbidden Error to the server when the referrer is equal to the domains specified. In this case, I am “forbidding” KeywordSpy.com and SpyFu.com, along with a few other domains owned by these companies from collecting data.

Note: In order to make use of rewrites, the Apache modrewrite module must be enabled on your server. In most cases this is enabled by default, but with some web hosts, it won’t be. Add “RewriteEngine On” to the top of your .htaccess (w/o quotes)

5) Save your file as .htaccess and upload to the root directory of your server. If Windows gives you problems with naming your file .htaccess, simply call it “htaccess.txt”, upload it with FTP, then rename it once it’s on the server.

Note: Depending upon your server security, youre .htaccess file could be viewable by others. To prevent this, add the following lines and also CHMOD your .htaccess to 644 permissions:

<Files .htaccess>
order allow,deny
deny from all
</Files>

If you want to see a complete working example of a .htaccess file you can use, download here (rename to .htaccess) :

.htaccess Sample

Now what we’ve just done is add two lines of defense. Not only are we blocking the named IP addresses and ranges from accessing our servers, we are also denying them based on domain name. The IP addresses may change at some point in time,  but most likely the domain names will stay the same. For example, by adding “deny from keywordspy.com”, we are blocking Keyword Spy from visiting our site, no matter which IP address keywordspy.com may be assigned. So blocking both domain names and IPs will provide extra protection from future changes.

Of course the above mods aren’t an end all solution. The IP addresses in the deny section may change over time, so it will be your job as a defensive marketer to continually monitor which online spy bots are out there, get a list of their IP addresses and possibly ranges, and update your .htaccess file accordingly.

Also, check your server logs regularly for suspicious queries and new spiders that may be accessing your site. If your landing pages, keywords, and ads are already included in the databases of SpyFu or KeywordSpy, then I’m afraid you’ll just have to wait for the data to be removed, if at all.

I’m not 100% sure, but once the bots are denied access, I believe that over time these services will consider your campaign “offline”, and purge the collection of data from your site. But then again, KeywordSpy has a “TimeMachine” feature that can go back in time and pull cached data.

Again, the above mentioned mod won’t be a fix all to your spy bot problems, but just one protective measure you can take to better defend yourself from “clone” affiliates copying your campaigns. But if you’re on the Internet, you’re vulnerable. Period.

Luckily though, the data collected by these bots is often either outdated or simply wrong. They can monitor how long your keywords have been running, but not necessarily detect which specific keywords are profitable for you.

Also, because of numerous factors such as CPCs, commission rates, upsells, etc, they really have no idea how accurate your ROI is per keyword. So if you use these tools yourself to collect information, take it with a grain of salt and test before you get excited thinking you’ve just found yourself a profitable campaign.

There are so many factors at play influencing whether or not a campaign is profitable, even having a list of of keywords you know for a fact that someone is making money with, isn’t necessarily the magic bullet toward the riches. Testing will out win “copying” any day.

If you have other methods or insite into how spy bots collect data, be sure to let me know so I can update this post (and of course give credit where credit is due).

Way of the Warrior - Tip of the Day: Learn how to use and master .htaccess. It can be your friend when in need. But remember to make a backup before making any changes. Mistakes in your .htaccess can be disasterous to your website.

Similar Posts Of Interest:

• September 17, 2008 -- Importing Negative Keywords Into MSN AdCenter (0)
• November 9, 2008 -- Improving Google Quality Score With Hidden Navigation (5)
• November 4, 2008 -- Prosper202 Self-Hosted Apps: 10 Best Practices To Securing Your Prosper202 Installation (6)
• November 1, 2008 -- New AdWords Technique for Competitive Spy Intelligence (1)