Detecting and blocking click fraud on asp.net sites

Click fraud/bomb problem noticed in Google AdSense

I recently noticed a problem with potential click fraud in my Google AdSense statistics.  The CTR (click through ratio) went very high over the course of a few days.  I pulled up the graphs in Google AdSense and saw that CTR had gone as high as 30% on one day.  After a few days I noticed the same thing happened to another website I maintain.  Knowing that Google policies dictate prevention of click fraud falls on the shoulders of the site owner, I began investigating how to detect and block what could potentially be fraudulent click throughs on my website site ads.

 

You can see where the yellow line representing CTR was flat lined and all of a sudden it surged as someone (or something) launched a click bomb attack on my ads.

 

Google AdSense graph showing a surge in click throughs

 

Here’s the tabular report on the custom channel (website) for the website.  Note the Page CTR which for this site, typically runs around 1-2%.  It surged as high as 34.10%.

Google AdSense tabular report showing unusually high Page CTRs

 

Why is this important?

This is important if you want to keep your Google AdSense account.  Right or wrong, Google bans accounts for click fraud with practically no chance of getting that account back in good standing.  Despite what Google says, I’ve seen hundreds of people complaining about losing their Google AdSense accounts because of click bombs that a competitor, errant bot, or whatever produced on their websites.  Google has an appeals process but I’ve never heard of anyone ever winning an appeal with Google.  In fact, I’ve never heard of a developer even getting a response back from Google after an appeal.  Forgot the days where companies like Microsoft treated their developers like royalty.  It’s a Google world now and at least at this time, they own the online advertising market with no descent alternatives available to website developers/owners.

Step 1 – Notify Google and put preventative measures in place

Notify Google

Firstly, I notified Google that I was experiencing unusually high CTR that I felt might be fraudulent.  Google will not respond to these notifications so you just have to hope your notification didn’t drop off into a black hole somewhere.  Don’t use Google’s “contact us” of “feedback” form or plain email.  Google has a specific form you should use to notify them of fraudulent activity.  It’s hard to find so just search “Invalid Clicks Contact Form” to find it.  In the notification form, you’ll need to supply your publisher id, dates and times of the activity, a description of why you think the activity is invalid, and any supporting evidence which could explain the invalid activity.

Preventative measures

While we implement a more permanent solution, there are a few quick things you can do to prevent invalid click-throughs.  Firstly, you could remove all Google ads from your website but that’s like throwing out the baby with the bathwater.  Google holds you responsible for clicks coming from your site but I don’t think they don’t expect you to commit suicide to keep the baddies away.  The other option is to temporarily disable payments from the website while you test to see if additional preventative measures (discussed below) will stop the excessive clicks.

Remove authorization for the offending website

From your Google AdSense dashboard, under Account Settings, you’ll find an option to set “Sites authorized to show ads”.  You can add your sites here and Google will only serve ads to the domains you specify.  This will stop anyone from scripting to fire off your ads or from scraping your add code into their own site.  It also stops bots from firing off the AdSense clicks.  Note that you’ll want to specify the domain as “yourdomain.com” without a protocol or subdomain specified.  If you have multiple subdomains, specifying the root domain will work just fine and all subdomains will be inherently allowed.  For example, enter “yourdomain.com” and “www.yourdomain.com”, “games.yourdomain.com”, and “sub.yourdomain.com” will all be allowed to serve AdSense ads.

There is an unusual consequence of doing this.  Google will still serve the ads to your site and apparently still collect their revenue from the advertisers, but they will not share any of the revenue collected with you – even for clicks that their fraud-detection algorithms deem valid.  I’m not sure how much I like this nor how it fits with Google’s “Don’t be Evil” corporate mantra but for now, that’s the way it is.  If you remove your domain temporarily from the list of authorized domains, you will see stats in Google AdSense, including click-throughs, but you will collect zero revenue from that domain until you add it back into the authorized sites list.

 

Here you can see that I only authorized one website on my account – brianhaddock.com.

Adding authorized sites to Google AdSense settings

 

After you have enabled Authorized Sites, Google AdSense will being reporting unauthorized sites as they attempt to serve your ads.

Google AdSense alerting that unauthorized sites have displayed your ad code

 

Clicking details will show which sites are displaying your ad code (but no more detail than that).  In my case, after only a few hours I saw Facebook and Bing appear in the list of unauthorized sites that attempted to “click” on my ads.  Yeah, I find it somewhat disconcerting that Google’s two top competitors appeared to have been trying to force advertising expenses to Google AdSense advertisers (and potentially causing my AdSense account to be shut down).

Google AdSense lists the sites

 

 

Step 2 – Determine where the errant clicks are coming from

Next I needed to determine where the invalid clicks were coming from.  Google AdSense reports unauthorized sites (after you enable the authorized sites option), but not how many clicks those sites generated nor how they generated the clicks.  I use Google Analytics to track my website usage stats but Google Analytics is missing an important measurement that we need to determine who is clicking our advertisements.  You can view Exit Pages in Google Analytics but you cannot view Exit Links (actually there is a complex solution which involves adding code to all your links but I digress).  There is a free alternative that will work though – StatCounter.

Using StatCounter to spot invalid clicks

Visit StatCounter and set up a free account.  Then add a “project” which is essentially the setup for the website you wish to monitor.  StatCounter will generate the JavaScript code for your project and you simply insert the JavaScript code into the footer area of your webpages.  Note that for WordPress users, StatCounter offers a WordPress plugin that accomplishes the same thing.

Once StatCounter is setup, you can view “Exit Links” and see when users are exiting your website through the Google AdSense links.  Note that with the free account, storage space is limited so for higher traffic sites, you’ll either want to purchase one of their monthly subscription plans or live with only seeing details of the most recent Google AdSense click-throughs.

 

Here StatCounter shows us the exit links clicked.

StatCounter showing Exit Links clicked

 

Clicking the magnifying glass icon next to the exit link lets us view details about that clickthrough, including the IP address and location of the offender.  We are looking for visitors that are producing an unusually high number of AdSense click-throughs or are clicking the advertisements repeatedly within a short period of time.

StatCounter details on an Exit Link

 

Google Analytics alerts

Although not as informative as StatCounter, we can configure Google Analytics to give us some information about AdSense click throughs.  Your Analytics account has to be linked to your AdSense account though.

 

From your Google Analytics dashboard, click the Add Widget link.

Click the Add Widget link

 

Choose the Table Metric and add the metrics as demonstrated below.  Unfortunately, we cannot get their IP address in this manner.  AdWord advertisers can get information on invalid clicks but AdSense publishers cannot.

Add these metrics

 

This will give you a general, albeit not very helpful, idea of how many AdSense clicks you are seeing by country.  If you use Google Analytics regularly, it may be helpful to add these metrics to your dashboard so you can easily keep an eye on click-through patterns.

Simple Google Analytics report showing clicks by country

 

Although the Google Analytics report above is not very helpful, it does set the foundation for a Google Analytics method that *will* be helpful – a custom alert that triggers when our CTR is running high.  To set up the custom Google Analytics alert, go to your custom alerts page in Google Analytics and choose to create a new alert.  Complete the screen as follows and apply the alert to any or all of your websites.

Google Analytics alert to high CTR rate

 

And why do we care to be informed the minute CTR is running high?  So we can jump out to Google Analytics or StatCounter and start gathering information on the visitor that is producing the invalid clicks.

Google Analytics also provides some interesting and possibly useful information under the Standard Reporting tab.  Go to Audience -> Technology -> Network –> AdSense and/or Audience -> Demographics -> Location –> AdSense for AdSense stats.  Both provide a bit of detail on what is going on.  Unfortunately, the do not give us the IP address related to the specific statistic so StatCounter is still the top solution for determining the identity of the offender.

Once we determine who is causing the fraudulent click-throughs, we can block them from our website.  Since setting up “allowed sites” in Google Adsense should cover any click-throughs from outside our website, blocking specific clicks that originate from within our website requires we block site access for those users.

Step 3 – Block fraudulent visitors from your site

Blocking IP addresses for WordPress sites is pretty simple – just install a plugin that blocks IPs and run with it.  For ASP.Net sites, you have to roll your own solution (or copy mine below).  In my instance, my ASP.Net website runs on a share host so I do not have access to the firewall settings nor IIS settings that I could use to block specific IP addresses.  To block fraudulent clicks in my ASP.Net website, I created a http module that reads a text file containing a list of IP addresses I wish to block.  This involved three steps.  First, code the http module which I give below.  Upload the http module to the app_code directory on your web server.  Next, upload a text file containing the list of IP addresses you wish to block.  The list of addresses should be separated with a simple carriage return/line feed.  You will need to change the code to specify the directory location and file name of your blocked ip address file.  Finally, modify the web.config to define the http module we created.  Note that once the solution is in place, you will simply edited the block ip address text file to add or remove IP addresses you wish to block.  The http module caches the blocked IP address file but once a change is made to the file (i.e. you add another IP address to block)  the cache is flushed and the newly edited file reloaded with the newest IP address list.

ASP.Net HTTP Module to block ip addresses

using System;
using System.Web;
using System.Collections.Specialized;
using System.Web.Caching;
using System.IO;

public class BlockIP : IHttpModule
{
public BlockIP()
{
}

#region IHttpModule Members

public void Dispose()
{
}

public void Init(HttpApplication context)
{
context.BeginRequest += new EventHandler(Application_BeginRequest);
}

private void Application_BeginRequest(object source, EventArgs e)
{
HttpContext context = ((HttpApplication)source).Context;
string ipAddress = context.Request.UserHostAddress;
if (!isBlockedIPAddress(context, ipAddress))
{
//context.Response.StatusCode = 403; // (Forbidden)
context.Response.StatusCode = 404; // (Not Found)
}
}

// Function to check if IP address is blocked
private bool isBlockedIPAddress(HttpContext context, string ipAddress)
{
StringDictionary badIPs = GetBlockedIPs(context);
if (badIPs != null && badIPs.ContainsKey(ipAddress))
{
return false;
}
return true;
}

const string BLOCKEDIPSKEY = “blockedips”; // cache name
const string BLOCKEDIPSFILE = “/include/BlockedIPs.txt”; // location of the text file containing a CR/LF delimited list of bad IPs

// Wrapper function to retrieve and return our dictionary object of bad iPs
public static StringDictionary GetBlockedIPs(HttpContext context)
{
StringDictionary ips = (StringDictionary)context.Cache[BLOCKEDIPSKEY];
if (ips == null)  // not cached so read IPs from our text file
{
ips = GetBlockedIPs(GetBlockedIPsFilePathFromCurrentContext(context));
context.Cache.Insert(BLOCKEDIPSKEY, ips, new CacheDependency(GetBlockedIPsFilePathFromCurrentContext(context)));
}
return ips;
}

private static string BlockedIPFileName = null;
private static object blockedIPFileNameObject = new object();
public static string GetBlockedIPsFilePathFromCurrentContext(HttpContext context)
{
if (BlockedIPFileName != null)
return BlockedIPFileName;
lock (blockedIPFileNameObject)
{
if (BlockedIPFileName == null)
{
BlockedIPFileName = context.Server.MapPath(BLOCKEDIPSFILE);
}
}
return BlockedIPFileName;
}

public static StringDictionary GetBlockedIPs(string configPath)
{
StringDictionary retval = new StringDictionary();
using (StreamReader sr = new StreamReader(configPath))
{
String line;
while ((line = sr.ReadLine()) != null)
{
line = line.Trim();
if (line.Length != 0)
{
retval.Add(line, null);
}
}
}
return retval;
}

#endregion
}

A couple of notes about the code above.  Firstly, I chose to issue a “404 – Page not found” response for any blocked ip addresses.  I could have used a 403 (forbidden) instead but chose to not tip off the banned users to the fact that I had blocked them.  If you choose instead to slap the offender in the face, uncomment the “403” line in the code and comment out the “404” line and they’ll receive the stern looking “you are banned” error page instead.

Secondly, you’ll need to change the BLOCKEDIPSFILE definition to the location of your banned ip address file.  I named my file “BlockedIPs.txt” and put it in a newly created /include directory.  Here’s the line in the code that you want to change:

const string BLOCKEDIPSFILE = “/include/BlockedIPs.txt”; // location of the text file containing a CR/LF delimited list of bad IPs

Adding IP Addresses to block

Here’s what my BlockedIPs.txt file looks like.  You can upload this file anywhere you like but be sure to change the location in the code (as explained above).

176.61.49.95
176.61.49.94

Modify web.config to add the HTTP module

Lastly, I modified the web.config to specify the http module I created.  In the <httpModules> section, add this line.  In this case, I named the class “BlockIP”.  If you do not change the class name in the code then you can simply cut and paste the line below into your web.config file.

<add name=”BlockIP” type=”BlockIP”/>

The site is now set up to block ip addresses.  I will continue to watch Google AdSense reports and StatCounter Exit Links and add any offending IP addresses to my BlockedIPs.txt file (while crossing my fingers that Google comes up with a more elegant, automated, open, and fair system to block click bomb and click fraud activity for us).

Print Friendly, PDF & Email

You may also like...

%d bloggers like this: