Are Ghost Referrers & Crawler Referrers Ruining Your Analytics?
Here’s How to STOP Spam Referrers In Their Tracks!
In Part 1 of this two-part blog series, we provided an in-depth explanation about what referral spam is. Here in Part 2, we will show you a number of ways that you can block or filter from your analytics and reports, those dastardly “spam referrals”, “ghost referrers” and “crawler spam referrers”. Can anything be done to block referral spam?
Yes! Read on as we show you some step-by-step how-tos. The steps are fairly simple for a Webmaster to follow. Quite challenging for the layperson who is not all that technologically adept. So if you are not all that sure-footed in working with hypertext access files or Google Analytics filters, you may want to ask your web developer (or us) to assist. We’re happy to help!
Part 2: How to Stop Referral Spam
There are a number of ways that spam referrers can be blocked (depending on the types of spam referrers). We recommend the following two. Please read our Disclaimer* prior to taking any of the steps outlined in this blog post; then proceed with caution.
- Your .htaccess File
- Google Analytics Filters
How to Block Spam Referrals With Your .htaccess File:
Your .htaccess File can be used to block crawler spam referrers by blocking spambots from being able to access your servers. In this Moz article, Jared Gardener of Red Door Interactive explains how to add code to the .htaccess file in the root directory of your website that locks out illegitimate crawlers. As Gardner warns in his article, a single coding error in the .htaccess file can result in your entire website crashing. Therefore, you should always make a backup of your original .htaccess file before making any changes to it. If you use WordPress, there are several plugins that offer the option to block visitors by IP, host name or referrer URL without the need to edit your .htaccess file.
It should be noted that this method of blocking spam referrals through the .htaccess file, only works for crawler spam referrals, not ghost spam referrals. As mentioned in Part 1, ghost spam referrals result from hits being sent directly to Google’s tracking service, and not by anything visiting your website, so these spam referrals will not be impacted by changes to your .htaccess file.
How to Block Spam Referrers With Google Analytics Filters:
Google Analytics Filters are the only known effective method for eliminating the appearance of ghost spam referrals in your analytics reports. In this Search Engine Journal article, Shane Jones of WebpageFX explains how to set up filters in Google Analytics to keep referrals that are from spammers (both crawlers and ghost referrals) from muddying up your data.
The most basic way to set up a filter is to filter out referrals from the domains of known spambots or ghost spam referrers. Filters can be established to filter out an individual referrer, or multiple referrers at one time. An example of establishing a filter for an individual spam referrer is included at the end of this blog post. When creating filters, you will need to be familiar with “regular expressions”, which gives meaning to special characters such as: – \ ? and $. Google uses regular expressions to help it understand exactly what you want to filter out of your analytics, and you can use them to quickly create a single filter that excludes visits from multiple referrers. The rules for using regular expressions can be found here.
Filters can also be established to filter out visits from particular regions. For example, if you only do business with customers living in the United States, then you can safely filter out visits coming from foreign countries such as Russia, China or India. Filtering out specific countries is also discussed in Gardner’s Moz article.
If you want to get creative, you can also setup a screen resolution filter, which acts as a broad-range filter. The concept behind a screen resolution filter is that unlike visits coming from physical devices, spam referrals do not have an accompanying screen resolution. So a filter set to exclude visits where the referrer’s screen resolution is “=(not set)” will exclude visits from most spam referrals. However, some spam referrals have become wise to this type of filtering and have started to encode their bots with screen resolutions to beat the filter.
Within Google Analytics there is a checkbox that will allow you to filter out “known spambots” from your Analytics reports. While it can’t hurt to check this box (and we suggest you do so), we’ve found that this method alone is not sufficient for filtering out the majority of spam referrals, especially ghost referrals that are not caused by spambots.
Keep in mind that filters permanently change your data. So if you filter out traffic, you will not be able to simply “uncheck” the filter and see what traffic looks like unfiltered – Google will not have recorded it. Here’s what you can do to protect your raw, un-filtered data:
Before setting up any filters, we recommend creating a new view in Google Analytics for your existing analytics account. Doing this will essentially create a duplicate copy of your website’s analytics information. When you create your view, name it something like “Filtered View www.websitexyz.com”. Creating a new view allows you to leave your original analytics untouched and it will continue to record all traffic normally. Having the original view on hand will come in handy in the event that filters are not set up correctly. Plus, it will allow you to maintain a virgin view that records all traffic, unfiltered.
We recommend that you set up any filters in the new “filtered view” that you create in your Google Analytics. After your filters have been running for one month, perform an investigation within your Google Analytics, to see if there is a difference in the traffic being recorded by your original view versus the filtered view – there should be fewer visits showing in the filtered view. This will give you a good idea of just how much traffic was coming from junk spam referrals.
During that same investigation, make sure that good referrals are still showing up in your filtered view, and that no desirable referrals are being filtered out, such as direct visits or AdWords clicks. If visits unrelated to spam referral traffic are also shrinking dramatically, that could be a sign that your filters were setup incorrectly and are filtering out desirable traffic. In this event, you will thankfully have your unfiltered view to fall back on!
Setting up Google Analytics filters can be tricky. Here are a couple of tips from Net Site Marketing when creating a filter to block a specific spam referrer:
1. For “Filter Type”, make sure to choose “Custom”, and under “Exclude” select “Campaign Source” (see Figure 1).
2. When entering the domain name (ex. “domainname.com”) of a spam referrer that you wish to block under “Filter Pattern”, be sure to put a backward slash \ before anything that is not a number or a letter (see Figure 2).
When setting up filters in your Google Analytics, keep in mind that it can take up to 24 hours for them to take effect. Also, filters are not retroactive, so they will only filter out future spam referrals and will have no impact on spam referrals that were recorded before filters were applied. Finally, unlike .htaccess files, analytics filters do not stop spambots from actually visiting your website; they just stop Google from recording their visits. You can learn more about creating and applying filters to views here.
*Disclaimer: Net Site Marketing and its owners, employees, subcontractors, associates, successors, heirs, assigns, etc. shall remain free from liability with regards to any & all real or perceived damages which may have been incurred as a result of following the instructions in this blog series about how to stop website referral spam.
If you have had difficulties with referral spammers or blocking them, please post your comments here, or reach out to us for assistance. We will be very gratified to help!
