Finding an unnatural links warning in your Google Webmaster Tools inbox is one of the last things anyone wants to see. It’s the stuff of nightmares if it’s your job to appease Google and follow its trends.
So if you are making development websites using existing content you had better beware.
You know it is a development site but don’t expect a search engine algorithm to take the same view.
If you build it, Google will come
Creating a development or test site is a fundamental part of migrating from an old website to its latest incarnation. So many elements require testing and refining. Content must be imported and often reworked to function within new parameters.
People will be concerned about how the new features work, if the site looks pretty or not in the latest version of IE and if you are lucky someone may have given a second thought to your test site being indexed by Google.
You’ve built a website. Google will find it and index it, rate it and compare it. This is what search engines do and they are very effective at it. Gone are the days of submitting your website to search engines so that they can find it. Anything you put online without taking preventative steps can and will be found.
The importance of not being seen
Here is how your company could receive an unnatural link warning:
1: Set up a development site using content from the old website.
2: Not notice that links within the old content are coded using the full domain name and so are linking from the new test site to your live public site.
3: Take no steps to prevent Google from indexing your test site.
4: Receive an unnatural link warning.
Why would you get an unnatural link warning? Suddenly in Googles eyes there are hundreds of links from a new domain pointing at your primary domain and this is exactly the sort of bad SEO technique that companies used to employ to bolster their rankings – the sort of techniques that get you warnings and penalties.
Not only do you now have lots of unnatural looking links, you have also generated an entire duplication of your websites content. Every carefully crafted page title and precise piece of page copy is now no longer unique to your domain.
If you are wondering at this point if your development website has been indexed then enter into Google search “site:yourtestsite.com” and hope that it returns no results. Anything that does appear has been indexed.
How to prevent your site from being indexed:
Step 1: Robots.txt
The most singular important and simple method of keeping your test site away from the prying eyes of search engine bots is to turn them away at the door. The robots.txt file acts as their instruction manual and is commonly used to inform search engines which directories should be ignored.
In the case of development sites we do not want any of the pages to be indexed and to encourage them from doing so we add the following into the robots.txt file.
If your test site is on the root of the domain:
User-agent: *
Disallow: /
If your test site is within a sub-folder:
User-agent: *
Disallow: /examplefolder/
Remember this robots.txt is for development not for live sites!
Step 2 : No Index
If you wish to further instruct search engines to look elsewhere then there is also the easy to implement ‘noindex’ meta tag. For most sites that use a template design this is simple to implement site wide within the <head> tags of each page.
<meta name=”robots” content=”noindex”>
Be sure not to forget it is there when you go live with the new website.
Step 3 : Google and Bing Webmaster Tools
Dont operate blind. GWMT is a very powerful tool set and includes numerous features to help you better understand what Google can see and how it is relevant to your site. You can even request that Google remove URLs that it has indexed.
Bing webmaster tools isn’t far behind Google and takes as little effort to set up. With such fantastic tools to monitor websites and instruct search engines there should be no need to guess or leave anything to chance.
You could of course leave it all to chance and hope that during the site migration process Google ignores your development website. But if your keyword rankings are business critical then it will save you time and money if you ensure that all of your testing is conducted behind closed doors.
Insights
