Whether you are setting up your WordPress site yourself or someone else is doing it for you, it’s quite common to tick the box in Settings > Writing that says ‘Discourage search engines from indexing this site’.
What does this box do?
It sends a request to well-behaved search engines such as Google and Bing asking them to not list your site, and it does this in two ways:
- it adds a meta tag to your site’s pages which says ‘<meta name=’robots’ content=’noindex,nofollow’ />’;
- and, if your site does not already have a robots.txt, it creates one with the command ‘Disallow: /*’.
Quite often web developers will tick this box while they are first building a site so that the search engines don’t find it and list it until it’s ready for public consumption, but the box should ALWAYS be unticked once the site is ready to go live.
What if the box isn’t unticked?
Generally this will mean that your site isn’t going to appear in Google or Bing no matter how well crafted it is, so if you’re having problems finding your site via the search engines it’s always worth checking that it has been unticked.
I’ve also seen one case where the site was being listed, but the description underneath the site’s name read ‘A description for this result is not available because of this site’s robots.txt’. In this instance, unticking the box wasn’t enough. I also had to amend the robots.txt to have the correct commands in it, and then present the site to Google via Google’s webmaster tools to get Google to pick up the new robots.txt.
If you have problem that you think might be related to your robots file, email me at firstname.lastname@example.org. I’m always happy to help.