There are many reasons why one might want to stop search engines from crawling or indexing WordPress websites. One might be to prevent the indexing of website content while the site is still under development. You probably don’t want Google to index your under-construction or maintenance mode page while the site is still under construction. Another reason might be to prevent a search engine from indexing or listing sensitive information that should only be hosted on your sites. Whatever the case, it’s easy to configure WordPress sites to instruct search engines not to crawl or index content from a site you don’t want to list online. When you’re ready to prevent search engines from crawling WordPress, follow the steps below:
Discourage Search Engine from Indexing the Site
WordPress comes with a built-in feature that allows this to be done easily. It instructs all search engines not to index the site and all pages produced on the site. Although the instruction to these crawlers is not to index or list your content, it’s up to the crawlers and bots to respect and honor the site’s request. Some bots online will still crawl and list your content regardless of what your no-index policy states. To instruct all bots not to index your site, go to do is visit Settings » Reading and check the box next to the Search Engine Visibility option as shown in the image below: Click the Save Changes button and you’re done. When you enable the settings above, WordPress will add the line below to your website’s header file. WordPress also modifies your site’s robots.txt file and adds these lines to it: These lines ask robots and web crawlers not to index your pages. Again, it is totally up to search engines to accept this request or ignore it. Most search engines respect will respect your request and this may not stop some of your pages to get indexed. It’s just how the web works.
Password Protection
A surefire way to make sure search engines don’t index or crawl your website is to password-protect the entire site at the server level. Password protecting your sites makes it so that another who wants to view the content will have to enter the correct password. We have shown you how to use Apache2 or Nginx HTTP server to password-protect directories and content. You may also install the Password Protected plugin to do that for you. To view our posts on protecting Apache2 or Nginx directory, click the links below: Apache2: Password protect directories using Apache2 Nginx: Password protect directories using Nginx These steps may help you protect your sites from search engines, bots, and other crawlers from listing or indexing your content. Hope this helps You may also like the post below:
