2

As robots.txt can be found:

site.com/robots.txt

everyone can see it, including also bad thinking persons.

How can I hide the robots.txt from everyone except the search-bots?

  • 1
    Remember that if Google bot can see it then it is likely to be in search results, meaning your attempt of hiding things (presumably what your trying to do) would be futile. – tim.baker Jun 2 '15 at 14:10
3

You can find a solution here:

http://forums.ukwebmasterworld.com/threads/hide-your-robots-txt-from-visitors-and-show-it-only-for-validated-robots.7256/

This will block robots.txt from everyone except googlebot, Yahoo Slurp and msnbot.

But again, Why would you want to hide it? It is perfectly OK to keep robots.txt accessible.

Pretty much every website has a robot.txt accessible to everyone. You can even see wikipedia's robot.txt here: https://en.wikipedia.org/robots.txt

  • 3
    It is worth noting that this is quite hack of a solution and I wouldn't recommend it. That is not all the bots there are and you run the risk of new ones (such as Google Mobile, Apple etc) starting and being blocked out : not good! – tim.baker Jun 2 '15 at 14:09
3

You can't, robots.txt is meant to be publicly accessible. If you want to hide content on your site you shouldn't try to do it with robots.txt, simply password protect any sensitive directories using .htaccess or similar.

3

Robots.txt and Google

I am also negatively positioned against the robots.txt file. I don't like it at all to announce publicly the directory tree of my Joomla sites.

Considering the recent changes in Google's algorithms, that are now looking for Images, CSS and JS files, in order to fully fetch and render a webpage, I am looking for ways to allow the Google bots to reach to the necessary files that make up a webpage, while at the same time will be able to hide from bots anything that shouldn't be indexed.

Joomla

To this direction, recent versions of Joomla are shipped with an updated robots.txt file, that allows bots to crawl the images and media files.

Yet, in the majority of the Joomla sites, there are more than 1 locations that those files may exist - It's not only the template or the media folder and images folder, but it can also be a module or a plugin folder, and yet it's possible that not all contents of the media folder should be available for crawling.

X-Robots-Tag HTTP header - .htaccess & Apache Web Servers

An alternative way I am currently experimenting and eventually want to make my standard implementation, is the use the X-Robots-Tag HTTP header, in an effort to completely avoid the use of the robots.txt.

We can add any kind of robots meta-tag directives with the X-Robots-Tag within .htaccess files, and .htaccess files can be placed at the root directory, that will have a global effect to whole site, but we can also place them insider any children directory of the site.

Example 1: X-Robots-Tag to not allow indexing of PDF and GIF files across the site:

.htaccess at the root directory:

<Files ~ "\.(pdf|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>


Example 2: X-Robots-Tag to not allow indexing of a specific directory::

.htaccess inside that directory

 Header set x-robots-tag "noindex"

This will add the no-index X-Robots-Tag at the HTTP response headers of the pages inside this directory:

Response Headers: X-Robots-Tag

With the use of Regular expressions we can achieve maximum flexibility in specifying all kind of directives.


Useful resources:

Google Webmasters Documentation: Robots meta tag and X-Robots-Tag HTTP header specifications.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.