As robots.txt can be found:
everyone can see it, including also bad thinking persons.
How can I hide the robots.txt from everyone except the search-bots?
You can find a solution here:
This will block robots.txt from everyone except googlebot, Yahoo Slurp and msnbot.
But again, Why would you want to hide it? It is perfectly OK to keep robots.txt accessible.
Pretty much every website has a robot.txt accessible to everyone. You can even see wikipedia's robot.txt here: https://en.wikipedia.org/robots.txt
I am also negatively positioned against the robots.txt file. I don't like it at all to announce publicly the directory tree of my Joomla sites.
Considering the recent changes in Google's algorithms, that are now looking for Images, CSS and JS files, in order to fully fetch and render a webpage, I am looking for ways to allow the Google bots to reach to the necessary files that make up a webpage, while at the same time will be able to hide from bots anything that shouldn't be indexed.
To this direction, recent versions of Joomla are shipped with an updated robots.txt file, that allows bots to crawl the images and media files.
Yet, in the majority of the Joomla sites, there are more than 1 locations that those files may exist - It's not only the template or the media folder and images folder, but it can also be a module or a plugin folder, and yet it's possible that not all contents of the media folder should be available for crawling.
An alternative way I am currently experimenting and eventually want to make my standard implementation, is the use the X-Robots-Tag HTTP header, in an effort to completely avoid the use of the robots.txt.
We can add any kind of robots meta-tag directives with the X-Robots-Tag within .htaccess files, and .htaccess files can be placed at the root directory, that will have a global effect to whole site, but we can also place them insider any children directory of the site.
Example 1: X-Robots-Tag to not allow indexing of PDF and GIF files across the site:
.htaccess at the root directory:
<Files ~ "\.(pdf|gif)$"> Header set X-Robots-Tag "noindex" </Files>
Example 2: X-Robots-Tag to not allow indexing of a specific directory::
.htaccess inside that directory
Header set x-robots-tag "noindex"
This will add the no-index X-Robots-Tag at the HTTP response headers of the pages inside this directory:
With the use of Regular expressions we can achieve maximum flexibility in specifying all kind of directives.