To answer your question, I think you need to understand the role of
robots.txt, which, in view of the very recent mobile friendliness changes in Google's algorithm, is something I have just been working on.
robots.txt exists to "request" that search engines and other such user agents do not index the specified contents of the website within the file. The vast majority of websites contain files that make no sense to be indexed as they are internal to the workings of the site (Joomla as well as most other CMSes and the like). Joomla 3.x ships with a
robots.txt that allows indexing of the root, the /media folder and the /templates folder as these contain file necessary for "correct" page rendering.
Now, the format of
robots.txt is not actually a well defined beastie, but broadly it contains one or more "groups" of URL specifications, each "group" corresponding to a specific user-agent (for most purposes here read that as "search engine bot").
In my experience, most
robots.txt files only have one "group" for the user-agent which is
* (i.e. every type of user-agent). Within these "groups", are a sequence of lines each specifying an "action" and then a path (be that just a directory e.g.
/admininistrator/ or a set of files e.g.
/images/*.html or a specific file
/administrator/index.php). Example extract:
Now, the "problem" arises because the only universally agreed "action" is
Disallow, that is "please user-agent, don't index the path specified to the right". I use that phrasing because
robots.txt is only a request mechanism, that is its actions are not binding, the visiting agent can choose to honour the actions or ignore them.
As it happens, the Googlebot does permit the use of an
Allow action, but at present it is the only one of the search engines to state that this is the case, so I suggest you don't rely on it.
In essence, the problem boils down to: you don't want the "internal" files that make Joomla work to be indexed, but equally you don't want to reveal to those with malintent too much about your website's internal structure - these are not quite mutually exclusive, but are not far off.
With having to rely wholly on
Disallow, it means I think you don't have many options to try to arrive at a
robots.txt that is manageable, doesn't tell the malintents too much but does result in Google et al indexing only what you want.
The approach I have taken (don't do this at home folks) is to patch
/libraries/joomla/document/document.php to subtly alter the behaviour of the
addStyleSheet methods. For me, what they do now is to check if there is a version of the specified script/stylesheet in a place like
/media/<original path> and also to test if there are minified versions of
.css files (minification is a whole other topic). I then have a script that I run regularly that checks everywhere that is
Disallowed in the
robots.txt that contains media files and copy these to the same type of URL, but under
So, as it happens, I use Gantry too, but now the gantry files are copied to
/media and as my
robot.txt doesn't include any reference to
/media/ the resources necessary for "correct" rendering are not blocked but equally I have a minimal and manageable