To answer your question, I think you need to understand the role of robots.txt
, which, in view of the very recent mobile friendliness changes in Google's algorithm, is something I have just been working on.
robots.txt
exists to "request" that search engines and other such user agents do not index the specified contents of the website within the file. The vast majority of websites contain files that make no sense to be indexed as they are internal to the workings of the site (Joomla as well as most other CMSes and the like). Joomla 3.x ships with a robots.txt
that allows indexing of the root, the /media folder and the /templates folder as these contain file necessary for "correct" page rendering.
Now, the format of robots.txt
is not actually a well defined beastie, but broadly it contains one or more "groups" of URL specifications, each "group" corresponding to a specific user-agent (for most purposes here read that as "search engine bot").
In my experience, most robots.txt
files only have one "group" for the user-agent which is *
(i.e. every type of user-agent). Within these "groups", are a sequence of lines each specifying an "action" and then a path (be that just a directory e.g. /admininistrator/
or a set of files e.g. /images/*.html
or a specific file /administrator/index.php
). Example extract:
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Now, the "problem" arises because the only universally agreed "action" is Disallow
, that is "please user-agent, don't index the path specified to the right". I use that phrasing because robots.txt
is only a request mechanism, that is its actions are not binding, the visiting agent can choose to honour the actions or ignore them.
As it happens, the Googlebot does permit the use of an Allow
action, but at present it is the only one of the search engines to state that this is the case, so I suggest you don't rely on it.
In essence, the problem boils down to: you don't want the "internal" files that make Joomla work to be indexed, but equally you don't want to reveal to those with malintent too much about your website's internal structure - these are not quite mutually exclusive, but are not far off.
With having to rely wholly on Disallow
, it means I think you don't have many options to try to arrive at a robots.txt
that is manageable, doesn't tell the malintents too much but does result in Google et al indexing only what you want.
The approach I have taken (don't do this at home folks) is to patch /libraries/joomla/document/document.php
to subtly alter the behaviour of the addScript
and addStyleSheet
methods. For me, what they do now is to check if there is a version of the specified script/stylesheet in a place like /media/<original path>
and also to test if there are minified versions of .js
and .css
files (minification is a whole other topic). I then have a script that I run regularly that checks everywhere that is Disallow
ed in the robots.txt
that contains media files and copy these to the same type of URL, but under /media
.
So, as it happens, I use Gantry too, but now the gantry files are copied to /media
and as my robot.txt
doesn't include any reference to /media/
the resources necessary for "correct" rendering are not blocked but equally I have a minimal and manageable robots.txt
file.