With the recent Google update, a correctly configured Robots.txt file is even more important than before. The updated Robots.txt from Joomla now allows the templates directory (or rather does not Disallow it). But I've seen cases where Google wants access to /modules/ (and also /libraries/ for sites using Gantry framework).
Why does the default robots.txt file disallow these directories, when many extensions might require them?
To answer your question, I think you need to understand the role of robots.txt, which, in view of the very recent mobile friendliness changes in Google's algorithm, is something I have just been working on.
robots.txt exists to "request" that search engines and other such user agents do not index the specified contents of the website within the file. The vast majority of websites contain files that make no sense to be indexed as they are internal to the workings of the site (Joomla as well as most other CMSes and the like). Joomla 3.x ships with a robots.txt that allows indexing of the root, the /media folder and the /templates folder as these contain file necessary for "correct" page rendering.
Now, the format of robots.txt is not actually a well defined beastie, but broadly it contains one or more "groups" of URL specifications, each "group" corresponding to a specific user-agent (for most purposes here read that as "search engine bot").
In my experience, most robots.txt files only have one "group" for the user-agent which is * (i.e. every type of user-agent). Within these "groups", are a sequence of lines each specifying an "action" and then a path (be that just a directory e.g. /admininistrator/ or a set of files e.g. /images/*.html or a specific file /administrator/index.php). Example extract:
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Now, the "problem" arises because the only universally agreed "action" is Disallow, that is "please user-agent, don't index the path specified to the right". I use that phrasing because robots.txt is only a request mechanism, that is its actions are not binding, the visiting agent can choose to honour the actions or ignore them.
As it happens, the Googlebot does permit the use of an Allow action, but at present it is the only one of the search engines to state that this is the case, so I suggest you don't rely on it.
In essence, the problem boils down to: you don't want the "internal" files that make Joomla work to be indexed, but equally you don't want to reveal to those with malintent too much about your website's internal structure - these are not quite mutually exclusive, but are not far off.
With having to rely wholly on Disallow, it means I think you don't have many options to try to arrive at a robots.txt that is manageable, doesn't tell the malintents too much but does result in Google et al indexing only what you want.
The approach I have taken (don't do this at home folks) is to patch /libraries/joomla/document/document.php to subtly alter the behaviour of the addScript and addStyleSheet methods. For me, what they do now is to check if there is a version of the specified script/stylesheet in a place like /media/<original path> and also to test if there are minified versions of .js and .css files (minification is a whole other topic). I then have a script that I run regularly that checks everywhere that is Disallowed in the robots.txt that contains media files and copy these to the same type of URL, but under /media.
So, as it happens, I use Gantry too, but now the gantry files are copied to /media and as my robot.txt doesn't include any reference to /media/ the resources necessary for "correct" rendering are not blocked but equally I have a minimal and manageable robots.txt file.
The default robots.txt file currently looks like this:
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
As per @Mark Bradley's excellent answer the robots.txt file format is not well defined but if you are mainly concerned with allowing Google access to the resources it wishes to see, then I suggest amending the default robots.txt file to include allow statements for the folders that may possibly contain CSS and JS files like this or similar:
User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Allow: /components/*.css
Allow: /components/*.js
Disallow: /components/
Allow: /includes/*.css
Allow: /includes/*.js
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Allow: /layouts/*.css
Allow: /layouts/*.js
Disallow: /layouts/
Allow: /libraries/*.css
Allow: /libraries/*.js
Disallow: /libraries/
Disallow: /logs/
Allow: /modules/*.css
Allow: /modules/*.js
Disallow: /modules/
Allow: /plugins/*.css
Allow: /plugins/*.js
Disallow: /plugins/
Disallow: /tmp/
Notes:
Web crawlers look for the most specific or longest rule when determining which rule to enforce. For example, Allow: /modules/*.css overrides Disallow: /modules/
