Brief Overview
My Joomla website has a strange problem with search engine robots. I checked ALL the standard things a webmaster can do to make sure it is not a trivial issue with robots.txt file or other search engine optimization parameters. My university's IT folks are clueless of what's going on and so are the people who tried to help on Joomla's forums.
I asked this question on Pro Webmasters stackexchange too. But I am starting to suspect that there may be some hidden setting somewhere in Joomla causing the issue. So, I am asking this question here to see if there is a setting I am missing somewhere.
The actual problem starts here.
The website in question is: http://gsa.ece.umd.edu/. It runs using Joomla 2.5.x (latest). The site was up since around mid December of 2013, and I noticed right from the get go that the site was not being indexed correctly on Google. Specifically I see the following message when I search for the website on Google:
ECEGSA - University of Maryland
A description for this result is not available because of this site's robots.txt – learn more.
The thing is in December till around March I used the default Joomla robots.txt file which is:
User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/
Nothing there should stop Google from searching my website. And even more confusingly, when I go to Google Webmaster tools, under "Blocked URLs" tab, when I try many of the links on the site, they are all shown up as "Allowed". I then tried adding a sitemap, putting it in the robots.txt file. That did not help. Same exact search result, same behavior in the "Blocked URLs" tab on the webmaster tools. Now additionally, the "sitemaps" tab says for several links an error saying "URL is robotted out". I tried those exact links in the "Blocked URLs" and they are allowed!
I then tried deleting the robots.txt file. No use. Same exact problem.
Here is an example screenshot from Webmaster tools.
At this point I cannot give a rational explanation to why this is happening and neither can anyone in the IT department here. No one on Joomla forums can seem to understand what is going on.
Does anyone know if there is a possible conflict in the Joomla site configuration based on what is described above?
Redirect /ecegsa http://gsa.ece.umd.edu # Sitemap rewriting URL RewriteCond %{REQUEST_URI} ^/sitemap.xml RewriteRule .* /index.php?option=com_xmap&id=1&view=xml ErrorDocument 404 /404