5

Brief Overview

My Joomla website has a strange problem with search engine robots. I checked ALL the standard things a webmaster can do to make sure it is not a trivial issue with robots.txt file or other search engine optimization parameters. My university's IT folks are clueless of what's going on and so are the people who tried to help on Joomla's forums.

I asked this question on Pro Webmasters stackexchange too. But I am starting to suspect that there may be some hidden setting somewhere in Joomla causing the issue. So, I am asking this question here to see if there is a setting I am missing somewhere.

The actual problem starts here.

The website in question is: http://gsa.ece.umd.edu/. It runs using Joomla 2.5.x (latest). The site was up since around mid December of 2013, and I noticed right from the get go that the site was not being indexed correctly on Google. Specifically I see the following message when I search for the website on Google:

ECEGSA - University of Maryland

A description for this result is not available because of this site's robots.txt – learn more.

The thing is in December till around March I used the default Joomla robots.txt file which is:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /images/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /media/
Disallow: /modules/
Disallow: /plugins/
Disallow: /templates/
Disallow: /tmp/

Nothing there should stop Google from searching my website. And even more confusingly, when I go to Google Webmaster tools, under "Blocked URLs" tab, when I try many of the links on the site, they are all shown up as "Allowed". I then tried adding a sitemap, putting it in the robots.txt file. That did not help. Same exact search result, same behavior in the "Blocked URLs" tab on the webmaster tools. Now additionally, the "sitemaps" tab says for several links an error saying "URL is robotted out". I tried those exact links in the "Blocked URLs" and they are allowed!

I then tried deleting the robots.txt file. No use. Same exact problem.

Here is an example screenshot from Webmaster tools. Index Status for my website showing no crawl errors and no blocked URLs, in direct contradiction to what the sitemap tab says and what the search result says! FRUSTRATION!

At this point I cannot give a rational explanation to why this is happening and neither can anyone in the IT department here. No one on Joomla forums can seem to understand what is going on.

Does anyone know if there is a possible conflict in the Joomla site configuration based on what is described above?

  • Does it reside in the root or in a subfolder? – FFrewin May 25 '14 at 8:28
  • Have any changes been made to the htaccess file? – Lodder May 25 '14 at 10:05
  • @FFrewin no, it is in the main directory. Some minor additions were made to the .htaccess, but I believe I noticed these issues even before the changes were made: Redirect /ecegsa http://gsa.ece.umd.edu # Sitemap rewriting URL RewriteCond %{REQUEST_URI} ^/sitemap.xml RewriteRule .* /index.php?option=com_xmap&id=1&view=xml ErrorDocument 404 /404 – lite-whowantstoknow May 25 '14 at 15:30
  • This question has an answer on Webmasters: webmasters.stackexchange.com/questions/63156/… – MrWhite May 25 '14 at 21:23
  • @w3d Yes, indeed. I managed to get an answer at webmasters. I am currently seeing if that answer does indeed explain what is going on or not. Perhaps this is no longer a Joomla specific concern, although a take home message seems to be "be careful with using 3rd Party login plugins for things like CAS, as they may do 3xx redirects and may cause this issue". I still need to ensure if indeed that is the problem. – lite-whowantstoknow May 25 '14 at 23:34
2

UPDATE: Issue has been resolved by correcting extension configuration

I have used a plugin called externallogin to allow my users to login using the University CAS (Central Authentication Server). It turns out that this plugin is adding a 303 redirect to the HTTP Header response to ALL my pages, due to a configuration error.

So, as a consequence none of my pages are getting indexed properly as Google either (a) penalizes 303 redirects or probably because (b) the page to which the 303 redirect points to (https://login.umd.edu/) has a robots.txt that disallows all bots.

FIX
To fix this issue, if you ever encounter it, you must correct the configuration of the plugin as follows:
1. In the extensions menu, under "External Login>", go to the server configuration.
2. In the connections tab, for "Automatic Login/Logout", choose 'No'. The error occurs if you choose 'Yes' in step 2, since in this case, the extension automatically adds a 303 redirect to all pages of your website, to check whether a user has been logged in to the CAS in the browser session, and if so it automatically logs you into the site. This feature will cause search engine indexing problems.

Acknowledgements
Thanks goes to @stephen-ostermiller (Relevant answer on Pro Webmasters) for directing me in the right direction by making me realize that it was a 303 redirect added by the extension.

  • Yep. Removed the plugin and now my site is indexed properly! This is the definitive solution! – lite-whowantstoknow May 26 '14 at 0:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.