4

I run the server where this Joomla! site is hosted, so I have root access to it. There is something in the configuration, or somewhere in its code, that keeps browsing the site itself. If I run

# tail -f /var/log/apache2/softwareliberopinerolo.org-access.log

I see several logs of accesses from the server IP address to the website itself. They come at average peace of about 5 per second, steady, with almost no significant peaks or rests.

Here is an excerpt of what I see:

10.7.33.103 - - [05/Jan/2019:08:22:04 +0100] "GET /index.php/masky/6/howto/news/howto/howto/5-incontri/templates/beez5/css/general_mozilla.css HTTP/1.1" 404 10141 "https://softwareliberopinerolo.org/index.php/masky/6/howto/news/howto/howto/5-incontri/news" "Serf/1.3.9 (mod_pagespeed/1.13.35.2-0)"
10.7.33.103 - - [05/Jan/2019:08:22:04 +0100] "GET /index.php/masky/6/howto/news/howto/howto/5-incontri/media/system/js/mootools-core.js?65367038804095b968188728ff07f5e3 HTTP/1.1" 404 10095 "https://softwareliberopinerolo.org/index.php/masky/6/howto/news/howto/howto/5-incontri/news" "Serf/1.3.9 (mod_pagespeed/1.13.35.2-0)"
10.7.33.103 - - [05/Jan/2019:08:22:04 +0100] "GET /index.php/masky/6/howto/news/howto/howto/5-incontri/media/system/js/core.js?65367038804095b968188728ff07f5e3 HTTP/1.1" 404 10093 "https://softwareliberopinerolo.org/index.php/masky/6/howto/news/howto/howto/5-incontri/news" "Serf/1.3.9 (mod_pagespeed/1.13.35.2-0)"
10.7.33.103 - - [05/Jan/2019:08:22:04 +0100] "GET /index.php/masky/6/howto/news/howto/howto/5-incontri/media/system/js/mootools-more.js?65367038804095b968188728ff07f5e3 HTTP/1.1" 404 10095 "https://softwareliberopinerolo.org/index.php/masky/6/howto/news/howto/howto/5-incontri/news" "Serf/1.3.9 (mod_pagespeed/1.13.35.2-0)"
10.7.33.103 - - [05/Jan/2019:08:22:04 +0100] "GET /index.php/masky/6/howto/news/howto/howto/5-incontri/templates/beez5/javascript/hide.js HTTP/1.1" 404 10225 "https://softwareliberopinerolo.org/index.php/masky/6/howto/news/howto/howto/5-incontri/news" "Serf/1.3.9 (mod_pagespeed/1.13.35.2-0)"
10.7.33.103 - - [05/Jan/2019:08:22:04 +0100] "GET /index.php/masky/6/howto/news/howto/howto/5-incontri/templates/beez5/javascript/md_stylechanger.js HTTP/1.1" 404 10225 "https://softwareliberopinerolo.org/index.php/masky/6/howto/news/howto/howto/5-incontri/news" "Serf/1.3.9 (mod_pagespeed/1.13.35.2-0)"
10.7.33.103 - - [05/Jan/2019:08:22:04 +0100] "GET /index.php/masky/6/howto/news/howto/howto/5-incontri/templates/beez5/favicon.ico HTTP/1.1" 404 10093 "https://softwareliberopinerolo.org/index.php/masky/6/howto/news/howto/howto/5-incontri/news" "Serf/1.3.9 (mod_pagespeed/1.13.35.2-0)"

Please note that:

  1. 10.7.33.103 in the logs is the source IP address of the requests and it happens to be the real IP address of the Apache server itself too, which is behind a NAT
  2. Almost every request gets a 404 because it's looking for non existing URLs. Once in a blue moon there is a request from 10.7.33.103 that hits a existing page.
  3. Beez5 template was set as default for a few seconds only to grab this log, in order to show that the URLs are being computed at runtime and they are not (fully) stored anywhere. However the same behavior happens with the site usual template, but in that case I get no beez5 in the logs.
  4. It's a up-to-date Joomla! 3.9.1 with no extension installed, except the ones that came bundled with official Joomla! package, and except the template (which is not the culprit, because of point 3 here above). However the site was born years ago and more or less regularly updated.
  5. Serf/1.3.9 and mod_pagespeed in the user agent string tell us it's actually Apache starting the HTTP request, which confirms the source IP address in the logs is correct. However there are no admin scripts in the server, nor exotic Apache configurations I know of, that index the site. The same Apache server hosts other sites as well, all configured the same as this one, and none of them exhibit such behavior. mod_pagespeed is not to blame, I've already tried disabling it to no avail.
  6. I already looked for malicious scripts months ago, found some and removed them. I'd argue there are leftovers, but I've also reinstalled clean Joomla! files over the existing ones afterwards, so I suspect this isn't a security related problem, because I do not know Joomla! enough to exclude configuration problems.

There is something that is not working as intended for sure. If you follow the Iscriversi all'associazione link in the side menu, you see it leads to the following wrong URL:

https://softwareliberopinerolo.org/index.php/howto/14-openoffice/eventi/eventi/6-eventi-futuri/news/news/arduino/iscrizione

where you get the correct page contents, but no CSS/JS or whatever other page requisites. howto, openoffice, eventi, news and so on are all categories inside the CMS, but that page does not belong to any of them. The menu item type is Single Article and the URL is being generated by Joomla!. Other menu items are working only because I've modified their type to custom URL and I've manually entered the correct URL.

EDIT: I had to deactivate the site, because it was taking up all the server RAM. If you click the link it won't work, but you can trust my description of what you'd see if it worked.

What is browsing the site? Is there any cache configuration in Joomla that can cause such behavior? Or is the site still hacked? How do I find out?

EDIT 2: I've tried the following procedure, which I think excludes any chance to blame malware leftovers, unless Joomla! executes any code it loads from the database:

  1. Downloaded a clean Joomla 3.9.1 package
  2. Removed installation folder immediately, before even actually using it
  3. Copied over configuration.php file from my site and NOTHING ELSE
  4. tested it: the same problem is still there and the site keeps browsing itself. Please note: I did NOT copy over the template, nor the images, so it's only Joomla 3.9.1 original code with my site database.

I think this pretty much shows that Joomla either browses itself server side by design, under some configuration, OR it has some feature/bug that makes it execute arbitrary code it loads from the database, and in my case that code was injected by some malware in the past. Do you agree?

Improve this question
10
  • I masked your IP and folder configuration because I felt it was a possible security risk. You should be aware of this in the future. It will still appear in previous edits I'm not sure if that can be changed.
    – Eoin
    Jan 5, 2019 at 18:55
  • thanks, but there's no point to mask the IP address when you know the domain name. Moreover that edit makes the question less clear to the network admins, because the original IP address was a typical non routable private IP, the current one is a random IP that wouldn't require any NAT. Please revert, thanks.
    – Lucio Crusca
    Jan 6, 2019 at 9:03
  • 1
    I've reverted it myself to the original IP
    – Lucio Crusca
    Jan 6, 2019 at 9:47
  • You don't have something like this <head> <meta http-equiv="refresh"content="30"> </head> do you?
    – Eoin
    Jan 6, 2019 at 13:00
  • @Eoin no, I have nothing like that. But, even if I had that, I think I'd see a different IP address as source in the logs, not the server IP, right?
    – Lucio Crusca
    Jan 6, 2019 at 16:14

1 Answer 1

Reset to default
3
+100

I do not see this as a Joomla issue (except the .htaccess file in Joomla root). From the small part of the log you provided in your question we can clearly read that the requests are made by mod_pagespeed and Serf as client. There is not really a question about it. The log identifies this user-agent info (Serf used by pagespeed). You cannot just simply avoid this in your access-log. This is what is creating a big amount of http requests on your site and gets 404 status response.

mod_pagespeed tries to optimize (or fetch) the site. However since there are a lot of 404 status response in your log, you should give attention to the following part of the Q&A section of pagespeed:

The most common reason that the rewritten resources 404 is because of mod_rewrite RewriteCond rules. For example:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^ /404 [L,R=301]

This rule causes 404 for all requests which don't exist on the filesystem, including mod_pagespeed rewritten resources and the mod_pagespeed admin console.

In order to fix this you must add an exception for mod_pagespeed URLs:

RewriteCond %{REQUEST_URI} !pagespeed

This will allow rewritten resources, the admin console and static resources necessary for some filters.

Thus you can fix the issue in the .htaccess file in Joomla root folder (for example).

And of course the reason you do not see this problem on your other sites is because on those there are no any problematic rewrite conditions modified by .htaccess which is influencing the behavior and responses on pagespeed/apache server (Or there is no conflicting caching turned on on the other sites).

You might want to study the Q&A more on this: https://www.modpagespeed.com/doc/faq#mod_rewrite

Also, you might want to check the caching of Joomla in Joomla admin System->Global configuration->System->system cache and turn that OFF since you do not need that and can cause conflict and 404 response to pagespeed/Serf theoretically, during fetching resources.

I hope this can help you on the question.

Additionally, I've never seen (in the last 13 years) that Joomla site is 'browsing itself' for a longer term (other than caching), nor from core, not from Joomla extentions, nor from any other malicious script. However that's one of what apache mod_pagespeed has to do primarily: to fetch resources using Serf (css, js, images) (If it would be otherwise you could see that specific source in the log too). So what you experience must derive from pagespeed (at least I cannot see other thing from your log). You should check that more deeply, turning pagespeed on and off and check the logs (or setting ModPagespeed standby) and also maybe configuring filters and rewrites more:

https://serverfault.com/questions/357918/apache2-access-log-thousands-of-logs-from-the-ip-of-the-same-server

https://www.the-art-of-web.com/system/mod-pagespeed-settings/

https://www.modpagespeed.com/doc/config_filters#RewriteRandomDropPercentage

Improve this answer
2
  • Thanks for not believing my words, clearly I made some mistake when I tried disabling mod_pagespeed, so I was sure that wasn't the problem. It now works.
    – Lucio Crusca
    Jan 10, 2019 at 12:27
  • @LucioCrusca - I'm glad that you checked it again and successfully solved the issue.
    – Zollie
    Jan 12, 2019 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.