0

I use Joomla 4.x, then I inspect the google search console and it only has a index.php crawled but any links shown in index.html are ignored, not followed.

this is the original robot.txt right from Joomla installation

# If the Joomla site is installed within a folder
# eg www.example.com/joomla/ then the robots.txt file
# MUST be moved to the site root
# eg www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to all of the
# paths.
# eg the Disallow rule for the /administrator/ folder MUST
# be changed to read
# Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# https://www.robotstxt.org/orig.html

User-agent: *
Disallow: /administrator/
Disallow: /api/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

I want google to crawl my website other than disallow URL, thanks

1 Answer 1

1

In System > Global Configuration > Site make sure that under Metadata > Robots is set to "index,follow" in the dropdown.

3
  • I did set to INDEX FOLLOW, but doesn't work, is it related to .htacces ?, but I know nothing about .htacces Commented Dec 7, 2022 at 23:06
  • @dhanysaguntoro If you think it might be related to .htaccess then you should update your question with the contents of your .htaccess file. However, there is nothing normally in .htaccess that would affect crawling.
    – MrWhite
    Commented Apr 28, 2023 at 10:51
  • I created a new Folder inside Joomla Installation, the Files within this new folder are indexed normally by Google, and the files I mention here are PHP FILES with SQLite as a database, while Joomla uses MySQL, is it possible that the Problem is with MySQL server? Commented Apr 30, 2023 at 1:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.