It sounds like your default "Home" menu item (marked with a star in the list of menu items) is set to the wrong access level.
Make sure the menu item is set to "Public" access.
In addition, if the menu item points to a single article, the article must also be set to public access.
Update:
It's also possible that there is a problem with your ACL (Access Control List) settings, since the page works fine when logged in as administrator, but not for guest users. It's hard to say exactly what's wrong without seeing the site, but if possible I suggest creating a separate installation of Joomla and compare the settings.
Here are some places in the admin area where you might want to check the settings:
Users → Manage → (Options button in upper right corner)
Make sure "Guest User Group" is set to "Guests"
Users → Access Levels
Check that your viewing access levels are similar to this screenshot, particularly for Public and Guest:
Check if you have any third-party extensions installed that affects access levels, like membership extensions or similar. If so, disable the extension in Extensions → Manage → Manage (don't uninstall, just disable by clicking the green checkmark.
Update Joomla and all your extensions.
This might not solve the issue, but you should always use the latest version of Joomla (currently 3.6.2) and all extensions, they contain security and bug fixes.