If you have access to SSH try to use top/htop (may require root). This can tell you the cpu/ram usage that can point at server related issue.
Also try to create a file with php info
Create this file and load it, does it load slow? That would point to the server as the main cause, if it loads fairly fast then it could be something to do with Joomla.
Another test is install a blank Joomla 3.3 site and see how it loads with full defaults.
If you have cpanel/plesk you can look into the resource usage as well, but both have different ways of doing it and can be dependent on plugins, so you would be on your own there but most likely there will be something to get that information.
If it ends up being server related the only hope is to either rebuild the vps (quicker sometimes) or try to fix it (update modules, check configs, ect).
If it is Joomla then it could be the php.ini settings or database (try an akeeba restore). If its php.ini in the extension manager and warnings you should see some sort of message about a setting that could be better.
However there is one last thing it could be, network.
5 minute load times would normally hit the php max_execution_time, so that makes me think its network connection. There is no way to really fix it other then time or talking to your host. It could also be the area you are in compared to the server (for that cloudflare or some other CDN service could help).
There are just some troubleshooting tips though, no guarantee that one of them is the answer, but a good place to start.