I turned on SSL and now it won't turn off. I used the Joomla! global configuration and set SSL for the whole site. Now when I set it to admin only it stays on SSL mode.
Sometimes HTTPS redirections are stored in the local DNS caches of your computer, which will keep your browser redirecting to the HTTPS version of the website. So, consider doing a Google Search on how to clear the DNS caches of your Operating System.
Another issue that may arise sometimes which can cause redirection loops, may be caused by Cookies stored on your local machine. So, if there are persisting issues of redirections, it's recommended to also clear any cookies or other data that might have been stored locally on your browser.
I manually edited the configuration.php file and that resolved my issue. I'm now going to raise it as a bug.