Depending on the purpose, there can be some advantages in using a CMS compared to a static HTML website and this is why Joomla has been downloaded over 80 million times as at March 2017.
Content is not stored directly in HTML files in Joomla but is usually stored in tables in the Joomla database. You could use a tool like phpMyAdmin or similar to edit content directly but it is usually much easier to use the Joomla Administration tools such as the Article Manager and the Module Manager.
Directly editing the Joomla core code is usually a mistake because subsequent Joomla and third party extension updates can overwrite your changes and Joomla and third party extension updates need to be done regularly to keep Joomla websites secure.
The recommended way to use your own code in Joomla is to create an override.
Similarly, and depending on the template you are using, you can use a custom CSS file or a third party extension to override the styling.
Creating your own Extensions
Other ways to use your own code are to create a component, module, plugin or template.
You can install a third party extension such as the free or paid version of Regular Labs Sourcerer which enables PHP and other code to be inserted into an article or Custom HTML module.
Static HTML Website
Another option (but maybe not possible for this particular client) is to convert the site to a static HTML website and then edit the code directly. Of course, going down this route can't easily be reversed and you lose the advantages of a CMS.