Improve Server/Web Hosting Plan Resources
Maybe not the answer the OP was looking for but worth mentioning in case it's not obvious to everyone (and no other answers have mentioned this yet): improve server resources such as memory, CPU and disk speed.
If you are using shared web hosting, upgrading from the "economy" plan to the "business" plan or another more advanced plan may improve server resources available to your website and reduce the server response time. Moving to a better quality hosting company (e.g. with less crowded shared web hosting servers) might also help.
A better hosting plan might also give you access to performance enhancements that are not included in the basic hosting plan. For example, the "GrowBig" and "GoGeek" SiteGround hosting plans include dynamic cache which helps reduce server response time significantly. The dynamic cache option is not available in the "Startup" plan.