What can I do, other than using a cache plugin like jotcache, to improve the server response time of my website?

I have already removed all inactive articles modules, plugins and components, is there something else I can do to improve it?

I am talking about the time before the client (browser) gets a response to his request. And not the time the page loads.