What can I do, other than using a cache plugin like jotcache, to improve the server response time of my website?
I have already removed all inactive articles modules, plugins and components, is there something else I can do to improve it?
I am talking about the time before the client (browser) gets a response to his request. And not the time the page loads.
There are several steps you can take to increase server side performance. Do note that some of these will not work with most shared hosts. For great performance you usually need more fine grained control.
As with any optimizations, you should measure the effect it has on your site's performance. For basic optimizations are tools like PageSpeed and YSlow helpful (I recommend the Firefox versions). For a more in-depth understanding you'll have to measure the actual response times of your live system over a period of time.
Enable Opcode Caching
This is probably the single biggest thing you can do to increase you site's performance. I recommend either using OpCache or APC. The configuration of these caches can be tweaked for even greater performance based on the servers and sites specific needs, but just installing is already a huge boost.
Update PHP version
Newer PHP version regularly come with performance improvements. Additionally there are a number of cases where Joomla can use better code when running on a newer PHP version. As of today you should not be using a PHP version older than PHP 5.4 if you care about performance.
PHP Settings
On older PHP versions (<= 5.3) take care to turn off register_globals and magic_quotes_gpc for a free performance win. On PHP >= 5.6 change always_populate_raw_post_data to -1.
PHP extensions
Joomla contains some code to workaround missing PHP extensions. This will always be slower than those extensions. For a decent performance improvement install the following extensions: mbstring, zip, ftp
Enable gzip
Enabling server side compression trades bandwidth for CPU resources. For most sites out there this is the right call. Be careful to only compress text data, compressing binary data will give you the CPU hit without the bandwidth advantages. You can also enable compression in the Joomla settings but it's better if you let the server deal with this.
Precompress static resources
I'm not talking about minfication (though this helps too) but you can configure Nginx and Apache to server already compressed resources, effectively eliminating the CPU hit from gzip compression. Bonus points: Use Zopfli for even smaller files.
Use FastCGI for PHP
More specifically you should use php-fpm. This prevents starting up PHP when the browser requests a non-PHP file.
Use appropriate cache headers
Using Expires header can greatly reduce the the server resources required. After all the fastest request is the one never done.
Generally I'd recommend using Nginx for high performance sites. If you absolutely do need to use Apache for some reason I got a bonus tip:
Disable AllowOverride
Setting AllowOverride to None prevents a lot of disk IO and
Do not use the FTP Mode
While it doesn't affect your users reading the site, it will slow down some operations like extension installation.
Turn off unused extensions
This goes especially for Plugins, many of these are run on every request. Disabling menus and components is mainly helpful when listing available extensions and menu types. Also take 5 minutes to go trough the extensions that come with the Joomla core and disable those you don't use.
Turn off Prepare content
If you use a lot of custom modules (mod_custom not custom as in self written) check if you can disable Prepare content in those. This saves some processing.
Use Caching
If you're serious about performance you'll want to enable caching. Especially useful for read heavy sites with many anonymous users. Logged-in users will bypass most of the cache, if you have a lot of these the effect won't be as big.
Use an alternative session handler
By using an alternative session handler you can free your database for more important things. I'd not recommend file (confusingly called None in the Joomla settings) as that will bring once again increase disk IO. Using APC is potentially the fastet but increases the risk of loosing all session data. The right middle ground is Memcache. While this requires an additional process, it's more reliable than APC and much faster than storing all the data in the database
Keep your site simple
Do you really need that Facebook button? What about that cute icon that no one ever sees because it's partly hidden behind some banner? The less your site displays the faster it will be. Also wherever possible use CSS3 instead of images.
Set a favicon
Thank to a decision dating back to Internet Explorer 4.0 browsers make a request to a file called favicon.ico whenever a user visits your site. If you don't have one this causes unnecessary IO, as 404 responses are not cached. Either specify your site's icon with a HTML meta tag or add a file called favicon.ico to the web root. Don't want a favicon? Place a transparent PNG with a size of 1x1 pixel in that location. Note that the extension still needs to be .ico.
Optimize your images
It's amazing what kind saving one can get by optimizing images, especially PNG files. There are a variety of tools available for any operating system, personally I use ImageOptim which is available for OS X. Joomla's core images are already optimized but go trough your own images, especially templates are important. You could also check the extensions you are using. If they're open source, why don't you contribute a pull request containing newly optimized images?
If you really wanna go all out, consider putting Varnish in front of your site. It caches requests to your site and if possible serves them from it's own cache, never hitting Joomla. It's a complex tool but with a lot of power. Joomla won't take advantage of advanced functionality like Edge Side Includes (ESI) but your own code could.
One BIG thing you can do to improve response times for your site is to update your module settings. Custom modules (using mod_custom) have an option to enable / disable the preparing of content. This allows you to run content plugins on the data content within your custom module. It's powerful when used properly, but it does add additional rendering time to the module output. Unless you are actually making use of a content plugin within your custom module content, it's best to turn that off. You'll see an decrease in module rendering time when you do this.
Screenshot of the setting: 
Note: If my answer helped you, please click the check to the left to accept the answer.
I assume that you have run your site through Google Page speed, and you are indeed looking at reducing the page response time i.e. time taken by your markup to be sent to the browser.
The most likely culprits for these are heavy queries. Turn on the debug and look at the bottom of the page to find out which are the most heavy queries and remove or reduce them. You can also take help of your MySQL server's slow query logs.
The next thing that you can look at is caching. Depending upon the kind of site you have you can turn of conservative or progressive caching. Progressive is to be used on sites that don't have and user related features.
The things I know of that can help (most of which have been listed) are these:
I've used a combination of the built in Joomla cache (or even APC enabled on the server) and plugins like gzip or rocket theme's rockbooster plugin. I also use nonumbers CDN plugin with an amazon account.
It definitely helps, but you have to watch your site and disable the cache for components that have issues. I had to bypass CiviCRM on our site because it didn't play nice with caching. Joomla comes with a cachecontrol plugin that can handle that.
You will want to also look into reducing local assets and proper caching. One thing that you can do is use CDN plugins. Here are two I would recommend:
'corePHP' jomCDN - https://www.corephp.com/joomla-products/jomcdn
NoNumbers CDN For Joomla! - http://www.nonumber.nl/extensions/cdnforjoomla
Also if none of the above seem to resolve the issues you may have a deeper issue - possibly within your template, the server itself, or other things. If that is the case it might be best to hire someone to hunt down the issue and resolve it for you.
You also can use this code in your .htaccess file:
RewriteEngine On
RewriteCond %{HTTP_HOST} ^www.your domain [nocase]
RewriteRule ^(.*) http://your domain/$1 [last,redirect=301]
RewriteEngine on
<ifmodule mod_deflate.c>
AddOutputFilterByType DEFLATE text/text text/html text/plain text/xml text/css application/x-javascript application/javascript text/javascript
</ifmodule>
<IfModule mod_expires.c>
ExpiresActive On
ExpiresByType image/jpg "access 1 year"
ExpiresByType image/jpeg "access 1 year"
ExpiresByType image/gif "access 1 year"
ExpiresByType image/png "access 1 year"
ExpiresByType text/css "access 1 month"
ExpiresByType text/html "access 1 month"
ExpiresByType application/pdf "access 1 month"
ExpiresByType text/x-javascript "access 1 month"
ExpiresByType application/x-shockwave-flash "access 1 month"
ExpiresByType image/x-icon "access 1 year"
ExpiresDefault "access 1 month"
</IfModule>
I have created more that hundreds of websites using Joomla!
In most cases people think that Joomla is working heavily on the server, but according to the analizes done by me, the most important thing to keep your website FAST is to OPTIMIZE the IMAGES and JS/CSS.
Don't use Joomla editors, like JCE, to just change the size of the image. Use image editors like photoshop or any other editor to resize your images to the size it's needed!
We always use Pingdom Tools to analyze the website speed and we use T3 Framework's built in feature to optimize JS/CSS!
Have you removed unused plugins? Joomla has to load and process every published plugin for events triggered in a request.
You have to understand the extensions you've installed and how they interact with Joomla!
Many extensions install plugins that manage content, system, or user activities to initiate code functions. Some of them may be superfluous.
I know that k2 installs many plugins that respond to language related events that produce no real output for single language sites. Unpublishing those plugins can reduce processing time.
Maybe not the answer the OP was looking for but worth mentioning in case it's not obvious to everyone (and no other answers have mentioned this yet): improve server resources such as memory, CPU and disk speed.
If you are using shared web hosting, upgrading from the "economy" plan to the "business" plan or another more advanced plan may improve server resources available to your website and reduce the server response time. Moving to a better quality hosting company (e.g. with less crowded shared web hosting servers) might also help.
A better hosting plan might also give you access to performance enhancements that are not included in the basic hosting plan. For example, the "GrowBig" and "GoGeek" SiteGround hosting plans include dynamic cache which helps reduce server response time significantly. The dynamic cache option is not available in the "Startup" plan.
