There are several steps you can take to increase server side performance. Do note that some of these will not work with most shared hosts. For great performance you usually need more fine grained control.
As with any optimizations, you should measure the effect it has on your site's performance. For basic optimizations are tools like PageSpeed and YSlow helpful (I recommend the Firefox versions). For a more in-depth understanding you'll have to measure the actual response times of your live system over a period of time.
Enable Opcode Caching
This is probably the single biggest thing you can do to increase you site's performance. I recommend either using OpCache or APC. The configuration of these caches can be tweaked for even greater performance based on the servers and sites specific needs, but just installing is already a huge boost.
Update PHP version
Newer PHP version regularly come with performance improvements. Additionally there are a number of cases where Joomla can use better code when running on a newer PHP version. As of today you should not be using a PHP version older than PHP 5.4 if you care about performance.
On older PHP versions (<= 5.3) take care to turn off
magic_quotes_gpc for a free performance win. On PHP >= 5.6 change
always_populate_raw_post_data to -1.
Joomla contains some code to workaround missing PHP extensions. This will always be slower than those extensions. For a decent performance improvement install the following extensions:
Enabling server side compression trades bandwidth for CPU resources. For most sites out there this is the right call. Be careful to only compress text data, compressing binary data will give you the CPU hit without the bandwidth advantages. You can also enable compression in the Joomla settings but it's better if you let the server deal with this.
Precompress static resources
I'm not talking about minfication (though this helps too) but you can configure Nginx and Apache to server already compressed resources, effectively eliminating the CPU hit from gzip compression. Bonus points: Use Zopfli for even smaller files.
Use FastCGI for PHP
More specifically you should use
php-fpm. This prevents starting up PHP when the browser requests a non-PHP file.
Use appropriate cache headers
Expires header can greatly reduce the the server resources required. After all the fastest request is the one never done.
Generally I'd recommend using Nginx for high performance sites. If you absolutely do need to use Apache for some reason I got a bonus tip:
None prevents a lot of disk IO and
Do not use the FTP Mode
While it doesn't affect your users reading the site, it will slow down some operations like extension installation.
Turn off unused extensions
This goes especially for Plugins, many of these are run on every request. Disabling menus and components is mainly helpful when listing available extensions and menu types. Also take 5 minutes to go trough the extensions that come with the Joomla core and disable those you don't use.
Turn off Prepare content
If you use a lot of custom modules (
mod_custom not custom as in self written) check if you can disable Prepare content in those. This saves some processing.
If you're serious about performance you'll want to enable caching. Especially useful for read heavy sites with many anonymous users. Logged-in users will bypass most of the cache, if you have a lot of these the effect won't be as big.
Use an alternative session handler
By using an alternative session handler you can free your database for more important things. I'd not recommend file (confusingly called None in the Joomla settings) as that will bring once again increase disk IO. Using APC is potentially the fastet but increases the risk of loosing all session data. The right middle ground is Memcache. While this requires an additional process, it's more reliable than APC and much faster than storing all the data in the database
Keep your site simple
Do you really need that Facebook button? What about that cute icon that no one ever sees because it's partly hidden behind some banner? The less your site displays the faster it will be. Also wherever possible use CSS3 instead of images.
Set a favicon
Thank to a decision dating back to Internet Explorer 4.0 browsers make a request to a file called
favicon.ico whenever a user visits your site. If you don't have one this causes unnecessary IO, as 404 responses are not cached. Either specify your site's icon with a HTML meta tag or add a file called
favicon.ico to the web root. Don't want a favicon? Place a transparent PNG with a size of 1x1 pixel in that location. Note that the extension still needs to be
Optimize your images
It's amazing what kind saving one can get by optimizing images, especially PNG files. There are a variety of tools available for any operating system, personally I use ImageOptim which is available for OS X. Joomla's core images are already optimized but go trough your own images, especially templates are important. You could also check the extensions you are using. If they're open source, why don't you contribute a pull request containing newly optimized images?
If you really wanna go all out, consider putting Varnish in front of your site. It caches requests to your site and if possible serves them from it's own cache, never hitting Joomla. It's a complex tool but with a lot of power. Joomla won't take advantage of advanced functionality like Edge Side Includes (ESI) but your own code could.