I have created more that hundreds of websitewebsites using Joomla!
In most cases people think that Joomla is working heavily on the server, but according to the analizes done by me, the most important thing to keep your website FAST is to OPTIMIZE the IMAGES and JS/CSS.
Don't use Joomla editors, like JCE, to just change the size of the image. Use image editors like photoshop or any other editor to resize your images to the size it's needed!
Use JS/CSS compressing tools.
We always use Pingdom Tools to analyze the website speed and we use T3 Framework's built in feature to optimize JS/CSS!