In addition to Shyam's answer...
Global Configuration ->
- Conservative caching is a cache generated for each view of a component / module
- Progressive caching is a cache generated for each view of a component / module on an individual per-user basis
This is an extension-level cache of views. It must be implemented by the extension developer to have any effect.
Note: logged in users will never see cached content via these means.
You can override individual module caching if 'Conservative caching' is enabled by disabling caching in the module's advanced options. You CANNOT override module caching on a per module basis if 'Progressive caching' is set.
The System - Cache plugin generates page caches of each whole page in
For most scenarios, enabling the System - Cache plugin is sufficient. If you only want to cache particular modules / components (which have caching implemented), disable the plugin and enable Conservative caching in the global configuration. You can then set exceptions by disabling caching in a module's advanced options.
Note that any caching is a balance between time and space - full page caching will reduce DB load and hence page load times, but will increase disk space usage.
You can find further analysis and benchmarks of the caching system here, as well as the Joomla Cache docs for admins and this excellent article detailing the various options and applicable caveats.