Some modules and plugins come with Joomla and others are installed with component packages and are not used. Should the modules and plugins simply be disabled or completely uninstalled?
For the extensions shipped with core that I don't use, I disable them only. Because of how core updates are handled, those files often re-appear on sites come update day.
Many times, you can uninstall unused extensions that come as part of an extension package. If packaged as a proper package extension, all the core code will gripe about is if it can't find an extension come uninstall.
All that said, I'd just disable anything unused that came as part of a package to be safe.
From a security point of view we uninstall any unused extensions (components, modules, plugins and templates).
This reduces potential exposure to bad code, most of the sites we've been engaged to fix have been compromised by old, un-patched extensions (admittedly, that often weren't even disabled).
From an admin user point of view it also results in less confusion, less menu choices and a cleaner overall experience. (It's also prevents newer users from assigning menu's to the wrong template, or creating modules that aren't using the latest version).
This is a touchy subject, as sometimes when installing packages, then uninstalling the component it uses and leaving any plugins it may have used.
Currently as far as I know there is no way to know if something does not need a specific plugin to work. This could turn into a feature request for some sort of added xml to components to term what plugins/modules it should be with, allowing Joomla to see what is needed and what is not. Though that is just an idea, something that would be hard to push out to all third party devs to make sure it is used correctly.
The best thing to do is to keep good track of what is installed on your site, know what plugins a component needs and such. Typically a component developer will name a plugin in such a way that it can be easy to know its for that component, or you can just use the author for that. Modules typically can just be uninstalled with almost no harm if they are not used anywhere, typically they rarely rely on a plugin as well.
As for if its an unused Joomla extension, it should not be uninstalled, having it in a clean site means something needs it usually, so its best left alone. There is talk of splitting Joomla more to allow more "specialized" installs but its best to wait until then before removing anything from the core.
Regarding components, modules and plugins I usually disable them. What I completely remove are the unused templates, which take more space and also create many useless position options when I have to assign a module in a position.
Update July 2016
--- Completely removing the templates at that time wasn't ideal and in some cases there were issues and bugs after updates, but as far as I remember it was the only way to not have the module positions of those templates appearing in the module manager position select.
However, at recent Joomla versions, disabling a template it will hide its positions from position select drop down in module manager, so templates can be disabled.
If the extension has been decoupled like weblinks package, it can be uninstalled. For the other core extensions, I feel it's better to keep them installed and disabled.
All unused extensions must be uninstalled.
My advidce for Joomla Core Components, Plugins, Templates, Modules: Don't delete them, if you have always the newest Supported Joomla Version. But if you use for example Joomla 1.0 then delete everything...
For 3rd Party Extensions: Delete them if not used and check your database for entries from the extensions and delete them, too.