There is no simple answer to this as exact performance would depend on a number of factors, including number of modules, menu items and installed plugins. The important thing to understand is that modules are not executed until actually needed. I.e. if module is assigned to a position not present in the template, it will not be rendered. So there is at least no major performance impact. Plugins that heavily process modules during
onAfterModuleList event could affect performance though.
There is also a trade off when using menu assignments. This is because for every menu item assigned/unassigned a database entry is made in
#__modules_menu table. This could affect performance when loading the initial module list. And when using "On all pages except those selected" assignment mode, plugins with
onAfterModuleList could be greatly affected since the module list contains duplicate instances of the same module, one for each menu item unassigned.