I'm currently on Joomla 2.5. I've been hanging out for Joomla 3.4 to be released before migrating, but I can't wait any longer.
Will I be able to remove the weblinks component once I upgrade from Joomla 3.3 to 3.4?
Weblinks component is not a locked extension and can be uninstalled anyway, even in your current J2.5 site.
Note though, that despite the component is allowed to be uninstalled, doing so can possibly give you minor issues when updating/upgrading Joomla, mostly within the database because of the absence of the relative tables.