You need to create a template override for the article view - default.php
template file.
So there you can create a "modal button
" for the register link, to open the registration view inside a modal window.
This blog post explains how to create modals in Joomla 3. It covers the bootstrap way for Joomla 3 with many examples, but also has a reference to an older article with examples using the mootools way - which still works on Joomla 3 by the way.
Regarding the loading of the registration screen, I would suggest to add the tmpl=component
in the link, so it will only load the component output and not the whole template.
So the link to the registration page will become like:
<?php $link = new JUri(JRoute::_('index.php?option=com_users&view=login&Itemid=' . $itemId .'&tmpl=component', false)); ?>
You will also might want to deal with the an auto-closing of the modal window after successful login and refresh of the current window.
This could be achieved with some JavaScript, but I haven't tried that with the bootstrap way so I dont't have the answer on this ready and it's late here to go and look for it. But if you manage to reach up to this point, then the remaining steps will come as well.
An alternative way to achieve that is to try and load a popup Ajax login module instead of linking to the registration page.
Check this module: SC Login.
See this answer here that describes how to load a module from inside a component: Displaying more articles in the article view
- Again I haven't tried this solution with the popup login module, but there is big chance it will work.