I have created a custom backend button (JToolbarHelper::custom()), when a user clicks the button it shows a popup window with some content. How can I do so, not using JToolbarHelper::preview() as this button will do more than just displaying the content. Is there some inbuilt library function to do so?
Popup windows are part of core Joomla. What you need to do to activate functionality is:
"a.popuplink" can be replaced by any jQuery selector. Script used by Joomla is squeezbox. Also 3+ versions of Joomla you can use bootstrap modals, but it will be a little bit mode complicated.
anywhere in your view put
<a class="btn popuplink" href="http://joomla.stackexchange.com/posts/17068">Link that opens up popup styled like button</a> <?php JHtml::_('behavior.modal','a.popuplink'); ?>
Since you can't be sure which backend template is used you have to bring your popup code with you.
So: no, there is no inbuild library function (afaik).