It's actually pretty simple.

In **view.html.php**, load modal library, if it's not already done:

    JHtml::_('bootstrap.modal');

For adding button to toolbar, use:
    
    // Toolbar object
    $toolbar = JToolBar::getInstance('toolbar');
    $layout = new JLayoutFile('joomla.toolbar.popup');
    
    // Render the popup button
    $dhtml = $layout->render(array('name' => 'test', 'text' => JText::_('Custom button'), 'class' => 'icon-archive'));
    $toolbar->appendButton('Custom', $dhtml);

`name` must match modal `id` after `-`. You can change `class` for different icons.

In **default.php** or in any file in `tmpl` folder add modal div, what will be shown after clicking on link

    <div class="modal hide fade" id="modal-test">
      <div class="modal-header">
        <button type="button" role="presentation" class="close" data-dismiss="modal">x</button>
        <h3>Modal title</h3>
      </div>
      <div class="modal-body">
        Modal content here
      </div>
      <div class="modal-footer">
        <button class="btn" type="button" data-dismiss="modal">
          <?php echo JText::_('JCANCEL'); ?>
        </button>
      </div>
    </div>