I have to write front-end form in my component with dropdown list of categories and subcategories. Categories and subcategories amount should be unlimited. Should I use a nested sets model to do that?
If you implement your own categories functionality you can use anything that suits you.
However you can take advantage of Joomla's built-in categories system for your component.
Have a read here:
Developing an MVC Component/Adding categories
At the bottom of this page it shows you how to do something similar. https://docs.joomla.org/Creating_a_custom_form_field_type
It sounds like you may wish to do an override of the already existing categories form field.
Here's a full list