Just a few quick thoughts:
I guess that you want to have a search module that will instantly show search results from a specific category. There are 3rd party search modules that are using AJAX to display results. Many of them might have options to specify categories from which to show the search results.
But usually the search results of these modules are displayed in the module and not in the component area, and if you want to have all results of a specific search, they usually take you either to the search component view, or if they work with their own component, to that components view.
However, even if the above is not what you are looking for, it might still be worthy to research on JED to see what is available there.
Content Category Page with Search/Filter Ability
If you want to have a content category page with a search field, that will filter instantly the displayed articles, based on the input of the field, then I guess you should customize the com_content.
Maybe the best option would be to create a complete MVC for this and have a new menu-item for your special category page, that will include a form with this filter/search input field on top and below that you will have the results display.
Then in your model you would have to add the search functionality - create the SQL query that will use the input from your search field, and that will give back the found results.
If you want your view to be updated instantly while typing in the search field, you will have to implement AJAX.