I tried this out and this worked for me:
<?xml version="1.0" encoding="utf-8"?>
<form>
<fields name="params">
<fieldset name="groups" label="Group settings">
<field name="group_id" type="list" default="" label="Select group:">
<option value=""></option>
<option value="1">Group 1</option>
<option value="2">Group 2</option>
<option value="3">Group 3</option>
</field>
</fieldset>
</fields>
</form>
I put it in the admin /forms folder and called the file category.two.xml (where 'two' is my category_name).
The other approach you might want to consider is to enable custom fields on your component, then you can add additional fields to your component's Category entity. How to enable them is described in the Joomla 3 MVC Tutorial Adding Custom Fields step. The work to do it in Joomla 4 is similar - you just have to move the getContexts() and validateSection() methods into your Extension class, and ensure that your extension class implements Joomla\CMS\Fields\FieldsServiceInterface.