Since there are no additional comments or answers, I will assume my approach was valid and I will post it here as an answer.
After further digging I found out that FormBehaviorTrait dispatches event onPrepareForm which is perfect for loading the field prefix.
The downside, or rather the inconvenience, of this approach is the requirement to create a getFormFactory method. This is not a big deal, but also not really something I'm willing to spend my time on.
I copied the dispatch part from preprocessForm method of the trait into my component's model and added the prepareForm event method in the plugin. This way worked for me.
