After some more googling and debugging, I figured it out. The trick was that I'm using a JForm and the file field can be accessed slightly differently. First you need to access the fields tag and then the file field itself (see the bold line below):
The form xml:
<fieldset name="basic" addfieldpath="/...