Using the following directory as a root for all invoices is perfectly fine:
administrator/components/com_mycomponent/invoices
However, what I would recommend doing is hashing the file name once the invoice is generated. Like so:
administrator/components/com_mycomponent/invoices/HASHED_FILE.pdf
Either that or, for each invoice, create a hashed directory and store the PDF in there, like so:
administrator/components/com_mycomponent/invoices/HASHED_DIR/file.pdf
Update:
If this method is chosen, you can use a htaccess file to prevent direct access and only allow PHP driven access to these PDF files.
For a PHP driven download, I would suggest storing the hash in a database tabling, along with the user's ID. When a request is made to download the file, you can then match the ID you've stored with the ID of the logged current logged in user, and of course, match the hash stored in the database.