A quick search brings up this result which has a lot of seemingly good comments about how to increase the memory for the SQL query, however there is also advice suggesting to review the Query you are running.
https://stackoverflow.com/questions/29575835/error-1038-out-of-sort-memory-consider-increasing-sort-buffer-size
So I would check what size memory you already and unless it is below the default look at the query.
As the message is a SQL issue it is not something you can expect to change in the Joomla options.
If you are just looking at the list of modules under the Admin view then it is possible you have too many modules created and that the default Query is running out of storage. I worked on a problem a few years back where the site had over 1'000 modules created and that would run out of memory when they tried to update some modules.
If you need more help you will need to provide more information about your environment.