Likely the given information is not 100% accurate, so I 'll try to clear out a few things and reach to a conclusion.
Overrides can be made in many ways, there can be many combinations of configuration settings to deal with them.
Layout Override (global): for all items for a view - which is to just copy the view tmpl files to the respective folder in template's html folder. Then those files should be loaded instead of the default ones for any categories, disregarding.
Alternative Layouts: Similar to the above, but this time we rename the tmpl files and we have to specify on the item level what layout to use.
Alternative Layouts/Menu items - Copy and rename/edit also the tmpl xml file, so this layout can be selected/assigned with this new custom menu item.
From your description, I may assume that you have followed the 3rd option... but please clarify.
In this case, the layout will be used only for the category that you have selected with your custom menu item.
Trying to "trick" the sub-categories links with the
&layout=customlayout is a fair attempt and would make sense. Actually
&layout=template:customlayout would be closer - but this also won't work.
At some point in the past (a year or so ago), I went deeper with this and had concluded on how it works and why it couldn't work like this, but I don't remember clearly now the reasons why. My overall conclusion was that I had to create my custom MVC for it with various customizations here and there.
However, since this was long time ago, maybe some things have changed now.
But in any case with these in mind, it's still possible to achieve what you want and it can be fairly simple. It somewhat depends on your overall setup, but here what you can do:
Your Override Options
Still assuming you have followed the 3rd option from above, just clone the
customlayout.php inside your overrides folder and rename it to something like
If needed, create similar overrides for the other tmpl files of that layout. Then in your backend, go to the categories that you want to use this layout, edit them and in their
layout option (under the
Options tab) select this
customlayoutsubs.php. Now the sub categories will use this new layout, that is the same as your customlayout.php alternative menu item.
In case this is only section of a category blog you have in your site, then you don't need an alternative menu-item. Just create your override (first option) and let all categories use that.
In case you just need only some categories to use that layout - create only an alternative layout (not with menu item) and assign that layout to those categories.
Some final thoughts.
I feel the overrides abilities in Joomla do actually need some polishing and improvement. For example, assign to a parent category a layout override and being able to inherit this to its subs through settings, or deeper support on the topic for alternative menu items as described above, would be nice.
It would also make sense if there was an option on the menu item level to configure such settings.
Even so though and with the confusion all these may produce, there are still plenty of options to achieve great things.