JCH Optimize is one of those plugins where you should not simply select everything to be optimised and expect your site to be 2x faster in loading speeds. The more you try and optimise, the more processing the plugin must do.
You may also reach a point, where there is no room for more optimisation.
What I would suggest you do first is only use JCH to minify the JS and CSS files, then compare the page load speed to your initial 200ms. If you see a performance increase, then try combining only JS or only CSS files.
I remember spending literally 3 full days testing loads of plugins, scripts, htaccess with different variations to get the best performance and noticed that over-doing it was not the solution.