Very slow HTTP response times possibly due to SEF rules in htaccess

Question

Im running Joomla 3.X and can't seem to pinpoint why I'm getting such high HTTP response times.

I think I narrowed it down to the SEF rules in my .htaccess

I ran some tests and thought to post my results here hoping someone familiar with the Joomla architecture can lead me in the right direction.

Right now the server is on a push style CDN so 200 response OK pages are a non-issue.

Its the 404 pages that will always have a TTFB of around 1-2 seconds - testing here

https://speedvitals.com/ttfb-test

Most sites I check return a 404 in 100-300 ms

Heres what I have so far

I've setup a "rule" on my cdn to bypass the cache for a random url.

When I test the random url as is, these are the times.

If I remove the below SEF snippet from the .htaccess, the 404 response times improve dramatically.

## Begin - Joomla! core SEF Section.
#
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
# 
# If the requested path and file is not /index.php and the request
# has not already been internally rewritten to the index.php script
RewriteCond %{REQUEST_URI} !^/index\.php
# and the requested path and file doesn't directly match a physical file
RewriteCond %{REQUEST_FILENAME} !-f
# and the requested path and file doesn't directly match a physical folder
RewriteCond %{REQUEST_FILENAME} !-d
# internally rewrite the request to the index.php script
RewriteRule .* index.php [L]
#
## End - Joomla! core SEF Section.

My question(s):

Could this be DB related or could I rule that out because I'm getting good times with the SEF snippet removed?

Apache or perhaps a Joomla config issue somewhere?

Any insight would be very much appreciated, I've been working on this for a while now.

Side note: ( Edited )

If I turn off the SEF setting directly in the Joomla configuration, 404's just stop working.

Any random url shows the home page and a 200 response (in address bar the random url is intact)

I have since tried turning off the SEF setting and removed the snippet from the .htaccess - it now will give the correct apache 404 response for random urls. They also seem to respond well (only when bypassing the CDN cache but thats another topic )

Another Edit

The index.php file

<?php
/**
 * @package    Joomla.Site
 *
 * @copyright  Copyright (C) 2005 - 2020 Open Source Matters, Inc. All rights reserved.
 * @license    GNU General Public License version 2 or later; see LICENSE.txt
 */

/**
 * Define the application's minimum supported PHP version as a constant so it can be referenced within the application.
 */
define('JOOMLA_MINIMUM_PHP', '5.3.10');

if (version_compare(PHP_VERSION, JOOMLA_MINIMUM_PHP, '<'))
{
    die('Your host needs to use PHP ' . JOOMLA_MINIMUM_PHP . ' or higher to run this version of Joomla!');
}

// Saves the start time and memory usage.
//$startTime = microtime(1);
//$startMem  = memory_get_usage();

/**
 * Constant that is checked in included files to prevent direct access.
 * define() is used in the installation folder rather than "const" to not error for PHP 5.2 and lower
 */
define('_JEXEC', 1);

if (file_exists(__DIR__ . '/defines.php'))
{
    include_once __DIR__ . '/defines.php';
}

if (!defined('_JDEFINES'))
{
    define('JPATH_BASE', __DIR__);
    require_once JPATH_BASE . '/includes/defines.php';
}

require_once JPATH_BASE . '/includes/framework.php';

// Set profiler start time and memory usage and mark afterLoad in the profiler.
//JDEBUG ? JProfiler::getInstance('Application')->setStart($startTime, $startMem)->mark('afterLoad') : null;

// Instantiate the application.
$app = JFactory::getApplication('site');

// Execute the application.
$app->execute();