It is difficult to give you an exact answer without any more details or being able to debug your processing but looking at the code that is running I might be able to give you somewhere to look.
If you switch Error Reporting to Maximum in Global Config, third tab across, and Debug System to Yes, second tab, you might get more useful messages that narrow down where the problem is occurring. You may need to turn Gzip off also.

The error message you are receiving is coming from administrator/components/com_config/src/Model/ApplicationModel.php
in the last function, sendTestMail()
on the second last line. I suspect that the line $mailSent = $mailer->send();
is returning something other than true
public function sendTestMail()
{
// Set the new values to test with the current settings
$app = Factory::getApplication();
$user = Factory::getUser();
$input = $app->input->json;
$smtppass = $input->get('smtppass', null, 'RAW');
$app->set('smtpauth', $input->get('smtpauth'));
$app->set('smtpuser', $input->get('smtpuser', '', 'STRING'));
$app->set('smtphost', $input->get('smtphost'));
$app->set('smtpsecure', $input->get('smtpsecure'));
$app->set('smtpport', $input->get('smtpport'));
$app->set('mailfrom', $input->get('mailfrom', '', 'STRING'));
$app->set('fromname', $input->get('fromname', '', 'STRING'));
$app->set('mailer', $input->get('mailer'));
$app->set('mailonline', $input->get('mailonline'));
// Use smtppass only if it was submitted
if ($smtppass !== null)
{
$app->set('smtppass', $smtppass);
}
$mail = Factory::getMailer();
// Prepare email and try to send it
$mailer = new MailTemplate('com_config.test_mail', $user->getParam('language', $app->get('language')), $mail);
$mailer->addTemplateData(
array(
'sitename' => $app->get('sitename'),
'method' => Text::_('COM_CONFIG_SENDMAIL_METHOD_' . strtoupper($mail->Mailer))
)
);
$mailer->addRecipient($app->get('mailfrom'), $app->get('fromname'));
try
{
$mailSent = $mailer->send();
}
catch (MailDisabledException | phpMailerException $e)
{
.... This bit of code isn't running so I don't think it is
causing an exception
}
if ($mailSent === true)
{
.... This code only runs if your mail was successful.
}
$app->enqueueMessage(Text::_('COM_CONFIG_SENDMAIL_ERROR'), 'error');
return false;
}