How to handle duplicate items coming from an external JSON API and being imported as article content

Question

I have adapted working code from Joomla 3 and the CLI to Joomla 4/5 and the API. The core functionality using the API system is working, but I am struggling to find a replacement for checking for duplicate aliases.

When a duplicate article alias is encountered, Joomla/Admin Tools send the following message via email: Message: Save failed with the following error: Another Article in this category has the same alias. This message is issued numerous times depending on how many news items are being imported. The max importation is set to 10 for individual requests. Requests are made once a day in the early morning via a cron job.

The external JSON API provides is a news source that has unique titles for individual items, but no aliases beyond the article's href element. I have included a sample snippet of a news item below for reference.

{
  "id": "18d02c72-7206-4a6a-bec1-0f608524c422__4086381",
  "links": {
    "article": {
      "href": "/publishing/news/articles/18d02c72-7206-4a6a-bec1-0f608524c422__4086381"
    }
  },
  "title": "DTN Closing Livestock Comments",
  "subTitle": "Markets Stabilized Friday Following Bearish Weak",
  "storySummary": "<span class=\"news_summary_content\">Limited buyer support trickled into livestock trade Friday, attempting to bring some sense of stability to a tumbling cattle market. Despite moderate support midday, live cattle contracts closed mixed with continued concerns of longer-term support. It is unlikely that the weekend will bring much clarity to livestock markets.</span>",
  "publishDate": "2023-11-10T21:56:00Z",
  "product": null,
  "author": {
    "name": "Rick Kment"
  },
  "thumbNailImage": {
    "name": "2020closinglivestock.jpg",
    "imageUrl": "https://www.dtnpf.com/mydtn-public-core-portlet/servlet/GetStoredImage?category=CMS&symbolicName=2020closinglivestock.jpg",
    "caption": "(DTN file photo)"
  },
  "cmsCategoryId": null,
  "dtnpfUrl": "https://www.dtnpf.com/agriculture/web/ag/news/article/2023/11/10/markets-stabilized-friday-following",
  "language": "en",
  "topic": "DTN/Ag/Livestock",
  "content": "<span class=\"news_summary_content\">Limited buyer support trickled into livestock trade Friday, attempting to bring some sense of stability to a tumbling cattle market. Despite moderate support midday, live cattle contracts closed mixed with continued concerns of longer-term support. It is unlikely that the weekend will bring much clarity to livestock markets.</span>",
  "status": null,
  "audioLinks": null,
  "mediaLinks": null
}

Here is the code that was previously used in the CLI for handling duplicate aliases. As this code utilizes the Joomla database, I do not think it could be used for the API.

private function removeDuplicateAliases(array $articles): array {
  $db = Factory::getDbo();
  $query = $db->getQuery(true);
  $query
    ->select('a.alias')
    ->from($db->quoteName('#__content', 'a'))
    ->where($db->quoteName('a.catid') . " = " . $db->quote('13'));
  $db->setQuery($query);
  $aliasLookup = array_flip($db->loadColumn());

  foreach ($articles as $i => $article) {
    if (isset($aliasLookup[OutputFilter::stringURLSafe($article->title)])) {
      unset($articles[$i]);
    }
  }
  return $articles;
}

The function was run within the insertArticles function foreach ($this->removeDuplicateAliases($articles) as $article).

Here is the working API-based code for the retrieval and processing of the external JSON API:

declare(strict_types=1);
ini_set('error_reporting', E_ALL & ~E_DEPRECATED);

$dataSourceUrl = Redacted;

$processArticles = function ($articles) {
  foreach ($articles as $article) {
    $data = [
      'id' => 0,
      'catid' => 13,
      'title' => ucwords(strtolower($article->title)),
      'articletext' => $article->content,
      'introtext' => $article->storySummary,
      'fulltext' => $article->content,
      'note' => 'DTN supplied topics: ' . str_replace('/', ', ', $article->topic),
      'state' => 1,
      'access' => 1,
      'created_by' => 386,
      'created_by_alias' => 'DTN News',
      'language' => '*',
    ];
    $articleData = json_encode($data);

    $apiUrl = Redacted;
    $token = Redacted;

    // Request timeout
    $timeOut = 10;

    $headers = [
      'Accept: application/vnd.api+json',
      'Content-Type: application/json',
      'Content-length: ' . mb_strlen($articleData),
      sprintf('X-Joomla-Token: %s', trim($token)),
    ];

    $curl = curl_init();

    curl_setopt_array($curl, [
        CURLOPT_URL => sprintf('%s/%s', $apiUrl, 'content/articles'),
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_ENCODING => 'utf-8',
        CURLOPT_MAXREDIRS => 10,
        CURLOPT_TIMEOUT => $timeOut,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_2TLS,
        CURLOPT_CUSTOMREQUEST => 'POST',
        CURLOPT_POSTFIELDS => $articleData,
        CURLOPT_HTTPHEADER => $headers,
      ]
    );
    $response = curl_exec($curl);
    if ($response === false) {
      echo 'CURL error: ' . curl_error($curl);
    } else {
      curl_close($curl);
    }
  }
};

$generator = function (string $dataSourceResponse): Generator {
  if (empty($dataSourceResponse)) {
    yield new RuntimeException( 'DTN API response must not be empty', 422 );
  }

  if ($data === false) {
    yield new RuntimeException( 'Could not read response', 500 );
  }

  try {
  } finally {
    echo 'Done processing data' . PHP_EOL;
  }
};

$process = function (string $dataSourceUrl, string $dataString) {
  $curlOptions = [
    CURLOPT_URL => $dataSourceUrl,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_ENCODING => '',
    CURLOPT_MAXREDIRS => 10,
    CURLOPT_TIMEOUT => 0,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
    CURLOPT_CUSTOMREQUEST => 'GET',
    CURLOPT_POSTFIELDS => $dataString,
  ];
  $transport = curl_init();
  curl_setopt_array($transport, $curlOptions);
  $data = curl_exec($transport);

  if (empty($data)) {
    throw new RuntimeException( 'Empty output', 422 );
  }

  return json_decode($data);
};

$dataSourceDataString = '';

try {
  $dataSourceResponse = $process($dataSourceUrl, $dataSourceDataString);
  $displayData        = $processArticles($dataSourceResponse);
} finally {
}

The above still needs to be refined and may include some code that could be redundant and could be removed.

I have been using this Joomla 4 API example as reference for completing the migration from the J3 CLI to the J4/5 API system.