Execute Joomla CLI script to bulk import JSON data to articles

Question

I have the following PHP code working to bulk add array items from JSON to com_content. The code is executed by directly accessing the file in the web browser. For example, http://examples.com/json-articles.php.

<?php
if (!defined( '_JEXEC' )) {
  define( '_JEXEC', 1 );
  define( 'JPATH_BASE', realpath(dirname(__FILE__)));
  require_once ( JPATH_BASE .'/includes/defines.php' );
  require_once ( JPATH_BASE .'/includes/framework.php' );
  defined( 'DS' ) or define( 'DS', DIRECTORY_SEPARATOR );
}
$app = JFactory::getApplication( 'site' );

$curl = curl_init();
curl_setopt_array($curl, array(
  CURLOPT_URL => 'https://api.dtn.com/publishing/news/articles?categoryId=16%2C17%2C18&limit=3&maxAge=30&apikey=placeholder',
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => '',
  CURLOPT_MAXREDIRS => 10,
  CURLOPT_TIMEOUT => 0,
  CURLOPT_FOLLOWLOCATION => true,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => 'GET',
));
$json = curl_exec($curl);
curl_close($curl);

$data = json_decode($json, true);

foreach ($data as $value) {
  $articleData = array(
    'id' => 0,
    'catid' => 8,
    'title' => ucwords(strtolower($value['title'])),
    'alias' => '',
    'introtext' => strip_tags($value['storySummary']),
    'fulltext' => strip_tags($value['content']),
    'state' => 1,
    'access' => 2,
    'created_by' => 332,
    'language' => '*'
  );
  $articleId = createArticle($articleData);

  if(!$articleId) {
    echo JText::_('Article(s) could not be created.');
  }
}

function createArticle($data) {
  $data['rules'] = array(
    'core.edit.delete' => array(),
    'core.edit.edit' => array(),
    'core.edit.state' => array(),
  );

  $basePath = JPATH_ADMINISTRATOR . '/components/com_content';
  require_once $basePath.'/models/article.php';
  $config = array();
  $articleModel = new ContentModelArticle($config);

  if(!$articleModel->save($data)) {
    $errMsg = $articleModel->getError();
    return false;
  } else {
    $id = $articleModel->getItem()->id;
    return $id;
  }
}

Now I am needing to convert it to a CLI app that can be executed at a set time everyday using a CRON job. I have reviewed existing CLI scripts in the CLI directory and have coded the following so far. I used this Joomla StackExchange post and started with the garbagecron.php CLI script as references.

<?php
/**
 * @package    Joomla.Cli
 *
 * @copyright  (C) 2012 Open Source Matters, Inc. <https://www.joomla.org>
 * @license    GNU General Public License version 2 or later; see LICENSE.txt
 */

/**
 * A command line cron job to bulk import JSON data into Joomla articles
 */

// Initialize Joomla framework
const _JEXEC = 1;

// Load system defines
if (file_exists( dirname(__DIR__) . '/defines.php' ) ) {
  require_once dirname(__DIR__) . '/defines.php';
}

if (!defined( '_JDEFINES' )) {
  define( 'JPATH_BASE', dirname(__DIR__) );
  require_once JPATH_BASE . '/includes/defines.php';
}

if (!defined( '_JEXEC' )) {
  define( '_JEXEC', 1 );
  define( 'JPATH_BASE', realpath(dirname(__FILE__)));
  require_once ( JPATH_BASE .'/includes/defines.php' );
  require_once ( JPATH_BASE .'/includes/framework.php' );
}

// Get the framework.
require_once JPATH_LIBRARIES . '/import.legacy.php';

// Bootstrap the CMS libraries.
require_once JPATH_LIBRARIES . '/cms.php';

/**
 * CLI script to bulk import JSON data into Joomla articles.
 *
 * @since  2.5
 */
class BulkArticles extends JApplicationCli {  
  public function doExecute() {
    $this->out(JText::_('Start'));

    // Fetch the JSON data
    $jsonData = $this->fetchData();

    // Run the insertArticle bulk data import to articles function
    $this->insertArticle($jsonData);
    
    $this->out(JText::_('Complete'));
  }

  private function fetchData() {
    $curl = curl_init();
    curl_setopt_array($curl, array(
      CURLOPT_URL => 'https://api.dtn.com/publishing/news/articles?categoryId=16%2C17%2C18&limit=10&maxAge=30&apikey=placeholder',
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_ENCODING => '',
      CURLOPT_MAXREDIRS => 10,
      CURLOPT_TIMEOUT => 0,
      CURLOPT_FOLLOWLOCATION => true,
      CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
      CURLOPT_CUSTOMREQUEST => 'GET',
    ));
    $json = curl_exec($curl);
    curl_close($curl);

    $data = json_decode($json, true);
    return $data;
  }

  private function insertArticle($data) {
    foreach ($data as $value) {
      // Outputs expected results from JSON data
      // echo($value['title']);
      // echo("<br/>");
      // echo($value['storySummary']);
      // echo("<br/>");
      // echo($value['content']);
      // echo("<br/>");
      $articleData = array(
        'id' => 0,
        'catid' => 8,
        'title' => ucwords(strtolower($value['title'])),
        'introtext' => strip_tags($value['storySummary']),
        'fulltext' => strip_tags($value['content']),
        'state' => 1,
        'access' => 2,
        'created_by' => 332,
        'language' => '*'
      );

      $data['rules'] = array(
        'core.edit.delete' => array(),
        'core.edit.edit' => array(),
        'core.edit.state' => array(),
      );
  
      $basePath = JPATH_ADMINISTRATOR . '/components/com_content';
      require_once $basePath.'/models/article.php';
      $config = array();
      $articleModel = new ContentModelArticle($config);
  
      if(!$articleModel->save($data)) {
        $errMsg = $articleModel->getError();
        return false;
      } else {
        $id = $articleModel->getItem()->id;
        return $id;
      }
  
      echo 'Article created';
    }
  }
}
JApplicationCli::getInstance('BulkArticles')->execute();

Update - Sample JSON data

[
  {
    "id": "fbb0427e-7529-4a74-b390-87bc21330068__2556796",
    "links": {
      "article": {
      "href": "/publishing/news/articles/fbb0427e-7529-4a74-b390-87bc21330068__2556796"
      }
    },
    "title": "DTN Closing Livestock Comments",
    "subTitle": "Lower Trend Blankets Livestock Futures Ahead of Close",
    "storySummary": "<span class=\"news_summary_content\">As the markets head into the last trading day of the week, they're hopeful that more support will push the contracts higher ahead of the weekend.</span>",
    "publishDate": "2022-01-27T22:21:00Z",
    "product": null,
    "author": {
      "name": "ShayLe Stewart"
    },
    "thumbNailImage": {
      "name": "2020closinglivestock.jpg",
      "imageUrl": "https://www.dtnpf.com/mydtn-public-core-portlet/servlet/GetStoredImage?category=CMS&symbolicName=2020closinglivestock.jpg",
      "caption": "(DTN file photo)"
    },
    "cmsCategoryId": null,
    "dtnpfUrl": "https://www.dtnpf.com/agriculture/web/ag/news/article/2022/01/27/lower-trend-blankets-livestock-ahead",
    "language": "en",
    "topic": "DTN/Ag/Livestock",
    "content": "<span class=\"news_summary_content\">As the markets head into the last trading day of the week, they're hopeful that more support will push the contracts higher ahead of the weekend.</span>",
    "status": null,
    "audioLinks": null,
    "mediaLinks": null
  },
  {
    "id": "07027E30",
    "links": {
      "article": {
        "href": "/publishing/news/articles/07027E30"
      }
    },
    "title": "Canadian Agriculture PM News",
    "subTitle": "Canadian Agriculture PM News",
    "storySummary": "<p class=\"news_content\">Canadian Financial Close: Loonie hits three-week low\nBy MarketsFarm</p>",
    "publishDate": "2022-01-27T21:35:42Z",
    "product": null,
    "author": null,
    "thumbNailImage": null,
    "cmsCategoryId": null,
    "dtnpfUrl": "",
    "language": "en",
    "topic": "DTN/Ag",
    "content": "\n<div style=\"background:#FFFFFF\">\n      Canadian Financial Close: Loonie hits three-week low\n      By MarketsFarm\n      WINNIPEG, Jan. 27 (MarketsFarm) - The Canadian dollar fell \nto its lowest level in three weeks the day after the Bank of \nCanada (BoC) announced it was holding its key interest rate at \n0.25 per cent.\n     The loonie was at US$0.7867 or US$1=C$1.2712 on Thursday, \ndown from Wednesday's close of US$0.7933 or US$1=C$1.2606. Both \nthe BoC and the United States Federal Reserve announced on \nWednesday they were keeping their interest rates steady for the \ntime being, but both signaled increases to counter inflation in \nthe coming months.\n     A strengthening greenback allowed the U.S. Dollar Index to \njump 1.30 of a point to 97.25, its highest level since June \n2020.\n     Benchmark crude oil prices were down on Thursday. Brent \ncrude oil slipped US$0.18 per barrel to US$89.78. West Texas \nIntermediate (WTI) crude oil dropped US$0.30 to US$87.05/barrel. \nWestern Canadian Select (WCS) crude oil increased US$0.64 at \nUS$74.27/barrel. \n     The TSX/S&amp;P Composite Index declined 51.78 points to \n20,544.11.\n     Gold nosedived US$35.50 per ounce to US$1,796.50, \ncontinuing its drop after the Fed's announcement.\n     Canada's agricultural sector fared as follows:\n\nFarmer's Edge Inc.               dn  $ 0.01     at $  2.76\nLinamar Corp.                    up  $ 1.15     at $ 69.07\nMaple Leaf Foods                 up  $ 0.09     at $ 29.55\nNutrien Ltd.                     dn  $ 0.44     at $ 88.31\nRitchie Bros Auctioneers Inc.    dn  $ 1.52     at $ 73.74\n     \n     (All figures are in Canadian dollars.)\n     END\n\n\n</div>\n",
    "status": null,
    "audioLinks": null,
    "mediaLinks": null
  }
]

Any help or tips is greatly appreciated.

Answer 1

I found when running CLI scripts that other things that you don't really need to execute, i.e. plugins, will cause failures because they don't check themselves to disable if they are running under a CLI environment and so they call things that aren't available to you and fail your script. To 'ignore' these errors I add several lines to bottom of my class to deal with these calls to things that don't exist. For example one of my CLI's has these entries being handled.

public function isClient() { return false; }
public function getTemplate() { return false; }
public function setHeader() { return false; }
public function enqueueMessage() { return false; }
public function allowCache() { return false; }

I think I picked up that tip from reading about CLI's on Akeeba's website.

I also use a __construct method in my class that contains the line

  Factory::$application = $this;

which I found was needed to run my CLI's but then mine are using methods in the main component and not like yours where all the code is in the CLI's methods. If things don't work for you, add it and see if it makes a difference.

I also put my path to the Model in the __construct so that it only gets processed once per run of the CLI rather than in your code it is added and or checked every time an article is created.

public function __construct()
{
    parent::__construct();
   Factory::$application = $this;

    BaseDatabaseModel::addIncludePath(JPATH_ADMINISTRATOR . '/components/com_inventory/models', 'Inventory');
    $this->model            = BaseDatabaseModel::getInstance('Inventory', 'InventoryModel');
}

Answer 2

Here is the final working code courtesy of @mickmackusa. It took the addition of $app = JFactory::getApplication( 'site' ); to fix the Failed to start application error in the command line.

/**
 * A command line cron job to bulk import JSON data into Joomla articles
 */
use Joomla\CMS\Factory;

define('_JEXEC', 1);

// Load system defines
if (file_exists(dirname(__DIR__) . '/defines.php')) {
  require_once dirname(__DIR__) . '/defines.php';
}

if (!defined('_JDEFINES')) {
  define('JPATH_BASE', dirname(__DIR__));
  require_once JPATH_BASE . '/includes/defines.php';
}

require_once JPATH_LIBRARIES . '/import.legacy.php';
require_once JPATH_LIBRARIES . '/cms.php';

$app = JFactory::getApplication( 'site' );

class BulkArticles extends JApplicationCli {
  public function doExecute(): void {
    $this->out(JText::_('Start'));

    // Fetch the JSON data and Run the insertArticle bulk data import to articles function
    $curlOpts = [
      CURLOPT_URL => 'https://api.dtn.com/publishing/news/articles?categoryId=16%2C17%2C18&limit=2&maxAge=30&apikey=placeholder',
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_ENCODING => '',
      CURLOPT_MAXREDIRS => 10,
      CURLOPT_TIMEOUT => 0,
      CURLOPT_FOLLOWLOCATION => true,
      CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
      CURLOPT_CUSTOMREQUEST => 'GET',
    ];
    try {
      $fetchedArticles = $this->curlToObject($curlOpts);
      if ($fetchedArticles) {
        $this->out(JText::_('Fetched Articles: ' . count($fetchedArticles)));
      }
      $insertedIds = $this->insertArticles($fetchedArticles);
      if ($insertedIds) {
        $this->out(JText::_('Inserted IDs: ' . implode(', ', $insertedIds)));
      }
    } catch (exception $e) {
      $this->out(JText::_($e->getMessage()));
    }
    $this->out(JText::_('Complete'));
  }

  /**
   * @throws Exception
   */
  private function curlToObject(array $curlOpts): array {
    $ch = curl_init();
    curl_setopt_array($ch, $curlOpts);
    $result = curl_exec($ch);
    $error = curl_error($ch); // Get the last error
    curl_close($ch);
    if ($error) {
      throw new Exception($error);
    }
    return json_decode($result);
  }

  /**
   * @throws Exception
   */
  private function insertArticles(array $articles): array {
    require_once JPATH_ADMINISTRATOR . '/components/com_content/models/article.php';
    $articleModel = new ContentModelArticle([]);

    foreach ($articles as $article) {
      $articleData = [
        'id' => 0,
        'catid' => 8,
        'title' => ucwords(strtolower($article->title)),
        'alias' => JFilterOutput::stringURLSafe($article->title),
        'introtext' => strip_tags($article->storySummary),
        'fulltext' => strip_tags($article->content),
        'state' => 1,
        'access' => 2,
        'created_by' => 332,
        'language' => $article->language,
        'rules' => [
          'core.edit.delete' => [],
          'core.edit.edit' => [],
          'core.edit.state' => []
        ],
      ];

      if (!$articleModel->save($articleData)) {
        throw new Exception($articleModel->getError());
      } else {
        $newIds[] = $articleModel->getItem()->id;
      }
    }
    return $newIds ?? [];
  }
}
JApplicationCli::getInstance('BulkArticles')->execute();

Answer 3

In relation to your code the issue is you are not returning the data from the fetchData() so that it can be used by the createArticle(). This value of $articleData is empty at this point.

// Run the createArticle bulk data import to articles function
$this->createArticle($articleData);

Update your code with

// Fetch the JSON data
$articleData = $this->fetchData();

and in the bottom of fetchData() update the code to return the data being fetched.

    curl_close($curl);

    $data = json_decode($json, true);
    return $data
  }

In your working piece of PHP the code is not using functions/methods so the $data is available and passed from the fetching part to the looping part of your code.

curl_close($curl);

$data = json_decode($json, true);

foreach ($data as $value) {