1

I have the following PHP code working to bulk add array items from JSON to com_content. The code is executed by directly accessing the file in the web browser. For example, http://examples.com/json-articles.php.

<?php
if (!defined( '_JEXEC' )) {
  define( '_JEXEC', 1 );
  define( 'JPATH_BASE', realpath(dirname(__FILE__)));
  require_once ( JPATH_BASE .'/includes/defines.php' );
  require_once ( JPATH_BASE .'/includes/framework.php' );
  defined( 'DS' ) or define( 'DS', DIRECTORY_SEPARATOR );
}
$app = JFactory::getApplication( 'site' );

$curl = curl_init();
curl_setopt_array($curl, array(
  CURLOPT_URL => 'https://api.dtn.com/publishing/news/articles?categoryId=16%2C17%2C18&limit=3&maxAge=30&apikey=placeholder',
  CURLOPT_RETURNTRANSFER => true,
  CURLOPT_ENCODING => '',
  CURLOPT_MAXREDIRS => 10,
  CURLOPT_TIMEOUT => 0,
  CURLOPT_FOLLOWLOCATION => true,
  CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
  CURLOPT_CUSTOMREQUEST => 'GET',
));
$json = curl_exec($curl);
curl_close($curl);

$data = json_decode($json, true);

foreach ($data as $value) {
  $articleData = array(
    'id' => 0,
    'catid' => 8,
    'title' => ucwords(strtolower($value['title'])),
    'alias' => '',
    'introtext' => strip_tags($value['storySummary']),
    'fulltext' => strip_tags($value['content']),
    'state' => 1,
    'access' => 2,
    'created_by' => 332,
    'language' => '*'
  );
  $articleId = createArticle($articleData);

  if(!$articleId) {
    echo JText::_('Article(s) could not be created.');
  }
}

function createArticle($data) {
  $data['rules'] = array(
    'core.edit.delete' => array(),
    'core.edit.edit' => array(),
    'core.edit.state' => array(),
  );

  $basePath = JPATH_ADMINISTRATOR . '/components/com_content';
  require_once $basePath.'/models/article.php';
  $config = array();
  $articleModel = new ContentModelArticle($config);

  if(!$articleModel->save($data)) {
    $errMsg = $articleModel->getError();
    return false;
  } else {
    $id = $articleModel->getItem()->id;
    return $id;
  }
}

Now I am needing to convert it to a CLI app that can be executed at a set time everyday using a CRON job. I have reviewed existing CLI scripts in the CLI directory and have coded the following so far. I used this Joomla StackExchange post and started with the garbagecron.php CLI script as references.

<?php
/**
 * @package    Joomla.Cli
 *
 * @copyright  (C) 2012 Open Source Matters, Inc. <https://www.joomla.org>
 * @license    GNU General Public License version 2 or later; see LICENSE.txt
 */

/**
 * A command line cron job to bulk import JSON data into Joomla articles
 */

// Initialize Joomla framework
const _JEXEC = 1;

// Load system defines
if (file_exists( dirname(__DIR__) . '/defines.php' ) ) {
  require_once dirname(__DIR__) . '/defines.php';
}

if (!defined( '_JDEFINES' )) {
  define( 'JPATH_BASE', dirname(__DIR__) );
  require_once JPATH_BASE . '/includes/defines.php';
}

if (!defined( '_JEXEC' )) {
  define( '_JEXEC', 1 );
  define( 'JPATH_BASE', realpath(dirname(__FILE__)));
  require_once ( JPATH_BASE .'/includes/defines.php' );
  require_once ( JPATH_BASE .'/includes/framework.php' );
}

// Get the framework.
require_once JPATH_LIBRARIES . '/import.legacy.php';

// Bootstrap the CMS libraries.
require_once JPATH_LIBRARIES . '/cms.php';

/**
 * CLI script to bulk import JSON data into Joomla articles.
 *
 * @since  2.5
 */
class BulkArticles extends JApplicationCli {  
  public function doExecute() {
    $this->out(JText::_('Start'));

    // Fetch the JSON data
    $jsonData = $this->fetchData();

    // Run the insertArticle bulk data import to articles function
    $this->insertArticle($jsonData);
    
    $this->out(JText::_('Complete'));
  }

  private function fetchData() {
    $curl = curl_init();
    curl_setopt_array($curl, array(
      CURLOPT_URL => 'https://api.dtn.com/publishing/news/articles?categoryId=16%2C17%2C18&limit=10&maxAge=30&apikey=placeholder',
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_ENCODING => '',
      CURLOPT_MAXREDIRS => 10,
      CURLOPT_TIMEOUT => 0,
      CURLOPT_FOLLOWLOCATION => true,
      CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
      CURLOPT_CUSTOMREQUEST => 'GET',
    ));
    $json = curl_exec($curl);
    curl_close($curl);

    $data = json_decode($json, true);
    return $data;
  }

  private function insertArticle($data) {
    foreach ($data as $value) {
      // Outputs expected results from JSON data
      // echo($value['title']);
      // echo("<br/>");
      // echo($value['storySummary']);
      // echo("<br/>");
      // echo($value['content']);
      // echo("<br/>");
      $articleData = array(
        'id' => 0,
        'catid' => 8,
        'title' => ucwords(strtolower($value['title'])),
        'introtext' => strip_tags($value['storySummary']),
        'fulltext' => strip_tags($value['content']),
        'state' => 1,
        'access' => 2,
        'created_by' => 332,
        'language' => '*'
      );

      $data['rules'] = array(
        'core.edit.delete' => array(),
        'core.edit.edit' => array(),
        'core.edit.state' => array(),
      );
  
      $basePath = JPATH_ADMINISTRATOR . '/components/com_content';
      require_once $basePath.'/models/article.php';
      $config = array();
      $articleModel = new ContentModelArticle($config);
  
      if(!$articleModel->save($data)) {
        $errMsg = $articleModel->getError();
        return false;
      } else {
        $id = $articleModel->getItem()->id;
        return $id;
      }
  
      echo 'Article created';
    }
  }
}
JApplicationCli::getInstance('BulkArticles')->execute();

Update - Sample JSON data

[
  {
    "id": "fbb0427e-7529-4a74-b390-87bc21330068__2556796",
    "links": {
      "article": {
      "href": "/publishing/news/articles/fbb0427e-7529-4a74-b390-87bc21330068__2556796"
      }
    },
    "title": "DTN Closing Livestock Comments",
    "subTitle": "Lower Trend Blankets Livestock Futures Ahead of Close",
    "storySummary": "<span class=\"news_summary_content\">As the markets head into the last trading day of the week, they're hopeful that more support will push the contracts higher ahead of the weekend.</span>",
    "publishDate": "2022-01-27T22:21:00Z",
    "product": null,
    "author": {
      "name": "ShayLe Stewart"
    },
    "thumbNailImage": {
      "name": "2020closinglivestock.jpg",
      "imageUrl": "https://www.dtnpf.com/mydtn-public-core-portlet/servlet/GetStoredImage?category=CMS&symbolicName=2020closinglivestock.jpg",
      "caption": "(DTN file photo)"
    },
    "cmsCategoryId": null,
    "dtnpfUrl": "https://www.dtnpf.com/agriculture/web/ag/news/article/2022/01/27/lower-trend-blankets-livestock-ahead",
    "language": "en",
    "topic": "DTN/Ag/Livestock",
    "content": "<span class=\"news_summary_content\">As the markets head into the last trading day of the week, they're hopeful that more support will push the contracts higher ahead of the weekend.</span>",
    "status": null,
    "audioLinks": null,
    "mediaLinks": null
  },
  {
    "id": "07027E30",
    "links": {
      "article": {
        "href": "/publishing/news/articles/07027E30"
      }
    },
    "title": "Canadian Agriculture PM News",
    "subTitle": "Canadian Agriculture PM News",
    "storySummary": "<p class=\"news_content\">Canadian Financial Close: Loonie hits three-week low\nBy MarketsFarm</p>",
    "publishDate": "2022-01-27T21:35:42Z",
    "product": null,
    "author": null,
    "thumbNailImage": null,
    "cmsCategoryId": null,
    "dtnpfUrl": "",
    "language": "en",
    "topic": "DTN/Ag",
    "content": "\n<div style=\"background:#FFFFFF\">\n      Canadian Financial Close: Loonie hits three-week low\n      By MarketsFarm\n      WINNIPEG, Jan. 27 (MarketsFarm) - The Canadian dollar fell \nto its lowest level in three weeks the day after the Bank of \nCanada (BoC) announced it was holding its key interest rate at \n0.25 per cent.\n     The loonie was at US$0.7867 or US$1=C$1.2712 on Thursday, \ndown from Wednesday's close of US$0.7933 or US$1=C$1.2606. Both \nthe BoC and the United States Federal Reserve announced on \nWednesday they were keeping their interest rates steady for the \ntime being, but both signaled increases to counter inflation in \nthe coming months.\n     A strengthening greenback allowed the U.S. Dollar Index to \njump 1.30 of a point to 97.25, its highest level since June \n2020.\n     Benchmark crude oil prices were down on Thursday. Brent \ncrude oil slipped US$0.18 per barrel to US$89.78. West Texas \nIntermediate (WTI) crude oil dropped US$0.30 to US$87.05/barrel. \nWestern Canadian Select (WCS) crude oil increased US$0.64 at \nUS$74.27/barrel. \n     The TSX/S&amp;P Composite Index declined 51.78 points to \n20,544.11.\n     Gold nosedived US$35.50 per ounce to US$1,796.50, \ncontinuing its drop after the Fed's announcement.\n     Canada's agricultural sector fared as follows:\n\nFarmer's Edge Inc.               dn  $ 0.01     at $  2.76\nLinamar Corp.                    up  $ 1.15     at $ 69.07\nMaple Leaf Foods                 up  $ 0.09     at $ 29.55\nNutrien Ltd.                     dn  $ 0.44     at $ 88.31\nRitchie Bros Auctioneers Inc.    dn  $ 1.52     at $ 73.74\n     \n     (All figures are in Canadian dollars.)\n     END\n\n\n</div>\n",
    "status": null,
    "audioLinks": null,
    "mediaLinks": null
  }
]

Any help or tips is greatly appreciated.

13
  • I don't see a question here. Is your new code not working?
    – Sharky
    Jan 27 at 8:38
  • @Sharky The fetchData function is working, but I am having issues with the createArticle function. I am getting the following warnings in Terminal when executing the CLI script: PHP Warning: Undefined variable $articleData, PHP Warning: Undefined variable $value, PHP Warning: Undefined variable $data, and PHP Warning: foreach() argument must be of type array|object, null given. I updated my question with the current code. Jan 27 at 17:35
  • What is the benefit of declaring $articleExists as false, then immediately checking if the value is falsey in the createArticle() method? Can you offer a sample payload that might be returned from https://api.dtn.com/publishing/news/articles? It looks like you are expecting upto 10 articles. This suggests after json_decoding, there will be an array of arrays(rows). This means that JFilterOutput::stringURLSafe($value['title']) outside of a loop will not correctly access the title element because $value needs to refer to a "row" and title is a key in a row.
    – mickmackusa
    Jan 27 at 23:08
  • 1
    @mickmackusa I have updated my question with some sample JSON data. The query was limited to ten, but I just included three. I was using a similar CLI script that created categories and that is where the $articleExists comes from. Just want to avoid duplicate articles or risk of failure of imports if duplicates exist. Thanks for your help. Jan 28 at 2:02
  • 1
    @Mike thanks for the feedback. Can you try this adjusted script? 3v4l.org/0kYDU
    – mickmackusa
    Jan 29 at 2:20

3 Answers 3

2

I found when running CLI scripts that other things that you don't really need to execute, i.e. plugins, will cause failures because they don't check themselves to disable if they are running under a CLI environment and so they call things that aren't available to you and fail your script. To 'ignore' these errors I add several lines to bottom of my class to deal with these calls to things that don't exist. For example one of my CLI's has these entries being handled.

public function isClient() { return false; }
public function getTemplate() { return false; }
public function setHeader() { return false; }
public function enqueueMessage() { return false; }
public function allowCache() { return false; }

I think I picked up that tip from reading about CLI's on Akeeba's website.

I also use a __construct method in my class that contains the line

  Factory::$application = $this;

which I found was needed to run my CLI's but then mine are using methods in the main component and not like yours where all the code is in the CLI's methods. If things don't work for you, add it and see if it makes a difference.

I also put my path to the Model in the __construct so that it only gets processed once per run of the CLI rather than in your code it is added and or checked every time an article is created.

public function __construct()
{
    parent::__construct();
   Factory::$application = $this;

    BaseDatabaseModel::addIncludePath(JPATH_ADMINISTRATOR . '/components/com_inventory/models', 'Inventory');
    $this->model            = BaseDatabaseModel::getInstance('Inventory', 'InventoryModel');
}
2
  • These are just things I found useful for my work, hopefully others can add their own tips.
    – Irata
    Jan 27 at 1:03
  • Thanks for your tips and code suggestions. I have implemented your public functions, but it looks my issues are PHP related and not caused by Joomla execution processes. Jan 27 at 17:42
1

Here is the final working code courtesy of @mickmackusa. It took the addition of $app = JFactory::getApplication( 'site' ); to fix the Failed to start application error in the command line.

/**
 * A command line cron job to bulk import JSON data into Joomla articles
 */
use Joomla\CMS\Factory;

define('_JEXEC', 1);

// Load system defines
if (file_exists(dirname(__DIR__) . '/defines.php')) {
  require_once dirname(__DIR__) . '/defines.php';
}

if (!defined('_JDEFINES')) {
  define('JPATH_BASE', dirname(__DIR__));
  require_once JPATH_BASE . '/includes/defines.php';
}

require_once JPATH_LIBRARIES . '/import.legacy.php';
require_once JPATH_LIBRARIES . '/cms.php';

$app = JFactory::getApplication( 'site' );

class BulkArticles extends JApplicationCli {
  public function doExecute(): void {
    $this->out(JText::_('Start'));

    // Fetch the JSON data and Run the insertArticle bulk data import to articles function
    $curlOpts = [
      CURLOPT_URL => 'https://api.dtn.com/publishing/news/articles?categoryId=16%2C17%2C18&limit=2&maxAge=30&apikey=placeholder',
      CURLOPT_RETURNTRANSFER => true,
      CURLOPT_ENCODING => '',
      CURLOPT_MAXREDIRS => 10,
      CURLOPT_TIMEOUT => 0,
      CURLOPT_FOLLOWLOCATION => true,
      CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
      CURLOPT_CUSTOMREQUEST => 'GET',
    ];
    try {
      $fetchedArticles = $this->curlToObject($curlOpts);
      if ($fetchedArticles) {
        $this->out(JText::_('Fetched Articles: ' . count($fetchedArticles)));
      }
      $insertedIds = $this->insertArticles($fetchedArticles);
      if ($insertedIds) {
        $this->out(JText::_('Inserted IDs: ' . implode(', ', $insertedIds)));
      }
    } catch (exception $e) {
      $this->out(JText::_($e->getMessage()));
    }
    $this->out(JText::_('Complete'));
  }

  /**
   * @throws Exception
   */
  private function curlToObject(array $curlOpts): array {
    $ch = curl_init();
    curl_setopt_array($ch, $curlOpts);
    $result = curl_exec($ch);
    $error = curl_error($ch); // Get the last error
    curl_close($ch);
    if ($error) {
      throw new Exception($error);
    }
    return json_decode($result);
  }

  /**
   * @throws Exception
   */
  private function insertArticles(array $articles): array {
    require_once JPATH_ADMINISTRATOR . '/components/com_content/models/article.php';
    $articleModel = new ContentModelArticle([]);

    foreach ($articles as $article) {
      $articleData = [
        'id' => 0,
        'catid' => 8,
        'title' => ucwords(strtolower($article->title)),
        'alias' => JFilterOutput::stringURLSafe($article->title),
        'introtext' => strip_tags($article->storySummary),
        'fulltext' => strip_tags($article->content),
        'state' => 1,
        'access' => 2,
        'created_by' => 332,
        'language' => $article->language,
        'rules' => [
          'core.edit.delete' => [],
          'core.edit.edit' => [],
          'core.edit.state' => []
        ],
      ];

      if (!$articleModel->save($articleData)) {
        throw new Exception($articleModel->getError());
      } else {
        $newIds[] = $articleModel->getItem()->id;
      }
    }
    return $newIds ?? [];
  }
}
JApplicationCli::getInstance('BulkArticles')->execute();
0

In relation to your code the issue is you are not returning the data from the fetchData() so that it can be used by the createArticle(). This value of $articleData is empty at this point.

// Run the createArticle bulk data import to articles function
$this->createArticle($articleData);

Update your code with

// Fetch the JSON data
$articleData = $this->fetchData();

and in the bottom of fetchData() update the code to return the data being fetched.

    curl_close($curl);

    $data = json_decode($json, true);
    return $data
  }

In your working piece of PHP the code is not using functions/methods so the $data is available and passed from the fetching part to the looping part of your code.

curl_close($curl);

$data = json_decode($json, true);

foreach ($data as $value) { 
1
  • Thanks for the code changes. I have the createArticle method executing with some success, but am now receiving this warning: PHP Warning: Undefined array key "catid" in /administrator/components/com_content/models/article.php on line 600. The catid is hardcoded into the CLI script as a numerical 8, which is defined in the Joomla category manager. Jan 28 at 3:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.