0

My site has a facility to download a potentially large body of data from my database as a CSV file. This is currently implemented using a view.csv.php file that serves this data by first setting the MIME encoding and content disposition as follows, then fetching the data using JDatabaseQuery::loadObjectList() and rendering to stdout:

    $document   = &JFactory::getDocument();
    $document->setMimeEncoding('text/csv');

    $filename = 'myfile.csv';

    JResponse::setHeader('Content-disposition', 'inline; filename="myfile.csv"', true);

With large enough data sets though, I get a PHP 'Allowed memory size of 134217728 bytes exhausted' error. Currently, I observe this happening during the rendering of the data to output. However, I am also concerned that if even larger datasets were requested, the error might occur when fetch the data from the database - I have not tested this to date.

One option I am considering to solve the immediate problem (errors during rendering) is to write the CSV content to a temporary file and return that somehow, rather than buffering the results directly to stdout, though I am unsure off-hand of how to return the file contents as the HTTP response once written, or if this will in fact solve the issue anyway.

So, my question is two-fold:

  1. Is using a temporary file the right approach to the memory-exhaustion-during-rendering issue, and if so, how do I subsequently return the content of this file as the HTTP response?
  2. Is there any mechanism to process SQL result sets one row at a time via JDatabase + JDatabaseQuery, rather than just fetching all of the data in bulk? I thought such a means existed, but on reviewing the Joomla API, it's not jumping out at me just now.
  • 1
    Just a hint: You could create a frontend controller and call it via com_ajax to write chunk of data to your file. When finished, you can return the path to a download view (window.location) which returns the file with the download header. This should trigger the download without reloading the current page. – Dennis Heiden Feb 19 '18 at 20:58
  • See my answer about com_ajax calls here: joomla.stackexchange.com/questions/146/… – Dennis Heiden Feb 19 '18 at 21:04
1

ad 1. The best option would be using streams. Unfortunately, JDocument does not support streams, so a temporary file might be the best available option. In your case it might be possible to send the file directly using streams, though.

ad 2. Use the JDatabaseDriver::getIterator() method to get a new iterator on the current query. It replaces the deprecated JDatabaseDriver::loadNextObject() method.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.