I have a controller with an action that accepts requests over AJAX, does some data manipulation, and returns some data:

function publicDirectoryAjaxSearch(){
    // Check the form token
    JSession::checkToken('get') or die('Invalid Token');
    
    // Don't show PHP errors in raw output
    ini_set('display_errors', 'Off');
    
    // Sort out the session status
    $session = JFactory::getSession();
    $session->set('getOrganisationsStatus', 'Searching...');
    
    // Get orgs
    $response = $this->doPublicDirectorySearch();

    // Get the document object.
    $document = JFactory::getDocument();
    
    // Set the MIME type for JSON output.
    $document->setMimeEncoding('application/json');
    
    // Change the suggested filename.
    JResponse::setHeader('Content-Disposition','attachment;filename="result.json"');
    
    echo json_encode($response);
}

This method can take a while, so I'm using JSession to store a status variable getOrganisationStatus that can be periodically checked via AJAX to see how far the $this->doPublicDirectorySearch() method has gotten...

function doPublicDirectorySearch(){
    ...
    while($running)
         $session->set('getOrganisationsStatus', 'Fetched '.$handled."/".$totalResults." results.");
    ...
}

However, I'm having some issues. If I set the Joomla global config session handler to Database the getOrganisationsStatus session variable always seems to 'lag' 1 data request behind. For example (in prep for this example I cleared all existing sessions):

1. Make a data request via AJAX to publicDirectoryAjaxSearch
2. Set an interval to get the request status session variable over AJAX to publicDirectorySearchStatus
3. Data request still running. Session value changing as $this->doPublicDirectorySearch() works through data
4. Status call made from setInterval to publicDirectorySearchStatus. Returned session variable is empty (this step repeats over the interval)
5. publicDirectoryAjaxSearch data request completes. Method returns JSON object
6. Final status call made over AJAX from setInterval to publicDirectorySearchStatus. Returned data (the session variable) now shows the correct text as set in the data request method.

If I set the Joomla global config to use None as the session handler (i.e. PHP's built in file-based handler) then the script blocks (as PHP locks the session file when a script accesses it, so the AJAX call to find out the status has to wait til the method updating the session variable finishes - defeating the point somewhat).

Where am I going wrong? How can I..

1. Make a call to get data (takes some time)
2. Make a separate call to get a session variable that's being manipulated over time by the first call