Queue API: Process execution by parts

June 18, 2021

The latest version of Drupal has sought to stop being just a content management system but to have more tools worthy of a framework. For these various improvements for building forms, queries to the database, better caching systems, improvements to libraries theming, etc. However, very little is mentioned about the queue API

What does it consist of?

The Queue API is a tool designed for the administration of certain instructions, which in other development environments or operating systems is called process queue. It is basically a deposit of expected instructions to be executed as if it were a list of personnel tasks for the framework of Drupal, where this is gathering the remaining tasks and will discard those already made, and does so in an orderly manner, in a simple and usable way for every programmer. It uses the same concepts of [queues as data structures] that can be observed in different programming languages.

What is it for?

Here's an example: I have a form in which a message is sent to all users who are of a certain role. The logic that builds mail at first glance does not seem so complicated.

Alternative 1: Send all a mail with the the form

It would issue a query to the database (see [API database Drupal]) for the users of a particular role, and each call a function to send an email with their respective information and your personal information.

It sounds pretty simple. The problem is that if they currently exist, for example about 200 users for a certain role on the site, the process of sending or even generating emails, can not be completed before the server abruptly stops the process for excess time or memory.

Alternative 2: Save information in cache

This alternative is what can be saved in the cache (either system or custom) and get the saved data later using a hook. The problem with this procedure is that even the risk that the process of sending information to be equally heavy cache for the server and PHP runs may slow down the process. Unfortunately for developers on Drupal 6 or lower, this was the only alternative available to emulate a process queue.

Alternative 3: Queue API

This alternative is based on the previous one, with the difference that with this API, there are functions available for the proper development of this problem. Queue API provides functions to save tasks in different queues that can create the programmer according to their needs, and Drupal is responsible for executing them later. While all tasks are properly organized for later execution, this way the programmer only detracts inform users form the previous example that mails are ready to be sent.

The way the Queue API works is based on cron. Each task being performed instead of proceeding with its execution, a function that invokes this process and a function is used to save the process to a new queue is used. Then these processes are executed and determined by the parameters of the queue time, these processes run on Drupal and the server without waiting for a user request, but this process queue for the cron system where the site is hosted.

How does it work?

Consider the example mentioned a moment ago about sending a form to send a mass mailing.

This function simulates the functionality that generates information for multiple users.


function mymodule_pending_users_info() {
  return array(
    array(
      'subject' => 'Asunto 1',
      'from' => 'mail1@example.com',
      'to' => 'mail2@example.com',
      'message' => '...'
    ),
    array(
      'subject' => 'Asunto 2',
      'from' => 'mail3@example.com',
      'to' => 'mail4@example.com',
      'message' => '...'
    ),
    array(
      'subject' => 'Asunto 3',
      'from' => 'mail5@example.com',
      'to' => 'mail6@example.com',
      'message' => '...'
    ),
    array(
      'subject' => 'Asunto 4',
      'from' => 'mail7@example.com',
      'to' => 'mail8@example.com',
      'message' => '...'
    ),
  );
}

This feature send mail according to the information received


function mymodule_send_mail_to_user($user_info) {
  drupal_mail(...);
} 

The code below shows how to store data in queues .


//Funcion para obtener la instancia de una cola. Como puede verse, se
//pueden generar cuantas se consideren necesarias usando esta función. Si la
//cola no existe, la funcion genera una nueva.
$queue = DrupalQueue::get('cola1');

//Por cada usuario obtenido por la funcion que obtiene los usuarios
//destinatarios, se agrega su información a la cola creada. Cabe destacar
//que estos items pueden ser de cualquier tipo que PHP soporte.
foreach(mymodule_pending_users_info() as $item) {
  $queue->createItem($item);
}
//Toma cada elemento de la cola y ejecuta una acción por cada uno. Posteriormente los remueve.
//claimItem sirve para buscar el ultimo elemento insertado en el queue.
//Éste código no se tomara en cuenta para el siguiente ejemplo.
while($item = $queue->claimItem()) {
  mymodule_send_mail_to_user($item->data);
  //Remueve el último item de la cola.
  $queue->deleteItem($item);
}

While this code works only serves to store data in queues, and so far not exploiting fully the advantages of using queues, since we're still processing data in a single call, and remains heavy for PHP carry out this petition.

For this Drupal has a hook that allows you to perform a task whose information is provided by the data from a queue. This is accomplished using cron. Keeping the above example (without the code that sends e), and we needed to fill our queue with the relevant data. Then hook_cron_queue_info you will be used to assign a task to each item whenever the claims cron periodically.


/**
 * Implements hook_cron_queue_info().
 */
function mymodule_cron_queue_info() {
  $queues = array();
  $queues['my_cron_queue'] = array(
     //Por cada item, se llamara a la funcion señalada abajo.
    'worker callback' => 'mymodule_send_mail_to_user',
    //Los segundos que tiene cron para continuar ejecutando cada item.
    'time' => 60,
  );
  return $queues;
}

Using this hook, the programmer need not do the respective operations to create, obtain or free up space in the queue, or use hook_cron for processing the data in the queue, since Drupal handles all that.

I hope this blog was useful. My most sincere greetings, young Drupalers!