Creating custom plugins for Drupal 7 to Drupal 8 Migrations

With Drupal 7 end-of-life on the horizon, you’ll see more and more people migrating their work to Drupal 8 and Drupal 9.

Let’s start with the basics, first:

What is a Source Plugin?

A source plugin is a way to provide source data to the migration that will convert it to a Drupal entity. Read more about Migrate source plugins here.

What is a Process Plugin?

A process plugin is a way to manipulate the value of a field during migration. There are limitless things that allow you to do this. Check out the plugins that are provided by the core Migrate module and official documentation on the migrate process overview and writing a process plugin. 

Migrating Scald Atoms to Drupal 8 Media Entities Concepts

Our team has done plenty of migrations including a migration from Drupal 7 file entities to Drupal 8 Media entities before, following the concepts from this blog post by PreviousNext. 

While the concepts described below are used to explain migrating Scald Atoms, the concepts can be used to migrate many other types of data.

What if I don’t want to migrate everything?

In our case, not every single file needed to be migrated. For example, there were still some old Flash files in the file system and since Flash is no longer supported, we want our migration to skip these files.

Thankfully, the Migrate Plus module provides a process plugin that will help us do this.

To skip these files from being migrated, our migrate yml file looked something like this:

 filemime:
   plugin: skip_on_value
   method: row
   value:
     - application/x-shockwave-flash

Read more about Skip On Value plugin and other process plugins provided by Migrate Plus

Getting entity fields and data from our Scald Atoms

In our new Drupal instance, we manually created each of the Media types we wanted to have. (ex: Images, Vimeo videos, Youtube Videos, Documents). Each of those media entity types was created and fields were added to them accordingly.

At this point, there is no existing migration that I know of that will help migrate Scald media (or Atoms as the module calls them) so we needed a custom migration source. As you’ll see from the source code below, to get everything we need, our query starts by getting base columns from the scald_atoms table. Then, depending on the media type, we query other fields from Drupal based on the atom_type configuration. One thing to note is that we used conditional statements for this but it may make sense for your project to break each type up into its own source plugin.

Once the source plugin file is created, in our migration yml file, we can tell our plugin to only query atoms that are of a specific type. Here’s an example of taking atoms of type file and creating a media entity of bundle type document.

source:
 plugin: d7_scald_atom
 atom_type: file
 constants:
   bundle: document

Here’s the full source plugin code for reference: 

<?php
namespace Drupal\your_module\Plugin\migrate\source;

use Drupal\Core\Database\Query\Condition;
use Drupal\migrate\Row;
use Drupal\migrate_drupal\Plugin\migrate\source\d7\FieldableEntity;

/**
 * Processes Scald Media items to Drupal 8 Media entities.
 *
 * @MigrateSource(
 *   id = "d7_scald_atom",
 *   source_provider = "scald",
 *   source_module = "scald",
 * )
 */
class ScaldAtom extends FieldableEntity {

  /**
   * {@inheritdoc}
   */
  public function query() {
    $query = $this->select('scald_atoms', 's')
      ->fields('s', array(
        'sid',
        'provider',
        'type',
        'base_id',
        'language',
        'publisher',
        'actions',
        'title',
        'data',
        'created',
        'changed',
        'publisher',
      ));
    
    if (isset($this->configuration['atom_type'])) {
      if ($this->configuration['atom_type'] === 'image') {
        $query->fields('st', array(
          'scald_thumbnail_alt'
        ))
        ->fields('cred', array(
          'field_credit_value'
        ))
        ->fields('url', array(
          'field_photo_credit_url_url'
        ))
        ->fields('cap', array(
          'field_caption_long_value'
        ))
        ->fields('cop', array(
          'field_copyright_value'
        ))
        ->fields('fm', array(
          'uri'
        ));

        $query->leftJoin('field_data_scald_thumbnail', 'st', 'st.entity_id = s.sid');
        $query->leftJoin('field_data_field_credit', 'cred', 'cred.entity_id = s.sid');
        $query->leftJoin('field_data_field_photo_credit_url', 'url', 'url.entity_id = s.sid');
        $query->leftJoin('field_data_field_caption_long', 'cap', 'cap.entity_id = s.sid');
        $query->leftJoin('field_data_field_copyright', 'cop', 'cop.entity_id = s.sid');
        $query->leftJoin('file_managed', 'fm', 'fm.fid = s.base_id');
      } else if ($this->configuration['atom_type'] === 'video') {
        $query->fields('cred', array(
          'field_credit_value'
        ))
        ->fields('cap', array(
          'field_caption_long_value'
        ))
        ->fields('cop', array(
          'field_copyright_value'
        ));

        $query->leftJoin('field_data_field_credit', 'cred', 'cred.entity_id = s.sid');
        $query->leftJoin('field_data_field_caption_long', 'cap', 'cap.entity_id = s.sid');
        $query->leftJoin('field_data_field_copyright', 'cop', 'cop.entity_id = s.sid');
        $query->leftJoin('file_managed', 'fm', 'fm.fid = s.base_id');
      } else if ($this->configuration['atom_type'] === 'file') {
        // Exclude JS files for now.
        $query->fields('fm', array(
          'filemime',
          'uri',
          'uid'
        ));
        $query->leftJoin('file_managed', 'fm', 'fm.fid = s.base_id');
        $query->condition('fm.filemime', ['application/octet-stream', 'application/x-shockwave-flash', 'application/x-javascript', 'text/css', 'application/zip', 'text/html', 'application/xml', 'text/x-component', 'text/calendar', 'application/vnd.google-earth.kmz'], 'NOT IN');
      }

      $query->condition('s.type', $this->configuration['atom_type']);
    }

    if (isset($this->configuration['provider'])) {
      $query->condition('s.provider', $this->configuration['provider']);
    }

    return $query;
  }

  /**
   * {@inheritdoc}
   */
  public function prepareRow(Row $row) {

    $sid = $row->getSourceProperty('sid');
    $type = $row->getSourceProperty('type');

    // Get Field API field values.
    foreach (array_keys($this->getFields('scald_atoms', $type)) as $field) {
      $row->setSourceProperty($field, $this->getFieldValues('scald_atoms', $field, $sid));
    }

    if ($type === 'audio') {
      $data =  unserialize($row->getSourceProperty('data'));
      $row->setSourceProperty('permalink_url', isset($data['permalink_url']) ? $data['permalink_url'] : 'somthing');
    }

    return parent::prepareRow($row);
  }

  /**
   * {@inheritdoc}
   */
  public function fields() {
    return array(
      'sid' => $this->t('Scald Atom ID'),
      'provider' => $this->t('Provider module name'),
      'type' => $this->t('Scald Atom type'),
      'base_id' => $this->t('Scald Atom base ID'),
      'language' => $this->t('Scald Atom language'),
      'publisher' => $this->t('Scald Atom publisher (User ID)'),
      'actions' => $this->t('Available Scald actions'),
      'title' => $this->t('Scald Atom title'),
      'data' => $this->t('Scald Atom data'),
      'created' => $this->t('Created timestamp'),
      'changed' => $this->t('modified timestamp'),
      'scald_thumbnail_alt' => $this->t('Alternative text for images'),
      'field_credit_value' => $this->t('Credit text value'),
      'field_photo_credit_url_url' => $this->t('photo credit text value'),
      'field_caption_long_value' => $this->t('Caption text value'),
      'field_copyright_value' => $this->t('Copyright text value'),
      'publisher' => $this->t('the user id'),
      'uri' => $this->t('File uri'),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function getIds() {
    $ids['sid']['type'] = 'integer';
    $ids['sid']['alias'] = 's';
    return $ids;
  }
}

What about migrating the references in a field?

Another very useful plugin available from the Migrate Plus module is the migration_lookup process plugin. This plugin allows you to lookup an entity from a previous migration and use the new entity id. Since the publisher column in the scald_atoms table held the Drupal 7 user id, that’s what we used as the source. And our user migration in our new Drupal site was named users. In this example, we set the default value to user id 1 if the lookup didn’t find a match.

uid:
   -
     plugin: migration_lookup
     migration: users
     source: publisher
   -
     plugin: default_value
     default_value: 1

How do I convert embedded media in Wysiwyg fields?

Now that we have converted all our Scald Atom types to Media Types, what about the WYSIWYG fields that have images, videos, etc embedded in them? For this, we created a process plugin that will search for the specific string associated with scald embeds.

In our case, the client was using the Drag and Drop module, so the embed string looked like this: 

<div 
    class="dnd-atom-wrapper"
    data-scald-align="left"
    data-scald-context="thumbnail"
    data-scald-options=""
    data-scald-sid="1234"
    data-scald-type="image"><!-- scald embed --></div>

Our plugin finds instances where this string exists, grabs the ID of the media by parsing out the data-scald-sid value, then looking up the new Drupal media UUID. You can do other things, such as parse the data-scald-context (Scald’s version of view modes) and replace it with the data-view-mode attribute to set a specific view mode. The Drupal media embed string looks like this:

<drupal-media
    data-entity-type="media"
    data-entity-uuid="0000000-0000-0000-0000-0000000000000"
    data-view-mode="thumbnail"></drupal-media>

Here’s the full process plugin code:

<?php
namespace Drupal\your_module\Plugin\migrate\process;

use \Drupal\Core\Database\Database;
use \Drupal\file\Entity\File;
use \Drupal\media\Entity\Media;
use Drupal\migrate\MigrateExecutableInterface;
use Drupal\migrate\ProcessPluginBase;
use Drupal\migrate\Row;
use Drupal\paragraphs\Entity\Paragraph;

/**
 * This processes text fields that contain scald embed codes in the wysiwyg.
 * It converts them to drupal-media embed strings
 *
 * @MigrateProcessPlugin(
 *   id = "scald_wysiwyg_replace"
 * )
 */
class ScaldWysiwyg extends ProcessPluginBase {
  /**
   * {@inheritdoc}
   */
  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    return $this->query($value);
  }

  /**
   * Checks for the existence of some value.
   *
   * @param mixed $value
   *   The body or text field value. 
   *
   * @return mixed|null
   *   Entity id if the queried entity exists. Otherwise NULL.
   */
  protected function query($value) {
    $sids = $this->getSIDs($value);
    $media = $file = NULL;

    foreach ($sids as $sid) {
      $media = $this->getMedia($sid);

      if ($media && $media->uuid->value) {
        //Scaled Media Embed:
        // <div class="dnd-atom-wrapper" data-scald-align="left" data-scald-context="thumbnail" data-scald-options="%7B%22hideCaption%22%3Afalse%2C%22usesSlideshow%22%3A%220%22%2C%22additionalClasses%22%3A%22%22%7D" data-scald-sid="1234" data-scald-type="image"><!-- scald embed --></div>

        // Media Library Embed:
        // <drupal-media data-entity-type="media" data-entity-uuid="0000000-0000-0000-0000-0000000000000" data-view-mode="thumbnail"></drupal-media>

        $value = str_replace('<div class="dnd-atom-wrapper"', '<drupal-media data-entity-type="media" ', $value);
        $value = str_replace('<!-- scald embed --></div>', '</drupal-media>', $value);
        $value = str_replace('data-scald-align="', 'data-align="', $value);
        $value = str_replace('data-scald-sid="' . $sid . '"', ' data-entity-uuid="' . $media->uuid->value . '"', $value);

        // Replace scald-context (old view mode) with a specific new view-mode
        $value = str_replace('data-scald-context="original_size_wysiwyg"', 'data-view-mode="wysiwyg_original"', $value);
      }
    }

    return $value;
  }

  function getSIDs($value) {
    $sids = [];
    // Find matching markup for scald ids. Match them to our media field_sid
    preg_match_all('/data-scald-sid="([0-9])*"/', $value, $matches);

    if (isset($matches[0])) {
      foreach ($matches[0] as $v) {
        if (strpos($v, 'data-scald-sid') !== false) {
          $sid = (int) str_replace(['data-scald-sid="', '"'], '', $v);
          $sids[] = $sid;
        }
      }
    }

    return $sids;
  }

  function getMedia($sid) {
    $media = $file = $fieldName = NULL;

    // See if we already imported media with matching sid.
    // We're using a field we created on migration but you could also query the migration_map_* table
    $query = \Drupal::entityQuery('media');
    $query->condition('field_sid', $sid);
    $entity_ids = $query->execute();

    if ($entity_ids && count($entity_ids) > 0) {
      $media = Media::load(array_values($entity_ids)[0]);
    }

    return $media;
  }

  function getSidData($sid) {
    // Find our media from the migrate db.
    Database::setActiveConnection('migrate');
    $migrateDB = Database::getConnection();
    $query = $migrateDB->select('scald_atoms', 's')
    ->fields('s', array(
      'sid',
      'provider',
      'type',
      'base_id',
      'language',
      'publisher',
      'actions',
      'title',
      'data',
      'created',
      'changed',
      'publisher',
      'data',
    ))
    ->fields('st', array(
      'scald_thumbnail_alt'
    ))
    ->fields('cred', array(
      'field_credit_value'
    ))
    ->fields('url', array(
      'field_photo_credit_url_url'
    ))
    ->fields('cap', array(
      'field_caption_long_value'
    ))
    ->fields('cop', array(
      'field_copyright_value'
    ))
    ->fields('fm', array(
      'uri'
    ));

    // Left Join to your own fields. Here's a few examples of what fields we are using.
    $query->leftJoin('field_data_scald_thumbnail', 'st', 'st.entity_id = s.sid');
    $query->leftJoin('field_data_field_credit', 'cred', 'cred.entity_id = s.sid');
    $query->leftJoin('field_data_field_photo_credit_url', 'url', 'url.entity_id = s.sid');
    $query->leftJoin('field_data_field_caption_long', 'cap', 'cap.entity_id = s.sid');
    $query->leftJoin('field_data_field_copyright', 'cop', 'cop.entity_id = s.sid');
    $query->leftJoin('file_managed', 'fm', 'fm.fid = s.base_id');

    $query->condition('s.sid', $sid);
    $scalds = $query->execute()->fetchAll();

    // Reset the connection to our default db.
    Database::setActiveConnection();

    return $scalds && count($scalds) > 0 ? $scalds[0] : null;
  }
}

At Interactive Knowledge, we specialize in migrations: Drupal 7 to Drupal 8, Wordpress, and more. As content migration always seems to be, no migration is ever the same. If you’re needing help with your migration, feel free to contact our team at Interactive Knowledge to learn how we can help you with your next content migration.