BTMash

Disclaimer: This posting was written with Migrate V2.3 and less. 2.4+ has a different handler for images. This post should hopefully still be useful if you just omit the parts about the images and handle those in the way Migrate 2.4 does. For more information on this, look at the excellent post by Mike Ryan. Also, I am diving right into the code with little explanation. If you are new to migrate, take a look at some of my older blog posts (again, mind the disclaimer about 2.3 until I get time to update those posts) to get a better understanding on how to use Migrate.
Updated August 29, 2012: I wrote out a blog post on the new image changes from Migrate 2.4+ a few weeks ago which you can read about here. So take a bit of what you learn from the previous blog posts, and add in all the things from the new one for anything file related (or just look at the new one since I link out to the new reference codebase from there as well).

A few months back, I blogged about creating dynamic migrations. With a small amount of code, you can do something very powerful. You can bring in large amounts of data that need to fit into different places with one simple class. And when all of these containers are holding close to the same kind of data, it makes it an obvious choice. Commerce Migrate approaches migrating data from Ubercart to Commerce in such a way and does a great job of bringing over the core fields of an Ubercart product. But what do you do when you need to add additional sets of data for a particular type of entity bundle? The client that needed my help had various kinds of information attached to their products - taxonomy terms for various vocabularies, additional image fields, text fields, stock, etc. Fields that do not get associated with Commerce products / product displays in the initial migration. When I initially saw this, I was completely stumped - it meant rewriting all the dynamic migrations that were being done by commerce migrate as actual migrations (not a task I was looking forward to given that I would essentially copying/pasting code to get the desired effect without actually using Commerce Migrate).

Thankfully, there is an option to actually set up to migrate pieces of data to an entity that has already been brought into the system. In the Migration class, there is a variable called systemOfRecord which tracks the kind of migration you are going to perform. By default, this is set to Migration::SOURCE. This means that the source data is what will provide the id and its appropriate mapping to your system. But by setting it to Migration::DESTINATION. You need to provide the entity id and migrate takes care of most of the rest for you (there are some hitches I ran into while doing a commerce migration, but I don't yet know if this is an issue within migrate - it is the reason why I have debated writing out this post for so long). So...lets dive into some code!

For this example, I have an ubercart product that was migrated into commerce. It is missing a fair amount of data that was added in by my client (his products had 4 additional text fields, 3 additional image fields, and product stock information which didn't make its way through). He had some other data that was mapped to a product display (which is arguably simpler than this code), but I want to focus on a product migration as I imagine others may be moving into this. As always, we're going to define our class and some of the override functions:

class DEMigrationCommerceProductMigration extends Migration {
 
  protected $file_public_path = "";
  protected $source_drupal_root = "";
 
  public function __construct() {
    ...
  }
 
  public function prepareRow($current_row) {
    ...
  }
 
  public function prepare($entity, stdClass $current_row) {
    ...
  }
 
  private function generateMigrateFile($fid = NULL, $file_data = '') {
    ...
  }
}

The __construct() function is where we will define our migration information, what kind of data from the source will map to in the destination. prepareRow() will define any data coming in that needs to get refined for the migration. And prepare() is where we are presented with the actual entity prior to being saved and we can manipulate it any way we want. I have one helper functions to map out my image and file field information (since there may be multiple rows of file data for a given entity, we cannot pull it in the first time around and so we need to call on it during either our prepareRow() or prepare() functions. And I have a couple of variables on where the files directory along with the base path to the drupal install (which...I believe can now be externally hosted as well so you just provide the url!)

Let's first walk piece by piece through the __construct() function. We first define our class information and some basic data.

  public function __construct() {
    parent::__construct();
    // This is to let migrate know the migration is going to be sourced from data that was already migrated.
    $this->systemOfRecord = Migration::DESTINATION;
    $this->description = t('Update fields for commerce products');
    $arguments = array(
      'type' => 'product',
    );
    $this->dependencies = array('CommerceMigrateUbercartProductProduct');
 
    $this->file_public_path = variable_get('commerce_migrate_ubercart_public_files_directory', variable_get('file_public_path', 'sites/btmash.com/files'));
    $this->source_drupal_root = variable_get('commerce_migrate_ubercart_source_drupal_root', DRUPAL_ROOT);
 
    // Create a map object for tracking the relationships between source rows
    $this->map = new MigrateSQLMap($this->machineName,
      array(
          'nid' => array(
            'type' => 'int',
            'unsigned' => TRUE,
            'not null' => TRUE,
            'description' => 'Ubercart node ID',
            'alias' => 'ucp',
          ),
      ),
      MigrateDestinationEntityAPI::getKeySchema('commerce_product', $arguments['type'])
    );
    // Code to define the connection and initial mapping coming up below.
  }

In the snippet above, I have defined the systemOfRecord = Migration::DESTINATION, which lets migrate get ready to provide me with a somewhat build entity object. I have also made this migration depend on a prior migration (in this case, CommerceMigrateUbercartProductProduct so that anytime I run this migration, it ensures everything from CommerceMigrateUbercartProductProduct has been migrated/updated. And that is really the main gist of it. We can now define our connection and initial mapping.

  public function __construct() {
    // ... Continued from above
    // Create a MigrateSource object, which manages retrieving the input data.
    $connection = commerce_migrate_ubercart_get_source_connection();
    $query = $connection->select('node', 'n');
    $query->innerJoin('content_type_product', 'ctp', 'n.nid = ctp.nid and n.vid = ctp.vid');
    $query->innerJoin('uc_products', 'ucp', 'n.nid = ucp.nid AND n.vid = ucp.vid');
    $query->fields('n', array('nid', 'vid'))
          ->fields('ctp', array('field_award_winner_value', 'field_lp_short_desc_value', 'field_lp_image_fid', 'field_lp_image_data', 'field_lp2_short_desc_value', 'field_lp2_image_fid', 'field_lp2_image_data', 'field_beauty_award_value', 'field_awards_lp_image_fid', 'field_awards_lp_image_data'))
          ->fields('ucp', array('model', 'sell_price'))
          ->condition('n.type', $arguments['type'])
          ->distinct();
 
    $this->source = new MigrateSourceSQL($query, array(), NULL, array('map_joinable' => FALSE));
 
    $this->destination = new MigrateDestinationEntityAPI('commerce_product', $arguments['type']);
    $this->addFieldMapping('product_id', 'nid')->sourceMigration('CommerceMigrateUbercartProductProduct');
 
    // Some more code to define the mappings coming up below.
  }

Since my prior posts, I have learnt some cool things thanks to the Commerce Migrate project :) Firstly, you can define external database sources. In my case, I have opted to use the database connection object Commerce Migrate provides (since my data is coming from that database) which is really just an extra database you would define in settings.php. Once you have your query built out, you then let migrate know about your source. The key part to this line is the map_joinable value. By default, this value is set to TRUE. When this happens. migrate assumes you are dealing with a local database and will try to join up against it. It is smart (because it is faster and there is less computation work required by your server) but if your db is outside the system, then it won't work. After that, we define what we are migrating content into (in this case, it is a commerce_product for which the migration is being handled by the MigrateDestinationEntityAPI - at the time of implementation, it was a part of commerce_migrate but it has since been moved to the migrate_extras module.). We are also going to let migrate know where to get the product ID for this migration. As I mentioned, this is an update to a migration, so this is the one time we actually provide the id :) In this case:

    $this->addFieldMapping('product_id', 'nid')->sourceMigration('CommerceMigrateUbercartProductProduct');

should do the trick. It will check from our prior migration on the product_id that was mapped out and append our new data there. At this stage, the rest of the constructor should look very familiar:

$this->addFieldMapping('commerce_stock', 'stock');
 
    // TEXT FIELDS
    $generic_textarea_arguments = MigrateTextFieldHandler::arguments(NULL, 'full_html');
    $this->addFieldMapping('field_award_winner', 'field_award_winner_value')
      ->arguments($generic_textarea_arguments);
    $this->addFieldMapping('field_lp_short_desc', 'field_lp_short_desc_value')
      ->arguments($generic_textarea_arguments);
    $this->addFieldMapping('field_lp2_short_desc', 'field_lp2_short_desc_value')
      ->arguments($generic_textarea_arguments);
    $this->addFieldMapping('field_beauty_award_desc', 'field_beauty_award_value')
      ->arguments($generic_textarea_arguments);
 
    // IMAGE FIELDS
    $generic_image_arguments = array(
      'file_function' => 'file_copy',
      'file_replace' => FILE_EXISTS_RENAME,
    );
    $this->addFieldMapping('field_lp_image', 'lp_image')
      ->arguments($generic_image_arguments);
    $this->addFieldMapping('field_lp2_image', 'lp2_image')
      ->arguments($generic_image_arguments);
    $this->addFieldMapping('field_awards_lp_image', 'awards_lp_image')
      ->arguments($generic_image_arguments);
 
    // All the items that do not require mappings
    $this->addFieldMapping('sku')->issueGroup(t('DNM'));
    $this->addFieldMapping('type')->issueGroup(t('DNM'));
    $this->addFieldMapping('title')->issueGroup(t('DNM'));
    $this->addFieldMapping('status')->issueGroup(t('DNM'));
    $this->addFieldMapping('created')->issueGroup(t('DNM'));
    $this->addFieldMapping('changed')->issueGroup(t('DNM'));
    $this->addFieldMapping('uid')->issueGroup(t('DNM'));
    $this->addFieldMapping('commerce_price')->issueGroup(t('DNM'));
    $this->addFieldMapping('field_image')->issueGroup(t('DNM'));
    $this->addFieldMapping('path')->issueGroup(t('DNM'));
 
    // All the source migration pieces that do not directly correlate to a destination.
    $this->addUnmigratedSources(array('vid', 'model', 'sell_price', 'field_lp_image_fid', 'field_lp_image_data', 'field_lp2_image_fid', 'field_lp2_image_data', 'field_awards_lp_image_fid', 'field_awards_lp_image_data'));

From the above, we have provided mappings for our stock, the text fields, and our image fields.

And with that, we have the beginnings of a migration for a migration :D Let's populate the prepareRow() and generateMigrateFile() functions:

public function prepareRow($current_row) {
    // There is some crazy stuff here which gets explained below
    // ...
 
    // Figure out stock.
    $connection = commerce_migrate_ubercart_get_source_connection();
    $current_row->stock = 0;
    $results = $connection->query("SELECT * FROM {uc_product_stock} WHERE nid=:nid", array(':nid' => $current_row->nid));
    foreach ($results as $result) {
      $current_row->stock = $result->stock;
    }
 
    // Load images;
    $current_row->lp_image = $this->generateMigrateFile($current_row->field_lp_image_fid, $current_row->field_lp_image_data);
    $current_row->lp2_image = $this->generateMigrateFile($current_row->field_lp2_image_fid, $current_row->field_lp2_image_data);
    $current_row->awards_lp_image = $this->generateMigrateFile($current_row->field_awards_lp_image_fid, $current_row->field_awards_lp_image_data);
  }
 
  private function generateMigrateFile($fid = NULL, $file_data = '') {
    $files = array();
 
    if (!empty($fid)) {
      $connection = commerce_migrate_ubercart_get_source_connection();
      $query = $connection->select('files', 'f');
      $query->fields('f', array('fid', 'filepath'))
        ->condition('fid', $fid);
      $results = $query->execute();
      foreach($results as $record) {
        $value = unserialize($file_data);
        $value['path'] = $this->source_drupal_root .'/'. $record->filepath;
        $files[] = drupal_json_encode($value);
      }
    }
    return $files;
  }

The above helps me get my mappings for the image files known to Migrate in a way it understands. As a note, this will need to get updated since Migrate 2.4 handles images differently (see the disclaimer at the top). But by doing things this way, I know exactly where to change my code for the future.

With this, we should basically be done. And if there weren't any caveats, you might even be done and ready with your migration (ie: I encountered few issues with the core entities supported by Migrate. However, I ran into issues where when I resaved the entity, I actually lost data that was initially associated from the initial migration. In the case of the products, I lost the price, and I lost the core image that came over.

For some reason, the entity lost part of its data during the process() handling and so the entity got saved without that data. A few speculated that it may be trying to look for a mapping and when it can't find it, it gets set to whatever your default value is. I have brought up the issue here and here - the latter may already be fixed but I'm unsure at this time. Anyways, back to the issue at hand. I had a simple way around this - I knew the fields this was affecting (the pricing and the image) and I knew the original entity had this information. So...why not just load the original entity and store it to a variable during prepareRow()?

public function prepareRow($current_row) {
    // Load the current product
    $results = db_query("SELECT destid1 FROM {migrate_map_commercemigrateubercartproductproduct} WHERE sourceid1 = :source_id", array(':source_id' => $current_row->nid));
    foreach ($results as $result) {
      $current_row->pid = $result->destid1;
      $product = commerce_product_load($current_row->pid);
      $current_row->product = clone $product;
    }
    // All that code from above ...

We clone the original product because the product object can be manipulated elsewhere - our clone is copied into memory elsewhere so we are free to work with it without any external influence :)

Since we now have the original product, we can go to the prepare() function and make our modifications.

  public function prepare($entity, stdClass $current_row) {
    $product = $current_row->product;
    $entity->commerce_price = $product->commerce_price;
    $entity->field_image = $product->field_image;
  }

So we have, in effect, "reset" our price and image fields. Alternatively, you could bring them in (again) with the migration so you might not have to deal with this kind of logic. But its certainly an approach that worked at the time.

I'm pasting this entire example on to a github gist so you can go through it as one file as opposed to it being broken down here. And as always, I hope all of this has been helpful. If something doesn't make a lot of sense, leave a comment! I want to keep improving this documentation and hopefully, all of this will be even more useful to other folk looking to migrate content in the Drupal community.