Migrating publishes

Migrating your Publishes with Toolkit v0.14

This document describes functionality only available if you have taken control over a Toolkit configuration. Please refer to the Shotgun Integrations Admin Guide for details. If you are running an older Toolkit installation (set up before June 2013), it is possible that you are creating and managing publishes in an old and now deprecated fashion. This document explains how to upgrade such an installation to match current best practices.


Core release v0.14 pushes through a long-discussed change to the Shotgun entities used for publishing. This began as a shotgun-dev list thread on support for multiple representations for Versions in the Shotgun, but quickly extended out into the relationship between review content and publish data. We are moving forward with the community-approved design and the first step is to standardize on common/generic publish entities and field linkages, laying the groundwork for later development in Screening Room.

Additionally, the v0.14 release coincides with Tank's product name change heading into public beta. Going forward, Tank will be considered the working name for the Shotgun Pipeline Toolkit (Sgtk), and having Tank-specific naming on any of the entities will not be maintained long term.

New clients who start with the v0.14 will do so with the new entities. For existing clients with existing publish data, core release v0.14 provides a a migration utility will help clients safely transfer their data from the old entities to the new ones. This document outlines what exactly the migration entails, and all of the options available for coordinating the switch.

The New Entities

Tank makes use of three entities, all of which are project-specific:

  • TankPublishedFile
  • TankType
  • TankDependency

These three entities will be replaced by three new common/generic equivalents that will be used by Sgtk:

  • PublishedFile
  • PublishedFileType
  • PublishedFileDependency

However, PublishedFileType is no longer project-specific, and there will be a global set of types that can be used by any Projects (essentially the same concept as Pipeline Steps). Not all PublishedFileTypes need to be used in all Projects, but there is no longer a need to duplicate the same type record in multiple/all Projects and the same one can be reused.

For more details, here is the approved design document from the shotgun-dev list thread, which covers all of the new entity structure and relationships:

Migration Overview

Here is an overview of what the migration will actually be doing:

  • For each TankType, create a PublishedFileType if one does not already exist with the same 'code' field, and preserve all field metadata including thumbnails.
  • For each TankPublishedFile, create a PublishedFile and preserve all field metadata including thumbnails.
  • For each TankDependency, create a PublishedFileDependency and link it to the appropriate PublishedFile records.
  • Convert existing PipelineConfiguration(s) with a setting to use the new entities going forward.

We are taking every measure possible to ensure a smooth transition, and there are a number of options that admins can utilize to test the migration before switching a production over to the new entities. The legacy 'Tank' entities will remain in place for now, but will ultimately be deprecated. We encourage clients to migrate to the new entities as soon as they reasonably can given production constraints!

Also, the migration can be safely run multiple times on the same data set. The migration will "remember" what it has already converted and only pick up new records. This can be useful in a testing context. More details on this below.

Enable the New Entities

When you are ready to run the migration, first enable the three new entities through Shotgun's Site Preferences:

Run the Migration Utility

To run the migration, use the new 'migrate_published_file_entities' utility:

> tank migrate_published_file_entities

If the command is run at the studio level, ALL PROJECTS will be migrated, ex.:

> /tank_demo/studio/tank migrate_published_file_entities

If the command is run within a specific Project context, only that Project will be migrated, ex.:

> tank_demo/ghosts/tank/tank migrate_published_file_entities

The migration utility will verify that the new entities are enabled and then calculate the number of records it will process, ex.:

Found 371 entities to migrate

Would you like to continue with the migration? [Yes]/No:

Validate Shotgun Schema

The last initial verification step the migration checks is the schema on each entity, making sure that any field on the legacy entities exists on the new entities so that the metadata can be transferred over:

If a field from the legacy entity is missing and should be migrated, create the field on the new entity. It must have the same field name for the migration to pick it up. By sure to also ensure that any entity or other restrictions are identical between the fields.

A Special Mention About the Status Field

It is a very common practice to modify the allowable values for the Status field, so make sure these have been set up correctly on the new entity and match those on the legacy entity.

Engine/App Updates

Nearly all of the engines and apps needed an update to account for the new entities in order to be aware of the setting in the Pipeline Configuration that drives the usage of the new entities. As a result, the migration will prompt you to update if you haven't already. If you run the migration in a Project, you can run the updates directly as part of the migration:

Click to see example output from this step

Update apps before migration?
It is recommended that you update all of the apps in your Pipeline
Configurations before migrating entities to ensure they are ready to support
the new 'PublishedFile' entity types.

Would you like to update all apps now? [Yes]/No:

If you run the migration on ALL PROJECTS, the utility will give you the option of exiting the migration so 'tank updates' can be run on each Project (since that utility can only be run on a single-Project basis).

Starting the Migration

After updating the engines and apps, you are now ready to run the actual migration! It will follow the order above, first migrating TankTypes to PublishedFileTypes and then moving on to the TankPublishedFiles/PublishedFiles.

Note that this migration can take a long time. The example migration of ~400 records took 2 minutes, which is about 3-4 records per second. The biggest performance bottleneck for the migration is thumbnails, since they must be transferred individually over from the old to the new record. Bandwidth will plays a role as well, particularly on the thumbnails, so please be aware and budget your time accordingly!

Click to see example output from this step

                           STARTING THE MIGRATION

Step 1. Migrating entities of type 'TankType'
  Migrating entities...
  [####################] 100% - 1-24 of 24 'TankType' entities
  Updating internal entity links...
  Updating external links from other entity types...

Step 2. Migrating entities of type 'TankPublishedFile'

  Migrating entities...
  [####################] 100% - 301-347 of 347 'TankPublishedFile' entities
  Updating internal entity links...
  Updating external links from other entity types...
  [####################] 100% - 201-216 of 216 'Note' entities
  [####################] 100% - 51-70 of 70 'Version' entities

All entities migrated successfully.

Updating Pipeline Configurations

The last step of the migration is to update the Pipeline Configuration(s) for the Project to toggle the setting that dictates which entities to use. If a Pipeline Configuration isn't converted, it will continue to use the legacy entities even if the migration has completed successfully.

The migration will allow you to choose which Pipeline Configuration(s) to update. The default is all, but it is also possible to update only your own if you are testing, so production can remain on the legacy entities while you test the new ones.

Click to see example output from this step

Step 3. Updating pipeline configurations

In order to fully switch to the new entity types, all Pipeline Configurations
for the 'The Ghosts of Pere Lachaise' project need to be updated.
If you are still testing the migration then you can choose to only update the
current Pipeline Configuration.

Which Pipeline Configurations would you like to update? [All]/Current/None:

Updating pipeline configuration 'Primary' for project 'The Ghosts of Pere
Lachaise' (1 of 1)
 - Updating published entity type in
 - Processing environment file

All Pipeline Configurations were updated successfully.


Last but not least, the migration will notify you of any errors or steps skipped. For example, if you didn't run the engine and app updates, it will remind you to! If you didn't update all of the Pipeline Configurations, it will remind you! And so forth.

Click to see example output from this step

                     Migration completed successfully.
The next steps you should take are:
1. Run the 'tank updates' command to update your apps - you skipped this step
at the beginning!
2. Update any of your own apps or hooks that explicitly use the old
'TankPublishedFile', 'TankType' or 'TankDependency' entity types
3. Migrate entities for all other projects that have not been migrated yet
4. Ensure everything works as expected.

You can safely run this migration command again if needed.  It will only
create new entities where they weren't migrated previously, otherwise it will
just update the existing ones.

Finally, if you have any problems with this migration, please contact: support@shotgunsoftware.com

New Projects

When starting up a new project, the setup_project utility will detect whether or not the new entities are enabled and use them by default if they are.

Testing by Admins

The migration utility was designed to be to be run by admins in a testing context before making a full transition. For clients using Tank in production, or close to it, we strongly encourage running a test migration (or migrations). You do not necessarily need to migrate all Projects at once. This section will go through some of the options available to you.

Rerunning the Migration

To reiterate what was mentioned above: the migration can be run multiple times. With that in mind, there are a few bulletpoints worth calling out:

  • If there are any errors, the migration will figure out how far along it was and pick up where it left off, so anything it already successfully processed won't need to be recreated again.
  • If any custom fields are missed, say if you have additional fields on TankPublishedFile but didn't create them on PublishedFile and skipped that step in the migration, you can rerun the migration after creating the matching fields, and the metadata in those fields will be transferred.
  • You can run the migration on a live Project but only convert your own (cloned) Pipeline Configuration, which will transfer all publishes at that point in time for you to test with. If more publishes are created with the legacy entities during your testing phase, you can rerun the migration to pick up the interim records as well as convert the rest of the Pipeline Configurations.

Backing Out of the Pipeline Configuration Conversion

If you have run the migration but later notice a problem, you can also undo the setting in the Pipeline Configuration to revert back to using the legacy entities. There is a --backout flag that can be passed, that will switch things back:

> tank migrate_published_file_entities --backout

There are two important things to be aware of with this flag:

  • It will not undo the entity migration, and any migrated entities will not be retired.
  • It will not migrate new PublishedFile entities back to the legacy TankPublishedFile entity type.

The flag follows the same hierarchy as any other shell engine command and flag. So, you can choose to back out your own Pipeline Configuration, all of the Pipeline Configurations for a Project, etc.

Finishing Up

Once you're satisfied with the migration, the final step is to disable the legacy Tank entities. These entities do not appear in the normal "Entities" section of the Shotgun Site Preferences. This is mainly to deter potential reuse of these entities for other purposes so we can deprecate them! To disable the legacy Tank entities, go to the "Advanced" section, find the toggle, and set it to no:

Finally, remember to remove any detail page tabs that display the legacy TankPublishedFile, since disabling them in the Shotgun Site Preferences will not remove them from the layouts. Go into Design Mode on your detail pages, delete the legacy tabs, and create new tabs for the new entity. You will also want to ensure your Global Default Layouts have the new entities as well!