Overview of Toolkit's New Default Configuration

Introduction

The Toolkit legacy configuration has been revamped and released as tk-config-default2. This configuration is now the default configuration when doing the Advanced Project Setup in Shotgun Desktop. The goal of this article is to give an overview of the new configuration structure and to help answer common questions raised by those familiar with the legacy Toolkit configuration. Included is an overview of how to update an existing configuration to the new structure as well as some pointers on how to migrate from the legacy publish app to the new Toolkit publisher, using the new Software entity, and supporting the new setup for populating menus in Shotgun web.

What is the default configuration?

The default configuration is installed for a project by running the Advanced Project Setup on that project in Shotgun Desktop.


Selecting the Default configuration during Advanced Project Setup

It provides a template for customizing and curating studio-specific workflows. The default configuration ships with a default filesystem schema and templates that drive where files live on disk for your project. Also included with the default configuration is all of the supported integrations including 3dsMax, Houdini, Mari, Maya, Motionbuilder, Nuke, and Photoshop. These integrations are configured to allow file management, publishing, and loading workflows for each of these DCCs. The default configuration provides a reference for Toolkit administrators and is an excellent starting point for building a complete studio pipeline.

Why should I use the default configuration structure?

The new default configuration has been reorganized based on client feedback and observation to help maximize efficiency when needing to manually curate your production environments. Much of the redundancy of the legacy configuration has been removed in favor of an include-based structure that allows common engine and app settings changes to be isolated to a single file. 

Understanding what changed

An overview of the structure of the new environment files can be found here.

There are a few structural components of the new configuration that should help administrators understand what has changed:

  • Location descriptors for apps, engines, and frameworks now exist in only one place in the configuration. In the legacy configuration, updating the version of an app or engine required changes in each of the top-level environment files. All app and engine definitions now include the location descriptors from a single place. These bundle descriptor files can be found in env/includes:



    • app_locations.yml      # All app location descriptors defined here
    • engine_locations.yml   # All engine location descriptors defined here
    • frameworks.yml         # All frameworks defined here
  • All engine-specific and app-specific settings are defined in files dedicated to that app or engine. In the legacy configuration, each top-level environment file defined the settings for all the engines and apps. With the new structure, the environment files include engine configuration from engine-specific settings files. The engine settings files then include app-specific settings. This isolates all of the various engine and app settings making it much simpler to determine where to update an engine or app (i.e. go to the engine or app settings file!). For example, in order to change how Maya is configured across all environments, you would edit env/includes/settings/tk-maya.yml. To change how the Publisher is configured across all production environments, you would modify env/includes/settings/tk-multi-publish2.yml

  • Within these app/engine settings files, multiple configurations exist for that app/engine. Each variation of the settings is included by a different environment and is named in a way to make it easy to know which collection of settings correspond to which environment. For example, the settings.tk-maya.asset_step settings in tk-maya.yml define the maya engine settings for the asset_step environment. 



    Similarly, the settings.tk-multi-publish2.nuke.shot_step settings in tk-multi-publish2.yml define the publisher settings for nuke in the shot_step environment.

  • Outside of the environment refactoring, another major change to the configuration is the upgrade from the legacy tk-multi-publish app to the new tk-multi-publish2 app. The new Publisher supports standalone publishing workflows and has been updated in conjunction with the new configuration to add support for classic template-based workflows. For more information on the new publisher, see the Integrations User Guide or the Integrations Developer Guide


    Shotgun Publisher

It should be noted that, for the most part, the core schema and templates remain largely unchanged in this new configuration. The bulk of the changes made involve the structure and organization of the environment files and their included settings.

Engine Settings

Engine settings are now isolated in a settings file that defines configurations of that engine across all environments. For example, the file env/includes/settings/tk-maya.yml defines all Maya configurations used by all environments. The following image shows the Maya settings for the project environment:

Here you can see where it is possible to make changes to engine level settings such as menu_favourites. You can also add app references here. By default, all apps are included from app-specific files. This structure allows administrators to see how an engine is configured for all environments from within a single file.

App Settings

Like engine settings, app settings are now isolated in a settings file that defines configurations of that app across all environments. For example, the file env/includes/settings/tk-multi-publish2.yml defines all Publish2 configurations used by all environments. Here's a snippet showing the Publish2 settings for Maya in the shot_step environment:

You can see here how to modify the app-specific settings for the target environment (as included in the engine settings file). This structure allows administrators to see how an app is configured for all environments from within a single file.

Software paths

The env/includes/paths.yml file found in the legacy configuration has been renamed to env/includes/software_paths.yml. The structure of these files is identical, however the new configuration defines far fewer software paths. This is because the preferred method for resolving software paths with Toolkit is to use the Software entity in Shotgun. 


Software Entity entries as seen in Shotgun

For more information on the Software entity and customizing it to work for your studio, please see the documentation found here.

Removal of shotgun_*.yml environment files

The shotgun_*.yml environment files have previously been used to configure what Toolkit actions are available within the Toolkit action menu in the Shotgun web app. We've now moved away from these extra environment files in favor of configuring the tk-shotgun engine in the same way that all other engines are configured. Instead of having a shotgun_xxx.yml file for each type of entity that we want Toolkit actions for, you will now see the configuration for the Shotgun engine in the more typical environments, like project.yml, shot_step.yml, and others; and hand control of picking the environment to the core pick_environment hook. The goal has been to remove the special behavior surrounding Toolkit's browser integration in favor of a more standardized configuration. These changes to browser integration configuration are backwards compatible, so existing configurations will continue to function as they always have.

Removal of app_launchers.yml

The app launcher definitions have been moved to env/includes/settings/tk-multi-launchapp.yml file. This brings app launching setting across various environments in line with the new structure of the configuration (each app/engine has its own settings file). It should also be noted that many of the app launch configurations have been removed in favor of the auto-discoverable app launch made possible by the Software entity. For more information on that, please see the Software entity documentation

Using the legacy Default configuration

The legacy default configuration will still be available for use via the Advanced Project Setup in Shotgun Desktop. The Github repository will also remain visible and available for download. The legacy default configuration has also been updated with the latest apps, engines, and frameworks to bring it inline, version-wise, with the new default configuration. 

Migrating existing configurations

With the recent release of tk-config-default2, several clients have expressed interest in converting their existing configurations over to this new organizational structure. The following sections aim to provide a recipe for making this transition. 

It should be noted that this process is completely optional. Existing, operational configurations should continue without requiring any changes. 

It should also be noted that migrating an existing configuration to the new structure is an advanced, manual process and requires a thorough understanding of the legacy Toolkit configuration structure, git, and the tank shell command. Proceed carefully. We do not have guardrails here.

Start with tk-config-default2

Our suggestion for migrating your existing configuration is to begin by creating a new test project in Shotgun and using the Advanced Project Setup in Desktop, choosing the new Default configuration when prompted. This will provide you with a working, clean configuration that we can begin to modify.

Next, make sure to familiarize yourself with the new configuration structure, specifically the environment files. You can read more about the new structure in this README and in the sections above.

Clean up unused apps and engines

Once you have a thorough understanding of the new structure, you should remove any engines and apps you don't use in your studio. To do this, you will need to remove any app and engine settings files you don't need from the env/includes/settings folder. You will also need to remove references to any unused apps from the remaining engine settings files and any references to unused engines from the top-level env files.

For example, if your studio doesn't use the tk-nuke-quickdailies app, you would remove the highlighted line above from the Nuke engine settings file.

Likewise, if your studio does not use Motionbuilder, you would remove the highlighted line above as well as the include for tk-motionbuilder.yml. The above image shows a portion of the project.yml file. You may need to remove the unused engine from multiple top-level environment files.

You should also remove the unused engine and app descriptors from the locations files in env/includes/*_locations.yml files.

To remove the Motionbuilder engine, you would need to remove the location descriptor defined in the engine_locations.yml file. Unused app location descriptors would be removed from app_locations.yml.

Update settings for shipped integrations

This should leave the configuration in a state where you're left with only the shipped integrations that your studio uses. You will need to update the remaining app and engine settings to match your existing configuration.

The image above shows the tk-nuke-writenode settings for asset contexts. Your existing configuration may use different/custom templates, or add additional write_nodes. You will need to inspect all of your existing app configurations to determine what settings differ or are missing and update them appropriately. You may find that you need more granular configurations as well, so you may need to add additional settings blocks to be included by the engine settings files.

Note: The new default configuration structure was built using sparse settings, so not all settings will exist within the new structure. You can add settings used in your configuration as needed.

Update to use Software Entities

With the new configuration, nearly all software is launched by way of the Software Entity in Shotgun. After updating the shipped integrations you may have seen the env/includes/settings/tk-multi-launchapp.yml file.

This file is where all software launchers in the configuration are defined:

Screen_Shot_2017-12-04_at_2.49.50_PM.png

You can see the top section in the image above is setup to use the software entity configuration of the launch app. The bottom section contains the remaining, manual software launcher configurations. We recommend using the Software Entity set up if possible. You will need to migrate any custom software launching in your existing configuration to this file and reference it from the appropriate engine settings file. 

One change that is introduced when using Software entity launchers is the need for tk-multi-launchapp to have access to the associated engine. The logic that scans the local machine for software installs is housed in the various engines (tk-maya knows how to find Maya installs, as an example). The result is that when tk-desktop and/or tk-shotgun are configured to make use of Software entity launchers, all necessary engine instances need to be available within the same environment.

The image above shows the engines configured for the project environment. You can see that Shotgun Desktop is included on the highlighted line. Any software that should be launched from Desktop via a Software Entity launcher should have the associated engine configured here as well. You can see the other DCC engines configured here that support Software Entity-based launching such as Maya, Nuke, and Photoshop. With the old shotgun_xxx.yml files in the legacy default configuration only the tk-shotgun engine was required to be configured. 

Validating the config

You can test the configuration at any point to make sure the environment files are valid by running the validate command in the shell.

./tank validate

or

./tank validate asset_step

Add custom apps and engines

Next, you'll want to add into the environment files any engines or apps your studio uses that aren't part of the shipped, default configuration. First, add in location descriptors in the app_locations.yml and engine_locations.yml files. Next, use the existing app and engine settings files as reference to insert your configuration for the new apps and engines. Be sure to reference any added engines in the top-level environment files.

Copy your templates and schema

You will need to copy over your existing configuration's templates.yml and schema folder and replace the ones in the new configuration.


The schema folder and templates.yml file within the config

You may want to keep a copy of the default templates.yml from the new configuration in case you need to update your studio's template.yml. This way you can retrieve and evaluate any new templates specified by the apps/engines you kept from the shipped configuration to see if they should be used or modified to work for your studio.

Update your top-level environments

If your studio makes use of custom environments such as "episode" or "playlist", be sure to create the corresponding environment files within the env folder as well.

The image above shows the top-level environment files. The publishedfile_version.yml is an example of a file that is new to the default2 configuration. It serves double duty to provide menu items for the PublishedFile and Version entity menus in Shotgun. You can use this file and the other top-level environments as reference when transitioning your custom environments.

You'll also need to modify the pick_environment hook to account for your custom environments. If you have a custom pick_environment hook already, you may be able to copy it over from your existing configuration, but you should compare it to the one that comes with the new default configuration to ensure you are accounting for all environments defined.

The image above shows a clause in the default2 hook that accounts for using the publishedfile_version.yml file shown in the previous image.

Update frameworks

Next, update the frameworks.yml file to include any custom frameworks required by your studio.


The frameworks.yml file in the env/includes folder.

This file is always included in the top-level environment files, so adding the proper descriptor should be sufficient.

The image above shows the frameworks being included into the top-level shot.yml file. Any frameworks included will be available to all engines/apps in that environment. 

This would be a good time to run the validation again to ensure you don't have any typos and haven't forgotten a setting.

Update hooks

Finally, you will need to copy over any custom hooks defined in your old configuration. This includes core hooks and hooks for apps and engines. These hooks should continue to work with a few exceptions.


Copy your existing hooks to the hooks folder in the new configuration

If you're using Software Entities for the first time and you have existing tk-multi-launchapp hooks defined, you will want to double check that your customizations are working as expected. The primary difference is the reliance of the legacy hooks on checking the launch app's settings to determine the engine name of the software about to be launched. With the new Software Entity-based launches, this value is not configured. Any hooks relying on that functionality will need to be updated. The launch app hooks now provide the engine name to the hook as an extra argument which should make the update fairly trivial. 

Similarly, the defer_keyword setting for tk-multi-launchapp is not used for Software Entity configurations. If you are relying on this for custom deferred folder creation then you may need to continue using the manual launch approach.

Any custom tk-multi-publish (publish1) hooks will not work with tk-multi-publish2 (publish2). If you've manually include the legacy publisher into the new configuration in the steps above, then your hooks should work just fine. However, if you plan to upgrade to publish2, then you will need to do additional work to migrate to the new publish2 plugin system. We recommend tackling that migration as a separate task (migrating the config is a big enough task). When you are ready to go down that path, see the next section for some pointers.

If you're at this point, you should be very close to a working configuration using the new structure. We suggest running the validation and making tweaks until you're no longer getting any errors. From there, you can use the new project to test the configuration and ensure everything is working the way you need it. 

Migrating from publish1 to publish2

Obviously migrating code and configuration to a new paradigm is a manual process and will be different for every studio. The goal of what follows is to provide some hints as to what to expect and how to translate publish1 concepts into publish2 concepts.

Note: Migrating from publish1 to publish2 is not dependent on using the new config structure. You can use either publisher with any working Toolkit configuration.  

Familiarize yourself with publish2 concepts

When you are ready to migrate existing publish1 hooks to work with publish2, assuming you are comfortable with the concepts defined by publish1, you will need to familiarize yourself with the concepts of the new publisher and its APIs. You can find a full write up in the Integrations Dev Guide. Specifically, it is important to understand the publish2 phases (Acceptance phase especially), the collector hook, and the publish plugins. Once you've read through that, given your understanding of the publish1 hook structure, hopefully the translation path will start to clarify.

Key differences between publish1 and publish2 can be seen in the diagrams below.

publish1.png
A diagram of the publish1 hook and item relationship


publish2.png

A diagram of the publish2 plugin and item relationship

Here are a few high level bullet points to help clarify the differences. Consider these before migrating your custom hooks:

  • Items to be published in publish2 are created by the collector hook. This corresponds to the scan_scene hook in publish1 See the green boxes in the images above. 
  • There is no concept of primary vs. secondary publishes in publish2. There are only items to published See the blue boxes in the images above.
  • In publish1, the pre_publish, publish, and secondary_publish hook operations were static and pre-defined. In publish2, the any number of publish plugins can operate on an item and the phases of the publish are defined by methods in the plugins (validate, publish, finalize). See the tan boxes above.
  • Publish2 makes use of new-style hooks which means the hooks can use inheritance. As you look at the shipped publish2 hooks, you can see how much of the core publish logic exists in a single, base class plugin defined in the app.

Additional notes:

  • One or more publish plugins can accept the collected items to act on them 
  • The collector hook and publish plugins in publish2 have the concept of settings which can be used to define per-hook values that drive their behavior. An example would be defining a publish template on a plugin so that it knows where to copy a file at publish time.
  • Engine-specific publish2 collectors and plugins live with the engines themselves. This can be seen by searching for the {engine} token in the tk-multi-publish2.yml settings file.
  • Reusable, base collector/publish plugins live with the tk-multi-publish2 app.
  • The base file publisher is designed to handle all of the common logic for publishing a file on disk. It can handle in-place publishes or template-based publishes. 
  • The shipped integrations inherit from the base collector and file publish plugins to avoid duplicating commonly used publish logic.

Translating settings from publish1 to publish2

Looking strictly at the settings for publish1, here are some notes on how they translate to publish2:

  • display_name: Not currently available in publish2 (though we are working on it)
  • hook_thumbnail: DCC-specific thumbnail generation can be defined by a session collector plugin, but there is no corresponding hook in publish2
  • hook_copy_file: File copying is handled by the base publish file plugin by default. Any custom publish plugin can be written to do file copying, but there is no corresponding hook in publish2.
  • primary/secondary hooks: As mentioned before, there are distinctions between primary and secondary publishes in publish2. You will need to migrate any custom scan scene logic into a collector hook. Any custom pre/publish/post logic will need to be implemented within a publish plugin. 
  • primary_*: The primary scene_item_type, display_name, description, and icon settings are all defined by the item concept in publish2. When items are created by the collector, the values are provided and will be displayed in the UI.
  • secondary_outputs: Secondary outputs are simply additional items created during the collection phase in publish2. They can be listed as a sibling to the traditional "primary" items like the current maya session, or they can be parented underneath them in a hierarchy to infer whatever type of relationship you require. 
  • expand_single_items: The collector hook can define how items are defined with the UI
  • allow_taskless_publishes: Items can have defined contexts that are associated at publish time. The context can be determined at collection time or manually defined by the user. The publish plugins can determine whether to allow an item without a task to be published in the validation method.
  • templates: Templates can be defined in the collector or plugin settings. It is up to the collector and plugins to define how the templates are used.

Additional references for publish2 plugins

Here are some additional resources to help you make the transition from publish1 hooks to publish2 plugins:

  • The tk-config-default2 publish2 settings file is a great place to see how the new publisher is configured. You can see how the base collector and file publish plugin are configured for the stand alone publisher, and you can see how those same hooks are inherited and used by the DCC-specific configurations. 
  • The base collector defines how files that are dragged/dropped are translated into items for publishing as well as the translation logic from file extension to item type.
  • The base file publish plugin defines the majority of the logic for publishing a file on disk including, how templates are handled, how the publish is registered with Shotgun, and how the publish type is defined.
  • The maya-specific publish2 hooks are also a great resource for seeing how DCC-specific collection and publishing is handled. These hooks live in the maya engine repository and have logic for collecting and publishing the current session, previously exported playblasts and alembic caches, as well as all session geometry. 

Conclusion

Hopefully the steps above were helpful in allowing you to understand the new configuration structure and how to transition to it from your existing configuration. If you have any questions about the new configuration or if you walk through the migration process and require additional clarification, please let us know via support@shotgunsoftware.com

Follow

0 Comments

Please sign in to leave a comment.