Toolkit Pipeline Tutorial

Toolkit Pipeline Tutorial

This document describes functionality only available if you have taken control over a Toolkit configuration. Please refer to the Shotgun Integrations User Guide for details. This document provides a step-by-step tutorial for Technical Directors (TDs) setting up a simple end-to-end pipeline using Toolkit in an advanced way. New Toolkit users who finish this tutorial will understand the fundamental aspects of the platform as well as how to customize it based on their studio's specific needs. The tutorial also provides an introduction to the out-of-the-box Toolkit apps and how they fit together in the artist workflow.

Table of Contents:

Overview

            The Pipeline

Prerequisites

      Software

      Shotgun Site

      Configured Shotgun Project

      Shotgun Desktop

      Additional Documentation:

Toolkit Configuration

      Understanding Toolkit Configurations

      Configuring the Content Creation Software

            Cleaning up what you don't need:

            Software Paths

      Filesystem Configuration

Customizing the Workflow

      Understanding Hooks

      Custom Publishing Overview

      Publishing Cameras

      Publishing Alembic Caches

      Publishing Shaders

      Publishing Rendered Images

      Configuring the Loader

Artist Workflow

      Modeling

      Rigging

      Surfacing

      Layout

      Animation

      Lighting

      Compositing

Review

Upstream Updates

Conclusion

Overview

The pipeline that follows was created as a part of a 10 week blog series entitled Two Guys and a Toolkit. Reading the blog series will provide additional detail and opinions not found in this tutorial.

The Pipeline

This tutorial covers building a simplified, yet typical, pipeline for animation or visual effects production. By following this tutorial you will build a pipeline that provides all of the pieces necessary to push Assets from modeling through look development, and then into and through a production scene. Here's an overview of how the pipeline will work:



Pipeline Overview

For simplicity, the digital content creation (DCC) software used will be kept to a minimum and limited to Maya and Nuke. Also for the sake of simplicity, data passed between pipeline steps will be limited to Maya ascii files, Alembic caches, and rendered image sequences.

Prerequisites

Software

This tutorial assumes you have experience using Shotgun for tracking and managing production data as well as a basic understanding of Maya and Nuke. You should also have Python scripting experience since the tutorial requires modifying the behavior of Toolkit's default apps via hooks written in Python.

Other technologies that will be referenced throughout the tutorial include Github and Alembic.

Shotgun Site

If you don't have a Shotgun site, you can sign up for a free trial here. You should receive an email with instructions for how to sign into your site once it is ready. When you have the site up and running, you may want to read through the Getting Started with Shotgun documentation.

Configured Shotgun Project

Once you have a site, you'll need to create a new project in Shotgun and configure it as if you were preparing for production to begin. This includes ensuring all of the necessary Shotgun entities are in place and linked up properly. For this tutorial, the Asset, Sequence, Shot, and Task entities are required and should be available by default in a new project. You will create:

  • Two Assets: Teapot character and Table prop
  • One Sequence
  • One Shot linked to the Sequence you created
  • A Task per pipeline step

The tasks will be linked to either the Shot or an Asset, depending on the pipeline step they are associated with.

Here are some screenshots of what your configured project entities should look like in Shotgun:



Teapot and Table Assets



One Shot (note the Sequence link)



Configured Tasks

Shotgun Desktop

You can find the tutorial with instructions for downloading and installing Shotgun Desktop here:


Shotgun Desktop provides an interface that will walk you through the process of setting up Toolkit for your project. The linked tutorial will help you determine which of the default Toolkit configuration templates is appropriate for your needs and how to set up your local filesystem storage.

Use the linked document to get Desktop installed on your machine, then follow the steps outlined in the Running Shotgun Desktop for the first time section using your newly configured Shotgun project.


Be sure to use the Default configuration when choosing a configuration template.




Keep track of the configuration location you choose in the Project Set Up Wizard as well. You'll use that path to explore the project configuration throughout this tutorial.




If this is your first time setting up a Toolkit project, you'll also be prompted to define a storage location for your production data.




Complete the Desktop installation tutorial before continuing with this tutorial.

Additional Documentation:

If you are new to Shotgun or Toolkit and are looking for some additional background information, here are some links you may find useful:

Toolkit Configuration

Understanding Toolkit Configurations

An important step in building a Toolkit pipeline is understanding how Toolkit's pipeline configurations are organized and work. Toolkit configurations define the application integrations to use in your pipeline. This includes identifying which digital content creation software (DCCs) to use as well as which Toolkit apps, engines, and frameworks will be used by artists within those DCCs.

The configuration location you specified during the Desktop installation tutorial is recorded in Shotgun in the Pipeline Configurations page for your project.




Browse to the path on your machine and you will see a directory structure that looks like this:




The three primary directories of interest when building a pipeline are the core, env, and hooks directories. The files in these directories drive how the pipeline works and how it is presented to users. The following sections will dive into modifying the files in these directories to configure the project.

For additional information you can always get more information on project configuration via these documents:

Configuring the Content Creation Software

Because you used the Default configuration while setting up your project in Shotgun Desktop, your project will be configured to use all of the DCCs that Toolkit integrates with. The pipeline outlined in this tutorial does not require all of the DCCs. In addition, the default paths to the DCCs we will be using need to be modified to match the paths on your filesystem. These next sections will walk you through configuring the content creation software for your project.

Cleaning up what you don't need:

The first order of business is to remove the DCC applications you won't be using from Shotgun Desktop. Below is an example of the default DCCs displayed in Shotgun Desktop:




To clean up the DCC launchers from Shotgun Desktop, you'll need to edit the tk-desktop engine settings in the config/env/project.yml config file.

Shotgun Desktop is a Toolkit engine that runs in the project level context. The engine and app configurations in the environment files take the following form:

engine:
<engine-name>:
    apps:
    <app-name>:
        <app-setting1>:
        <app-setting2>:
        ...
    <engine-setting1>:
    <engine-setting2>
    ...

The default Desktop engine configuration looks like this:



Note: The @ is used to import a chunk of configuration from an included file.

You can see a list of launcher apps, one for each DCC, such as tk-multi-launchnuke and tk-multi-launchmaya. To remove a DCC launcher from the tk-desktop engine, simply remove the line from the config.




For the purposes of this tutorial, remove all of the launcher lines except Maya, Nuke, Photoshop, and Screening Room. When you return to the project in Shotgun Desktop, you should see something like this.




In addition to launching DCCs from the Shotgun Desktop application, each of the config/env/shotgun_*.yml files defines launchers available in the browser for a given context. For example, launchers configured in the shotgun_asset.yml will be available in the Project Actions menu on the Assets page in Shotgun.




As with the Desktop engine configuration in project.yml, you can remove the DCCs from the shotgun_*.yml files to provide only the launchers needed by artists on production.

Software Paths

To make sure the DCCs you'll be using in your pipeline are referenced properly, you need to update the paths specified in the default configuration. The paths to the DCCs are stored in the config/env/includes/paths.yml file. Here is what the Maya path configuration looks like in that file:




In this file, find the Maya, Nuke, and Photoshop sections and update the paths to reflect the locations of those software packages on your filesystem.

Filesystem Configuration

The simple pipeline outlined in this tutorial uses the directory structure provided by the Default Toolkit configuration. For additional information about configuring the project directory structure, see the File System Configuration Reference documentation.


This tutorial also uses the templates defined in the Default Toolkit configuration. For more information about the templating system, see the documentation for Configuring File System Templates.

Customizing the Workflow

In order to build the simple pipeline we are outlining, you need to make a few customizations to Toolkit's core app configurations and hooks before an artist could (theoretically) begin working.

Understanding Hooks

Before diving into the workflow customization, you should have a basic understanding of what Toolkit hooks are, how they work, and where they live. If you're not familiar with Toolkit hooks, you should read through the documentation linked below before continuing with the tutorial.


The sections below talk about modifying or customizing an app's hook. For each of these customizations, you will follow these steps:

  1. Locate the app you want to customize under your configuration's install directory, and find the hooks subdirectory. See the image below for an example of the structure of an installed app and how to locate a hook to customize.
  2. Copy the hook (renaming if necessary) into your configuration's top-level hooks directory.
  3. Update the appropriate hook setting in the configuration for the app you are customizing so that it refers to the new overridden hook. Most of the customizations in this tutorial are done in the shot_step.yml file. The exception is the Publishing Shaders section which requires modification to the asset_step.yml config.
  4. Make the code changes to customize your copy of the hook.



Locating the scan_scene_tk-maya hook from the Publisher app and making a copy in the configuration's hook directory.

Custom Publishing Overview

The following sections of this tutorial will outline the steps involved to create a custom publishing workflow. The basic steps that will be covered in each section are:

Add a new custom publish type to the configuration for the appropriate engine
Add logic to the Publisher app's scan_scene hook to identify items in the scene to publish
Add validation of the items to publish in the Publisher's secondary_pre_publish hook
Add export and registration logic to the Publisher's secondary_publish hook

The colored bullets above will be reflected in the sections below to help you identify each step.

Publishing Cameras

An easy customization to add that provides a lot of flexibility to a pipeline is the ability to publish stand-alone cameras. This makes it possible to generate the camera once, typically in layout, and then have all other pipeline steps, such as animation, lighting, and compositing, consume it directly.

Before continuing, read through the Publisher app's documentation to get an overview of the concepts behind the app and how it is configured.


To make Camera publishes work, you'll need to add a new secondary publish type to the Publisher configuration in the appropriate environment. In the project's shot_step.yml environment add the following entry to the Publisher's secondary_outputs field in the tk-maya engine configuration:

        - description: 'Publish a camera'
          display_group: Cameras
          display_name: Camera
          icon: 'icons/camera.png'
          name: camera
          publish_template: maya_shot_camera
          required: false
          scene_item_type: camera
          selected: false
          publish_group: false
          group_name: default
          tank_type: 'Camera'

A quick YAML tip: In YAML syntax, an empty list is represented as [], but a non-empty list has no brackets, and list items each begin on a new line and start with a dash. In this case, secondary_outputs is a list of dictionaries, and the snippet above is a single list item. Indentationwise, the dash that starts a list item should be flush with the list name. So, if the above is the only item in secondary_outputs, it would look like this:

        secondary_outputs:
        - description: 'Publish a camera'
          display_group: Cameras
          display_name: Camera
          icon: 'icons/camera.png'
          name: camera
          publish_template: maya_shot_camera
          required: false
          scene_item_type: camera
          selected: false
          publish_group: false
          group_name: default
          tank_type: 'Camera'

This indicates to the Publisher that there is an additional, secondary publish of type camera that we want the Publisher to recognize.

Note the maya_shot_camera template specified in the new secondary output settings. This identifies where the publish hooks should write the camera to disk. You will need to make sure you have a corresponding template in your config/core/templates.yml file. Here's an example:

    maya_shot_camera:
        definition: '@shot_root/publish/maya/camera/{name}.v{version}.ma'
        root_name: 'primary'


Next, you'll need to tell the Publisher how to identify items of type camera in the file. This is done by modifying the Publisher's scan_scene hook. Here is the snippet you need to add to the hook's execute method:

        # look for cameras to publish
        for camera in cmds.listCameras(perspective=True):
            items.append({"type": "camera", "name": camera})

This code adds a dictionary to the list of secondary publish items for each perspective camera in the session. In a true production scenario, the code might be more sophisticated and only add cameras that match a certain naming convention or that are marked as renderable.


The next step in adding a custom publish type is to modify the Publisher's secondary pre publish hook. This hook allows for validation and verification of the identified secondary publish items.

You'll need to add an additional conditional for the camera type in the execute method. In that conditional, you will call a validation method for the supplied camera item.

elif output["name"] == "camera":
    errors.extend(self.__validate_item_for_camera(item))

Here is a stubbed example of what that validation method you add should look like:

    def __validate_item_for_camera(self, item):
        """
        Validate that the item is valid to be exported to a camera

        :param item:    The item to validate
        :returns:       A list of any errors found during validation

        """

        # add error checking here!
        errors = []
        return errors


Finally, you need to actually handle the publishing of the camera items. This is handled in the execute method of the Publisher's secondary publish hook. As with the pre-publish hook, you will add a conditional for the camera type to call a method to export, register, and publish the camera. The conditional you need to add should look something like this:

         elif output["name"] == "camera":
             try:
                 self.__publish_camera(item, output, work_template,
                                       primary_publish_path, sg_task, comment,
                                       thumbnail_path, progress_cb)
             except Exception, e:
                 errors.append("Publish failed - %s" % e)

And the actual logic for publishing needs to be added as well. Here's the method to handle publishing cameras that you should add to the hook.

Click to expand the publish camera method...

    import re    # add this to the top of your file.

    def __publish_camera(self, item, output, work_template,
        primary_publish_path, sg_task, comment, thumbnail_path, progress_cb):
        """
        Publish a shot camera and register with Shotgun.

        :param item:           The item to publish
        :param output:         The output definition to publish with
        :param work_template:  The work template for the current scene
        :param primary_publish_path: The path to the primary published file
        :param sg_task:        The Shotgun task we are publishing for
        :param comment:        The publish comment/description
        :param thumbnail_path: The path to the publish thumbnail
        :param progress_cb:    A callback that can be used to report progress
        """

        # determine the publish info to use
        #
        progress_cb(10, "Determining publish details")

        # get the current scene path and extract fields from it
        # using the work template:
        scene_path = os.path.abspath(cmds.file(query=True, sn=True))
        fields = work_template.get_fields(scene_path)
        publish_version = fields["version"]
        tank_type = output["tank_type"]
        cam_name = item['name']
        fields['obj_name'] = cam_name
        fields['name'] = re.sub(r'[\W_]+', '', cam_name)

        # create the publish path by applying the fields
        # with the publish template:
        publish_template = output["publish_template"]
        publish_path = publish_template.apply_fields(fields)

        # ensure the publish folder exists:
        publish_folder = os.path.dirname(publish_path)
        self.parent.ensure_folder_exists(publish_folder)

        # determine the publish name
        publish_name = fields.get("obj_name")
        if not publish_name:
            publish_name = os.path.basename(publish_path)

        # Find additional info from the scene:
        #
        progress_cb(10, "Analysing scene")

        cmds.select(cam_name, replace=True)

        # write a .ma file to the publish path with the camera definitions
        progress_cb(25, "Exporting the camera.")
        cmds.file(publish_path, type='mayaAscii', exportSelected=True,
            options="v=0", prompt=False, force=True)

        # register the publish:
        progress_cb(75, "Registering the publish")
        args = {
            "tk": self.parent.tank,
            "context": self.parent.context,
            "comment": comment,
            "path": publish_path,
            "name": publish_name,
            "version_number": publish_version,
            "thumbnail_path": thumbnail_path,
            "task": sg_task,
            "dependency_paths": [primary_publish_path],
            "published_file_type":tank_type
        }
        tank.util.register_publish(**args)

The exported and published camera files in this pipeline are Maya files that contain only the camera itself. They are now able to be referenced downstream via the Loader app.

Publishing Alembic Caches

The Alembic cache publishing support that comes with the default config is geared toward the publishing of Assets. This is great for passing Alembic caches from a modeler to a texture painter that is using something like Mari, but it isn't suited to exporting cached geometry out of Maya in a shot context, to be used by other steps in the shot pipeline downstream. For that workflow, there is already a tutorial that is very well documented. Follow that tutorial, linked below, to setup publishing Alembic caches within a Shot.

Publishing Shaders

Shader management can be a time consuming and complex task when building a pipeline. For the purposes of this example pipeline, we will customize the Publisher app to export Maya shader networks as secondary publishes from the surfacing step. In addition, we'll put together a quick and dirty solution that allows the shaders to be reconnected to the Alembic geometry caches when referenced downstream.

This is, admittedly, a very simple and fragile system. A more robust solution might take into account alternate representations of a surfaced character as well as the asset management side of using external images as textures. Hopefully this example presents a starting point for building a real-world solution.


For this customization, like the camera publishes, the first step is to add an additional secondary publish type to the environment configuration. Since shaders are created upstream while working on an asset, add the following to the asset_step.yml file in the tk-maya section.

        - description: Publish shader network for all geometry in the scene
          display_group: Shader Network
          display_name: Maya Shader Network
          icon: 'icons/maya_shader_network_publish.png'
          name: maya_shader_network
          publish_template: maya_asset_shader_network
          required: false
          scene_item_type: shader_network
          selected: false
          publish_group: false
          group_name: default
          tank_type: 'Maya Shader Network'

As with the Alembic exports, this section of configuration is used by the various hooks to trigger additional code to run, specific to the type defined - in this case the shader_network type. The other fields here define how the additional export type is displayed in the Publisher app, whether it is required, what template to use when writing to disk, etc.

Add this template definition to your project's config/core/templates.yml config file.

    # The location where maya shader networks are stored on disk
    maya_asset_shader_network:
        definition: '@asset_root/publish/maya/surfacing/{name}.v{version}.ma'
        root_name: 'primary'


Now that we've told the Publisher to look for this new type, we need to modify the scan_scene hook to identify the shader networks in the session. Add the following code to the scan_scene hook:

        # export shader networks for any root level meshes.
        for grp in cmds.ls(assemblies=True):
            if cmds.ls(grp, dag=True, type="mesh"):
                items.append({"type":"shader_network", "name":grp})

The code above appends a dictionary to the items list for all root-level meshes. The publish hooks will key off of these items to export shaders for meshes selected in the Publisher UI.


As we saw in the previous sections, the secondary_pre_publish hook runs next to validate the selected items. In that hook's execute method, add the following to the existing conditional to call shader network specific validation code.

    elif output["name"] == "maya_shader_network":
        errors.extend(self.__validate_item_for_maya_shader_network_publish(item))

Here is a stubbed in example of the validations method:

    def __validate_item_for_maya_shader_network_publish(self, item):

        """
        Validate that the item is valid to be exported to a maya shader network

        :param item: The item to validate
        :returns: A list of any errors found during validation that should be reported
                  to the artist
        """

        # add error checking here!
        errors = []
        return errors


Once the user clicks the Publish button in the UI, the actual publish routine will execute. As we know from the previous customizations, the routine lives in the secondary_publish hook. Next, add to the existing conditional by inserting a call to publish the shader network.

            elif output["name"] == "maya_shader_network":
                try:
                    self.__publish_maya_shader_network(item, output,
                        work_template, primary_publish_path, sg_task, comment,
                        thumbnail_path, progress_cb)
                except Exception, e:
                    errors.append("Publish failed - %s" % e)

Then add the method to publish the shader network.

Click to expand the publish shader method...

    def __publish_maya_shader_network(self, item, output, work_template,
        primary_publish_path, sg_task, comment, thumbnail_path, progress_cb):
        """
        Publish shader networks for the asset and register with Shotgun.

        :param item:                    The item to publish
        :param output:                  The output definition to publish with
        :param work_template:           The work template for the current scene
        :param primary_publish_path:    The path to the primary published file
        :param sg_task:                 The Shotgun task we are publishing for
        :param comment:                 The publish comment/description
        :param thumbnail_path:          The path to the publish thumbnail
        :param progress_cb:             A callback that can be used to report progress

        """

        # determine the publish info to use
        #
        progress_cb(10, "Determining publish details")

        # get the current scene path and extract fields from it
        # using the work template:
        scene_path = os.path.abspath(cmds.file(query=True, sn=True))
        fields = work_template.get_fields(scene_path)
        publish_version = fields["version"]
        tank_type = output["tank_type"]
        obj_name = item['name']
        fields['obj_name'] = obj_name
        fields['name'] = re.sub(r'[\W_]+', '', obj_name)

        # create the publish path by applying the fields
        # with the publish template:
        publish_template = output["publish_template"]
        publish_path = publish_template.apply_fields(fields)

        # ensure the publish folder exists:
        publish_folder = os.path.dirname(publish_path)
        self.parent.ensure_folder_exists(publish_folder)

        # determine the publish name:
        publish_name = fields.get("obj_name")
        if not publish_name:
            publish_name = os.path.basename(publish_path)

        # Find additional info from the scene:
        #
        progress_cb(10, "Analysing scene")

        # clean up any hookup nodes that existed before
        _clean_shader_hookup_script_nodes()

        # there's probably a better way to do this. i am jon snow (i know
        # nothing)
        shading_groups = set()
        shad_group_to_obj = {}
        if cmds.ls(obj_name, dag=True, type="mesh"):
            faces = cmds.polyListComponentConversion(obj_name, toFace=True)
            for shading_group in cmds.listSets(type=1, object=faces[0]):
                shading_groups.add(shading_group)
                shad_group_to_obj[shading_group] = obj_name

        shaders = set()
        script_nodes = []
        for shading_group in list(shading_groups):
            connections = cmds.listConnections(
                shading_group, source=True, destination=False)
            for shader in cmds.ls(connections, materials=True):
                shaders.add(shader)
                obj_name = shad_group_to_obj[shading_group]

                # can't seem to store arbitrary data in maya in any
                # reasonable way. would love to know a better way to
                # do this. for now, just create a script node that
                # we can easily find and deduce an object name and
                # shader name. Yes, this is hacky.
                script_node = cmds.scriptNode(
                    name="SHADER_HOOKUP_" + obj_name,
                    scriptType=0, # execute on demand.
                    beforeScript=shader,
                )

                script_nodes.append(script_node)

        if not shaders:
            progress_cb(100, "No shader networks to export.")
            return

        select_nodes = list(shaders)
        select_nodes.extend(script_nodes)

        cmds.select(select_nodes, replace=True)

        # write a .ma file to the publish path with the shader network definitions
        progress_cb(25, "Exporting the shader network.")
        cmds.file(
            publish_path,
            type='mayaAscii',
            exportSelected=True,
            options="v=0",
            prompt=False,
            force=True
        )

        # clean up shader hookup nodes. they should exist in the publish file
        # only.
        _clean_shader_hookup_script_nodes()

        # register the publish:
        progress_cb(75, "Registering the publish")
        args = {
            "tk": self.parent.tank,
            "context": self.parent.context,
            "comment": comment,
            "path": publish_path,
            "name": publish_name,
            "version_number": publish_version,
            "thumbnail_path": thumbnail_path,
            "task": sg_task,
            "dependency_paths": [primary_publish_path],
            "published_file_type":tank_type
        }

        tank.util.register_publish(**args)

    def _clean_shader_hookup_script_nodes():

        # clean up any existing shader hookup nodes
        hookup_prefix = "SHADER_HOOKUP_"
        shader_hookups = {}
        for node in cmds.ls(type="script"):
            if node.startswith(hookup_prefix):
                cmds.delete(node)

Once the export and publish logic is in place, shader network publishes should begin being created. In order to consume them downstream, you'll need to make a modification to the Loader app.


The published shaders in this example are just Maya files, so they can be referenced by the Loader without changing the existing logic. The only change required is to add new logic to hook the shaders back up to the appropriate mesh after they are referenced into the file.

You'll add the following logic to a custom action_hook in the Loader.

def _hookup_shaders(reference_node):

    hookup_prefix = "SHADER_HOOKUP_"
    shader_hookups = {}
    for node in cmds.ls(type="script"):
        if not node.startswith(hookup_prefix):
            continue
        obj_pattern = node.replace(hookup_prefix, "") + "\d*"
        obj_pattern = "^" + obj_pattern + "$"
        shader = cmds.scriptNode(node, query=True, beforeScript=True)
        shader_hookups[obj_pattern] = shader

    for node in (cmds.referenceQuery(reference_node, nodes=True) or []):
        for (obj_pattern, shader) in shader_hookups.iteritems():
            if re.match(obj_pattern, node, re.IGNORECASE):
                # assign the shader to the object
                cmds.file(unloadReference=reference_node, force=True)
                cmds.setAttr(reference_node + ".locked", False)
                cmds.file(loadReference=reference_node)
                cmds.select(node, replace=True)
                cmds.hyperShade(assign=shader)
                cmds.file(unloadReference=reference_node)
                cmds.setAttr(reference_node + ".locked", True)
                cmds.file(loadReference=reference_node)
            else:
                print "NODE: " + node + " doesn't match " + obj_pattern

Don't forget to add a call to _hookup_shaders() at the end of the _create_reference() method.

A full implementation of these customizations are in the hooks defined in the tk-framework-simple repository and the asset_step environment in the tk-config-simple repository.

Publishing Rendered Images

Another customization for this simple pipeline is the ability to publish rendered images from Maya to Nuke for the lighting-to-compositing handoff. To get this handoff working, you'll first create a template that represents the path to the rendered images on disk. Add the following template to the project's config/core/templates.yml config file.

maya_shot_render:
  definition: '@shot_root/work/images/{maya.camera_name}/{maya.layer_name}/v{version}/{name}.{SEQ}.exr'
  root_name: 'primary'

You'll notice the use of the maya.layer_name and maya.camera_name template strings. Add these to the keys section of the same file to act as placeholders that the customized publish hooks will populate when searching for the rendered images on disk.

    # reserve a template key for the maya camera name and render layer name
    maya.camera_name:
        type: str

    maya.layer_name:
        type: str


Once the template is in place, and just like with Alembic caches and shader networks, you'll need to add a new secondary output type for rendered images in the Publisher config for the appropriate environment. Add the following to shot_step.yml.

        - description: 'Publish a rendered image sequence'
          display_group: Renders
          display_name: Rendered
          icon: 'icons/publish_nuke_writenode.png'
          name: rendered_image
          publish_template: maya_shot_render
          required: false
          scene_item_type: rendered_image
          selected: false
          publish_group: false
          group_name: default
          tank_type: 'Rendered Image'

The secondary output definition for the Rendered Image type helps drive how the rendered frames will be displayed in the Publisher. The publish_template field is slightly misleading here since the frames will have already been rendered when the Publisher is visible. This is unlike other secondary Maya publishes, which uses the template to drive where to write those additional files as the publish hooks execute.


Once again, the first hook to modify to get this customization working is the Publisher's scan_scene hook. Unlike the other customizations where we're searching for items in the current Maya session, this code is looking for rendered frames on disk. Add this code to the scan_scene hook:

        # we'll use the engine to get the templates
        engine = tank.platform.current_engine()

        # get the current app
        app = self.parent

        # look up the template for the work file in the configuration
        # will get the proper template based on context (Asset, Shot, etc)
        work_template = app.get_template("template_work")
        work_template_fields = work_template.get_fields(scene_name)
        version = work_template_fields["version"]

        # get all the secondary output render templates and match them against
        # what is on disk
        secondary_outputs = app.get_setting("secondary_outputs")
        render_outputs = [out for out in secondary_outputs if out["tank_type"] == "Rendered Image"]
        for render_output in render_outputs:

            render_template = app.get_template_by_name(render_output["publish_template"])

            # now look for rendered images. note that the cameras returned from 
            # listCameras will include full DAG path. You may need to account 
            # for this in your, more robust solution, if you want the camera name
            # to be part of the publish path. For my simple test, the cameras 
            # are not parented, so there is no hierarchy.

            # iterate over all cameras and layers
            for camera in cmds.listCameras():
                for layer in cmds.ls(type="renderLayer"):

                    # apparently maya has 2 names for the default layer. I'm
                    # guessing it actually renders out as 'masterLayer'.
                    layer = layer.replace("defaultRenderLayer", "masterLayer")

                    # these are the fields to populate into the template to match
                    # against
                    fields = {
                        'maya.camera_name': camera,
                        'maya.layer_name': layer,
                        'name': layer,
                        'version': version,
                    }

                    # match existing paths against the render template
                    paths = engine.tank.abstract_paths_from_template(
                        render_template, fields)

                    # if there's a match, add an item to the render 
                    if paths:
                        items.append({
                            "type": "rendered_image", 
                            "name": layer,

                            # since we already know the path, pass it along for
                            # publish hook to use
                            "other_params": {
                                # just adding first path here. may want to do some
                                # additional validation if there are multiple.
                                'path': paths[0],
                            }
                        })

The goal of the changes are to find out if there were any rendered frames on disk matching the maya_shot_render template. The logic involves iterating over the cameras and render layers in the current Maya session and evaluating the render template with those names. If there are frames on disk that match the evaluated path, then there is something to publish. The code also keeps track of the located rendered images in order to prevent the publish hook from having to do the same work.


Now that rendered images will be added to the list of items to publish, you'll need to modify the secondary_pre_publish hook to make sure the rendered frames are valid and in a good, working state. This should look familiar after seeing the previous customizations.

        elif output["name"] == "rendered_image":
            errors.extend(self.__validate_item_for_rendered_image_publish(item))

Here is a stubbed in example of the validations method:

    # validate rendered images...
    def __validate_item_for_rendered_image_publish(self, item):

        """
        Validate that the item is valid to be exported as a rendered image

        :param item:    The item to validate
        :returns:       A list of any errors found during validation
        """

        # add error checking here. here you can validate the rendered images in
        # whatever way you need to. right number of frames, no missing frames,
        # able to generate a thumbnail, all expected layers present, whatever
        # else you need.
        errors = []
        return errors

Here are some potential validation steps that could be added for a real production:

  • All frames exist
  • File sizes are consistent
  • Permissions are correct
  • Secondary files exist (thumbnails or quicktimes, for example)


The final change to make is to modify the secondary_publish hook to handle the new type. In the hook, add the conditional to call the method to publish rendered images.

            elif output["name"] == "rendered_image":
                try:
                   self.__publish_rendered_images(item, output,
                       work_template, primary_publish_path, sg_task, comment,
                       thumbnail_path, progress_cb)
                except Exception, e:
                   errors.append("Publish failed - %s" % e)

Then add the method to perform the registration and publishing of the rendered-image paths.

Click to expand the publish rendered images method...

    def __publish_rendered_images(self, item, output, work_template,
        primary_publish_path, sg_task, comment, thumbnail_path, progress_cb):
        """
        Publish rendered images and register with Shotgun.

        :param item:                    The item to publish
        :param output:                  The output definition to publish with
        :param work_template:           The work template for the current scene
        :param primary_publish_path:    The path to the primary published file
        :param sg_task:                 The Shotgun task we are publishing for
        :param comment:                 The publish comment/description
        :param thumbnail_path:          The path to the publish thumbnail
        :param progress_cb:             A callback that can be used to report progress
        """

        # determine the publish info to use
        #
        progress_cb(10, "Determining publish details")

        # get the current scene path and extract fields from it
        # using the work template:
        scene_path = os.path.abspath(cmds.file(query=True, sn=True))
        fields = work_template.get_fields(scene_path)
        publish_version = fields["version"]
        tank_type = output["tank_type"]

        # this is pretty straight forward since the publish file(s) have
        # already been created (rendered). We're really just populating the
        # arguments to send to the sg publish file registration below.
        publish_name = item["name"]

        # we already determined the path in the scan_scene code. so just
        # pull it from that dictionary.
        other_params = item["other_params"]
        publish_path = other_params["path"]

        # register the publish:
        progress_cb(75, "Registering the publish")
        args = {
            "tk": self.parent.tank,
            "context": self.parent.context,
            "comment": comment,
            "path": publish_path,
            "name": publish_name,
            "version_number": publish_version,
            "thumbnail_path": thumbnail_path,
            "task": sg_task,
            "dependency_paths": [primary_publish_path],
            "published_file_type": tank_type
        }

        tank.util.register_publish(**args)

In the code, you can see the publish path comes from the other_params dictionary. The rendered-image paths were identified by the scan_scene hook, as mentioned above, so there's no reason to search for them again. In a true production pipeline, you might also do additional processing here, such as setting file permissions, creating symlinks, etc.

Once all of this is in place, the published Rendered Images should be registered with Shotgun and available to the Loader in Nuke without any additional changes.



Secondary publish of rendered images



Rendered Images in the Nuke Loader

Configuring the Loader

Now that you've added several new secondary publish types, you need to make sure that the app recognizes them in order to create new upstream references at each step of the pipeline. Referencing the published files allows the Scene Breakdown app to quickly scan for new versions as upstream artists iterate and publish their work. The Scene Breakdown app supports updating maya references out-of-the-box.


To make the Loader app recognize the new secondary publish types you've added, update the action_mappings section in your configuration's shot_step.yml:

tk-multi-loader2:
  action_mappings:
    Alembic Cache: [reference, import]
    Camera: [reference, import]
    Maya Shader Network: [reference, import]
    Rendered Image: [texture_node]
    Maya Scene: [reference, import]

Once you've added the new action mappings, the Loader will display the customized secondary publish types for referencing.

Artist Workflow

Now that you've put all the customizations into place, it's time to see how they all fit together. The following sections will walk you through the full pipeline from Modeling to Compositing.

Modeling

The first step in the simple pipeline is Modeling. You should have created a Teapot character asset in Shotgun in the prerequisite section titled Configured Shotgun Project. If you did not, go back and create the asset before continuing.

Launch Maya from Shotgun Desktop. Next, model a Teapot or download and import the provided Teapot.




When you're happy with the way your Teapot looks, click the File Save ... button in the Shotgun > Shotgun File Manager ... menu at the top of the Maya interface. In the File Manager interface, browse to the Teapot Asset's Model task and give the file a name.




Next, launch the Publisher via the Shotgun > Publish ... menu.




Click the camera icon to take a screenshot of your Teapot and enter a comment. Then click Publish and wait for the process to complete.




You've now successfully published the model as a Maya file to be consumed by downstream users.

Repeat this section by modeling (or downloading) and publishing a Table prop. When finished, you should have a model published as a Maya file for each of the Teapot and Table assets.

Rigging

Once a model has been published, the next step is to build and publish a rig.

Launch Maya from Shotgun Desktop. When it opens, click the File Save ... button in the Shotgun > Shotgun File Manager ... menu at the top of the Maya interface. In the File Manager interface, browse to the Teapot Asset's Rig task and give the file a name.




Now launch the Loader app using the Shotgun > Load ... menu action. With the Loader app open, you can now select the Teapot model that was published and reference that into your scene.




Now rig the Teapot model that has been loaded into the scene.




Once the rig is ready to be published, launch the Publisher via the Shotgun > Publish ... menu. Publish the rigged asset in the same way as was done for the model. In this case it isn't necessary to publish an Alembic cache for the rig, though doing so will not cause any harm.




Repeat this section by rigging and publishing a Table prop. When finished, you should have a rig published as a Maya file for each of the Teapot and Table assets.

Surfacing

The final step of the asset portion of the workflow is to surface the published models and publish their shader networks.

Launch Maya from Shotgun Desktop. When it opens, click the File Save ... button in the Shotgun > Shotgun File Manager ... menu at the top of the Maya interface. In the File Manager interface, browse to the Teapot Asset's Texturing and Shading task and give the file a name.




As in the rigging step before, launch the Loader app using the Shotgun > Load ... menu action. With the Loader app open, you can now select the Teapot model that was published and reference that into your scene.

Now that the model is in place, surface the teapot.




Once that is done, publish the shader network by launching the Publisher app via the Shotgun > Publish ... menu. Publish the surfaced asset in the same way as was done for the model. When doing so, be sure that the Maya Shader Network secondary item is enabled.




Repeat this section by surfacing and publishing a Table prop. When finished, you should have shaders published as a Maya file for each of the Teapot and Table assets.

Layout

Now it's time to move on to the shot production portion of the workflow. The first step will be to put together a basic layout of the shot, including animation of the shot's camera, and placement of the teapot and table assets in frame.

Launch Maya from Shotgun Desktop. When it opens, click the File Save ... button in the Shotgun > Shotgun File Manager ... menu at the top of the Maya interface. In the File Manager interface, browse to the shot's Camera task and give the file a name.




Now launch the Loader app using the Shotgun > Load ... menu action. With the Loader app open, you can now select the rigs that were published and reference those into your scene.




Now that the rigs for the teapot and table are in the scene, place them as desired. Create and animate a camera for the shot.




Once that is done, publish all components of the shot's layout by launching the Publisher app via the Shotgun > Publish ... menu. Be sure to publish Alembic caches for the two assets, as well as the secondary output for the camera.




Animation

Now that the shot's layout has been published, it's time to add additional animation to the assets in the shot.

Launch Maya from Shotgun Desktop. When it opens, click the File Save ... button in the Shotgun > Shotgun File Manager ... menu at the top of the Maya interface. In the File Manager interface, browse to the shot's Animation task and give the file a name.




Now launch the Loader app using the Shotgun > Load ... menu action. With the Loader app open, you can now select the Maya scene file that was published from the Camera task and reference that into your scene. Once the assets have been referenced, also use the Loader to pull in the published camera.




Now that the layout is in the scene, animate the rigs as desired.




Once that is done, publish all components of the shot's layout by launching the Publisher app via the Shotgun > Publish ... menu. Be sure to publish Alembic caches for the two assets. In this step it's likely not necessary to publish the camera again, as changes to that will more often be performed from the Camera task.




Lighting

In Lighting, you'll pull together all of the published files you've created for the Assets and the Shot.

Launch Maya from Shotgun Desktop. When it opens, click the File Save ... button in the Shotgun > Shotgun File Manager ... menu at the top of the Maya interface. In the File Manager interface, browse to the Shot's Lighting task and give the file a name.




Now launch the Loader app using the Shotgun > Load ... menu action. With the Loader app open, you can now browse to and create references for the following published items:

  • Camera from Layout
  • Table and Teapot Alembic caches from Animation
  • Table and Teapot shader networks




The result should be the combination of everything that was generated upstream.




The next step is to place your lights in the scene as you wish.




Then you'll need to setup your render layers and render to disk. You'll want to make sure the output paths in the render layers align with the templates defined in the Publishing Rendered Images section.

Once that is done, publish the rendered image layers by launching the Publisher app via the Shotgun > Publish ... menu.




Compositing

The last step in the Shot workflow is compositing. Here we can load the rendered elements published from lighting, as well as the Shot's camera from layout and the Alembic caches published from animation. Once all of the necessary data is loaded into Nuke, the final shot can be rendered.

Launch Nuke from Shotgun Desktop. When it opens, click the File Save ... button in the Shotgun > Shotgun File Manager ... menu at the top of the Nuke interface. In the File Manager interface, browse to the shot's Shot Finaling task and give the file a name.




Now launch the Loader app using the Shotgun > Load ... menu action. With the Loader app open, you can now select the rendered elements that were published from the Lighting task and read those into your scene.




The Loader app will create Read nodes in Nuke for each item imported.




Now that the rendered elements have been read into the scene, finish the shot. When this is complete, you can create a Shotgun Quick Daily node, which will allow you to create a Version in Shotgun from the output of the comp.




In addition to the Shotgun Quick Daily node, a Shotgun Write Node is also available to manage the output of the comp with Shotgun Toolkit.




With a Shotgun Write Node in place, the output from the comp can be published using the Publisher app launched from the Shotgun > Publish ... menu.




Review

During the course of production, there will be the need to review the work that has been done in a shot. As mentioned in the compositing section above concerning the ShotgunQuickDaily node, it is possible to submit the media produced by a DCC application to Shotgun.


To browse the media that has been registered with Shotgun, the Screening Room app can be launched from Shotgun Desktop.




Once Screening Room is open, the available media can be filtered by their linked entities, whether that be a Shot or an Asset.




Double click on the media you want to view and then use the playback controls in RV to play it.

Upstream Updates

On production, artists at each step of the pipeline will iterate and publish new versions of their work. Toolkit provides the Scene Breakdown app which displays what versions of referenced published files are out-of-date.


Part of an artist's workflow should be to check for updates to make sure their file is up-to-date with the latest upstream work. This is done, in all pipeline steps, by launching the Scene Breakdown app from Maya's Shotgun menu via Shotgun > Scene Breakdown ...

The app will show all recognized published file references in the file. Out-of-date references will be displayed with a red dot with an "x" inside. Up-to-date references will be displayed with a green dot with a check mark inside.



The Scene Breakdown app. The left shows the state of the file's references before update.
On the right, the camera has been updated from version 6 to version 11.

To update the out-of-date references, simply select the rows with out-of-date references and click the Update Selected button. The images above show the state of the app before and after the update.

If all of the items are shown in green, then the file is up-to-date with the most recent upstream published files.

Conclusion

Hopefully this tutorial has given you a starting point for building your own custom pipeline using Toolkit. You should have an understanding of how to extend the default Toolkit apps to meet the specific needs of your studio.

If there are features or workflows that you feel are outside of the default Toolkit apps, then you can always write your own apps. Here is an excellent document to help you get started writing your first Toolkit app.

As always, if you have additional questions about this tutorial or about Shotgun or Toolkit in general, feel free to email us at support@shotgunsoftware.com.

Follow

30 Comments

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco

    Great Job. Thanks a lot.

  • 0
    Avatar
    Oliver Hilbert

    Brilliant! This is just what I needed. Thanks so much for your efforts - especially the shader pipeline work. Great stuff!

  • 0
    Avatar
    Josh Tomlinson

    That's great to hear Oliver! I'm glad it is proving useful!

    Just to reiterate, the customizations presented are just examples and don't represent battle tested solutions. The shader stuff in particular is pretty fragile and shouldn't be considered much more than a starting point. I'd love to swap it out with a really streamlined example from a Maya expert. :)

    Thanks!

    Josh T.

  • 0
    Avatar
    Oliver Hilbert

    No worry Josh! It's a great starting point for someone learning in the deep end. We have a couple of tools ourselves from previous dev work that handles shader publishes and alembic exports but the UX calls for constant wiki references each pipeline step on how to correctly work. Also not having the clearest picture of how some of our mods where implemented makes things like app updates a challenge. I'll be looking at learning from this tut, and potentially take some ideas from our current pipe and hopefully combine the best of both!

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco

    Hi.

    Maybe i am wrong but it seems that shader networks templates.yml entry is not indicated in the tutorial.

    Thanks

  • 0
    Avatar
    Josh Tomlinson

    Hi Abraham!

    You're right! I've updated the tutorial to include the template we used for publishing shader networks. Thanks for the heads up!

    -Josh T.

  • 0
    Avatar
    Martijng

    Hi Josh,

    great tutorial, I'm getting grips on the implementation now!

    however I'm getting an error on this line in scan_scene_tk-maya when I'm trying to publish:

    work_template = engine.tank.templates.get("maya_shot_work") 

    error:

    Template <Sgtk TemplatePath maya\_shot\_work: sequences/{Sequence}/{Shot}/{Step}/work/maya/{name}.v{version}.{extension}>: Tried to extract fields from path 'P:\m\mediamonks_films\projects\shotgun_dev_sg\assets\Prop\Box-2-Thin\MOD\work\maya\thinbox.v001.ma', but the path does not fit the template.

     

    but my template is fine and the work file has been saved already:

    maya_shot_work:

    definition: '@shot_root/work/maya/{name}.v{version}.{maya_extension}'

    root_name: 'primary'

     

    any ideas what this could be? (or should I mail support with these kind of questions?)

    thanks in advance

    Martijn

     

  • 0
    Avatar
    Josh Tomlinson

    Hi Martijn!

    Looking back at the scan scene hook code we supply in the tutorial, I think that what we propose is less than ideal. It looks like you're publishing form an Asset context (the file you're publishing has \assets\ in the path) which is perfectly valid. By hardcoding the work_template value to maya_shot_work though, it means our implementation will only work in a Shot context. What I think it should do is something more like this:

    # get the current app

    app = self.parent()

    # look up the template for the work file in the configuration

    # will get the proper template based on context (Asset, Shot, etc)

    work_template = app.get_template("template_work")

    Further, I think the render_template stuff on the subsequent lines is the hook could be improved as well. It's hardcoded to look for files matching maya_shot_render. But what if you render in an Asset context? It should probably look more like this:

    secondary_outputs = app.get_setting("secondary_outputs")

    render_outputs = [out for out in secondary_outputs if out["tank_type"] == "Rendered Image"]

    for render_output in render_outputs:

        render_template = app.sgtk.get_template(render_output["publish_template"])

        # process each render template ...

    I haven't tested the above, but I will put in an internal ticket to address this soon since I think what we're presenting currently is not ideal.

    The basic idea that we need to address is that the hooks should be valid in any context, and instead of hard coding template names (which kinda defeats the whole point of configuration) we need to look up the template name in the config. 

    I hope that helps you keep moving forward! Sorry you ran into this, but it's great to get the feedback so we can update for other folks who go through the tutorial!

    Thanks!!

    -Josh T.

  • 0
    Avatar
    Martijng

    I get this error only when I try to publish an asset, which is obvious, cause an asset wont have the rendered image sequence.

    so I guess I would need to check if im publishing for a shot or an asset to enable the scanning for rendered images, right?

    so how could i do this? and where? scan_scene_tk-maya?

     

  • 0
    Avatar
    Josh Tomlinson

    Hey Martijn,

    Yes, the changes I suggested in my previous comment should get the proper templates based on the configuration. You may need to tweak them since I didn't test yet, but hopefully they'll help you continue along. 

    I put in the ticket to update the tutorial and will do that as soon as I can. 

    Thanks again!

    -Josh T.

  • 0
    Avatar
    Oliver Hilbert

    Hi Josh,

    Just going through implementation. To confirm should the camera secondary publishes be registered with the loader? Everything runs successfully and debug appears to be fine - yet I don't see any cameras in the loader - only the actual primary scene publish.

    Cheers,

    Ollie

  • 0
    Avatar
    Josh Tomlinson

    Hi Oliver!

    Yes, see the section on ** Configuring the Loader**. You'll notice the line:

    Camera: [reference, import]

    That tells the loader that upstream registered publishes with the tank_type of Camera to be available for either referencing or importing.

    Let me know if that doesn't get them to show up for you!

    -Josh T.

  • 0
    Avatar
    Martijng

    Hey Josh,

    I'm trying your suggested method but I'm getting this error when calling 

    app = self.parent()

     

    'MultiPublish' object is not callable

    The current environment is asset_step.

    Code Traceback:

    File "p:\m\mediamonks_films\projects\_shotgun__configurations__\shotgun_dev_sg_cfg\install\apps\app_store\tk-multi-publish\v0.8.7\python\tk_multi_publish\publish.py", line 80, in show_publish_dlg

    form = self._app.engine.show_dialog(display_name, self._app, PublishForm, self._app, self)

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\core\python\tank\platform\engine.py", line 1145, in show_dialog

    dialog, widget = self._create_dialog_with_widget(title, bundle, widget_class, *args, **kwargs)

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\core\python\tank\platform\engine.py", line 1046, in _create_dialog_with_widget

    widget = self._create_widget(widget_class, *args, **kwargs)

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\core\python\tank\platform\engine.py", line 1024, in _create_widget

    widget = derived_widget_class(*args, **kwargs)

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\apps\app_store\tk-multi-publish\v0.8.7\python\tk_multi_publish\publish_form.py", line 57, in __init__

    self._initialize()

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\apps\app_store\tk-multi-publish\v0.8.7\python\tk_multi_publish\publish_form.py", line 112, in _initialize

    tasks = self._handler.get_publish_tasks()

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\apps\app_store\tk-multi-publish\v0.8.7\python\tk_multi_publish\publish.py", line 93, in get_publish_tasks

    items = self._scan_scene()

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\apps\app_store\tk-multi-publish\v0.8.7\python\tk_multi_publish\publish.py", line 369, in _scan_scene

    items = [Item(item) for item in self.\
    app.execute_hook("hook_scan_scene")]

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\core\python\tank\platform\bundle.py", line 403, in execute_hook

    return self.__execute_hook_internal(key, hook_name, None, **kwargs)

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\core\python\tank\platform\bundle.py", line 743, in __execute_hook_internal

    ret_value = hook.execute_hook_method(resolved_hook_paths, self, method_name, **kwargs)

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\install\core\python\tank\hook.py", line 283, in execute_hook_method

    ret_val = hook_method(**kwargs)

    File "p:\m\mediamonks_films\projects\__shotgun__configurations__\shotgun_dev_sg_cfg\config\hooks\scan_scene_tk-maya.py", line 95, in execute

    app = self.parent()

    //

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco

    Hi.

    I had the same problem following the tutorial. You purpose some tests like teapot publish, but teapot is an asset in that example, so i got errors in scan scene as long as it had no shot context. So i splitted the scan scene script in two different ones, one for asset and one for shot. So i modified the default hook for asset_step.yml and shot_step.yml. 

    In multi publish definitions for asset step i put: hook_scan_scene: scan_scene_shader_networks

    In multi publish definitions for shot step i put: hook_scan_scene: default

    So i have scan_scene_tk-maya.py and scan_scene_shader_networks.py

    I´m still working on it and i don't know if is a good idea, but i share.

    Thanks.

  • 0
    Avatar
    Josh Tomlinson

    Hi Martijn!

    Oops, sorry about that! That's what I get for posting snippets of untested code... 

    The correct syntax is:

    app = self.parent

    Since parent is a property on the app instance. Here are the docs if you're interested. 

    https://support.shotgunsoftware.com/entries/95441117-Core-API-Reference#The%20Hook%20Base%20Class

    I hope that helps!

    -Josh T.

  • 0
    Avatar
    Josh Tomlinson

    Hi Abraham! 

    Thanks for sharing! Sorry for the confusion! As a part of the internal ticket I submitted, I'll be going back and making sure each step is a little clearer in terms of the context (Asset vs Shot) and clean up the code examples to work for both.

    The separation of the hooks is totally valid if that works for you. You could also use conditionals to perform different scanning operations depending on the context.

    Let me know if you have more questions!

    -Josh T.

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco

    Hi Josh.

    I cannot find where do i need to put the render_template stuff:

    -
    secondary_outputs = app.get_setting("secondary_outputs")

    render_outputs = [out for out in secondary_outputs if out["tank_type"] == "Rendered Image"]

    for render_output in render_outputs:

        render_template = app.sgtk.get_template(render_output["publish_template"])

        # process each render template ...
    Thanks.

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco

    Hi again.

    I think i have carefully followed all the instructions but i cannot have shader network as option in asset secondly publishes. How can i debug each step in publish process?

    Thanks.

  • 0
    Avatar
    Josh Tomlinson

    Hi Abraham! 

    I was suggesting to Martijn that the section in the scan_scene_tk-maya hook probably shouldn't be hardcoding the render_template to maya_shot_render. Rather, it should be looking at the configuration for the environment to find the render templates for each of the secondary outputs of type "Rendered Image" and use their respective publish_templates instead. 

    That would look something like this (again, this is not tested):

    secondary_outputs = app.get_setting("secondary_outputs")
    render_outputs = [out for out in secondary_outputs if out["tank_type"] == "Rendered Image"]
    for render_output in render_outputs:

    render_template = app.sgtk.get_template(render_output["publish_template"])

    # now look for rendered images. note that the cameras returned from
    # listCameras will include full DAG path. You may need to account
    # for this in your, more robust solution, if you want the camera name
    # to be part of the publish path. For my simple test, the cameras
    # are not parented, so there is no hierarchy.

    # iterate over all cameras and layers
    for camera in cmds.listCameras():
    for layer in cmds.ls(type="renderLayer"):

    # apparently maya has 2 names for the default layer. I'm
    # guessing it actually renders out as 'masterLayer'.
    layer = layer.replace("defaultRenderLayer", "masterLayer")

    # these are the fields to populate into the template to match
    # against
    fields = {
    'maya.camera_name': camera,
    'maya.layer_name': layer,
    'name': layer,
    'version': version,
    }

    # match existing paths against the render template
    paths = engine.tank.abstract_paths_from_template(
    render_template, fields)

    # if there's a match, add an item to the render
    if paths:
    items.append({
    "type": "rendered_image",
    "name": layer,

    # since we already know the path, pass it along for
    # publish hook to use
    "other_params": {
    # just adding first path here. may want to do some
    # additional validation if there are multiple.
    'path': paths[0],
    }
    })

    return items

    I was just suggesting replacing the one line:

    work_template = engine.tank.templates.get("maya_shot_work")

    with a loop to retrieve the templates defined in the config. I hope that makes sense.

    -Josh T.

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco

    HI Josh.

    Thanks for the explanation, it was a few lines below and i did´t see that render_template reference.

    Anyway, i change it and still not getting shader network secondary out option. 

    I suppose, when i try to publish something, first of all toolkit looks for asset_step.yml in order to show secondary outputs in secondary outputs of tk-multi-publish section.

    There i have alembic and shader networks configured, but only get alembic in public menu. Maybe if is there any errors in scan scene file, the menu will not appear?

    I´m still have no idea about debuging in order to know if is there any error in some files.

    Thanks again.

  • 0
    Avatar
    Martijng

    Hi Josh,

     

    I've implemented your suggestion and it seems to be working, so thank you!

    The next problem I am going to tackle is the fact that the alembic publish does not work with namespaces apparently.

    If you have any suggestions they are welcome ofcourse.

    cheers,

    Martijn

  • 0
    Avatar
    Josh Tomlinson

    Hi Abraham!

    If the shader networks aren't showing up in the publisher, then there's likely a configuration issue or an issue with the scan scene hook. My suggestion would be to look carefully at the script editor in Maya to see if there are any errors either during startup or when the publisher is launched. You can get additional debugging output by turning on the debug_logging setting in the config for the context you're working in. Here is a doc that explains how to do that:  https://support.shotgunsoftware.com/entries/95445587-How-Do-I-Turn-On-Debug-Logging-

    Hope that helps!

    -Josh T.

  • 0
    Avatar
    Josh Tomlinson

    Hi Martijn! 

    That's great news! I'm glad it's working for you! 

    -Josh T.

  • 0
    Avatar
    Martijng

    Ok I've found that the abcexport mel command does not export to file with more than 1 namespace (:) in the name.

    one solution might be to replace the ':' with '_' but maybe that will have implications down the line we have to deal with.. we'll see

    right now I'll do this in my custom secondary_publish_tk-maya.py

    in function __publish_alembic_cache

    replace this line

    group_name = item["name"].strip("|")

    with:

    group_name = item["name"].strip("|").replace(":", "_")

     

    next problem to tackle will be that the published cameras/alembics are not checked/listed by the breakdown app

    (again suggestions welcome)

     

    and is it ok I post my progress here? I'm guessing this will be beneficial to the development of this tutorial right?

     

     

  • 0
    Avatar
    Josh Tomlinson

    Hi Martijn,

    Have you used the Loader to reference any of the upstream cameras/geometry? The breakdown app looks at the referenced files and matches them against upstream publishes to see if there are any newer versions. If you have a reference to a camera, for example, and a new version is published upstream (same name and everything), then the Breakdown app should show that particular reference with a red dot indicating it can be updated.

    Let me know if you have more questions!

    -Josh T.

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco

    Hi Josh.

    Thanks for your help. I finally realized that i needed to fill the hook name on the asset_step.yml instead of "default" entry, so now i get it.

    Anyway, continuing with tutorial tests for Artist Workflow, i got an error publishing Shader network. I think you forgot to include the method _clean_shader_hookup_script_nodes() in publish hook. I got it from your github, but it seems is not refered in this tutorial.

     

  • 0
    Avatar
    Josh Tomlinson

    Hi Abraham! 

    Thanks for the heads up! I've inserted it into the expandable section: Click to expand the publish shader method...

    -Josh T.

  • 0
    Avatar
    gabriel white

    Do you have any information on sequence workflow. Specifically lead lighters will setup a whole sequence and be rendering not single shots but dozens if not hundreds of shots at a time. Most of the workflow I have seen in Shotgun seems tailored more to a one shot at a time workflow. I need lead lighters to have a "setup sequence" kind of task where they may start on some key shots and then finally end up rendering the whole sequence over a weekend to judge the sequence key lighting in all shots. They would need to submit renders, submit test comps and then submit for review large groups of shots at a time. Since it may take some time to launch a shot in Maya much of this process needs to take place without actually loading each shot Maya. If you have any pointers to setting up this sort of workflow that would be great!

  • 0
    Avatar
    Josh Tomlinson

    Hi Gabriel!

     

    You are correct in that most of the existing toolkit apps and workflows were designed for the common case of an artist working on a task within a shot. That said, the apps are fairly flexible with the hooks they provide, so it may be possible to use them to drive the workflow you're looking to build. I'm not sure how the standard toolkit apps would fit into a workflow where you're not working within a Maya session, but it should be possible to at least publish from a master workfile out to individual shots. Perhaps there's some workflow you could build on top of that to trigger the processes you need. Let me know if you have more questions!

    -Josh T.

  • 0
    Avatar
    Tony Li

    Hello Josh.

    This is working great!  

    I have one question with the rendered image publishing. Is there a way I could publish it into the publish directory? At the moment the images stays into the work folders, but we really want to have the ability to isolate the work directory from the publish directory.( So Comp wont be using wip files that might be overwritten and only use the ones we publish for them that lives inside the published directory)

    My guess is that I have to re path it inside the "Secondary Publish" hook after it scans that files exists but with my limited python knowledge I am having trouble implementing this into the code.

    Any tip to do this will be very helpful!

    Thanks Tony

Please sign in to leave a comment.