File System Reference

File System Configuration Reference

This document is a complete reference of the file system centric configurations in the Shotgun Pipeline Toolkit. It outlines how the template system works and which options are available. It also shows all the different parameters you can include in the folder creation configuration.

Table of Contents:

Introduction

      Part 1 - Folder Creation Syntax

      Shotgun Query Folders

      Create With Parent Folder

      Optional fields

      Examples

      Shotgun List Field Folders

      Pipeline Step Folder

            Different file system layouts for different pipeline steps

      Advanced: Specifying a parent

      Shotgun Task Folder

            Advanced: Specifying a parent

      Workspaces and Deferred Folder Creation

      Current User Folder

      Static folders

      Symbolic Links

      Ignoring files and folders

      Customizing I/O and Permissions

            Data passed to the hook

            Passing your own folder creation directives to the hook

            Adding custom configuration to static folders

      Simple customization of how folders are created

      Part 2 - Configuring File System Templates

      The Keys Section

            Example - An alphanumeric name

            Example - Version number

            Example - A stereo eye

            Example - Image sequences

            Example - Two fields both named version via an alias

            Example - Timestamp

            Example - Shotgun mappings

            Example - String field with two valid values

            Example - Disallowing a value

            Example - Substrings

      The Paths Section

      The Strings Section

      Using Optional Keys in Templates

Introduction

This document explains how to configure the part of Toolkit's configuration related to your file system, including examples. Toolkit handles a lot of files and directories, and you can leverage Toolkit's configuration as a way of expressing how paths are put together and what they mean. The file system is typically accessed in two different and completely separate ways:

  1. Folder Creation: After an object has been created in Shotgun, folders on disk need to be created before work can begin. This can be as simple as having a folder on disk representing the Shot, or can be more complex-for example setting up a user specific work sandbox so that each user that works on the shot will work in a separate area on disk.

    • Toolkit automates folder creation when you launch an application (for example you launch Maya for shot BECH_0010), Toolkit ensures that folders exist prior to launching Maya. If folders do not exist, they are created on the fly. Folders can also be created using API methods, using the tank command in the shell and via the Create Folders menu in Shotgun. A special set of configuration files drives this folder creation process and this is outlined in Part 1 of the document below.
  2. Opening and Saving Work: While working, files need to be opened from and saved into standardized locations on disk. These file locations typically exist within the folder structure created prior to work beginning.

    • Once a folder structure has been established, we can use that structure to identify key locations on disk. These locations are called Templates. For example, you can define a template called maya_shot_publish to refer to published Maya files for Shots. Toolkit apps will then use this template-a publish app may use it to control where it should be writing its files, while a Workfiles App may use the template to understand where to open files from. Inside Toolkit's environment configuration, you can control which templates each app uses. All the key file locations used by Toolkit are therefore defined in a single template file and are easy to overview.

Part 1 - Folder Creation Syntax

The folder configuration maps entities in Shotgun to locations on disk. Rather than using a single configuration file, the configuration is in the form of a "mini file system" which acts as a template for each unit that is configured-this is called the schema configuration. Folders and files will be copied across from this "mini file system" to their target location when Toolkit's folder creation executes. It is possible to create dynamic behavior. For example, a folder can represent a Shot in Shotgun, and you can control the naming of that folder. More specifically, you can pull the name of that folder from several Shotgun fields and then perform character conversions before the folder is created.

configuration

The above image shows a schema configuration. When you run the Toolkit folder creation, a connection is established between an entity in Shotgun and a folder on disk. Toolkit uses this folder schema configuration to generate a series of folders on disk and each of these folders are registered as a Filesystem Location entity in Shotgun. One way to think about this is that Shotgun data (e.g., Shot and Asset names) and the configuration is "baked" out into actual folders on disk and in Shotgun. Configurations always start with a folder named "project". This will always represent the connected project in Shotgun and will be replaced with the Toolkit name for the project. Below this level are static folders. The folder creator will automatically create the sequences folder, for example.

Digging further inside the sequences folder, there is a sequence folder and a sequence.yml file. Whenever Toolkit detects a YAML file with the same name as a folder, it will read the contents of the YAML file and add the desired dynamic behavior. In this case, the sequence.yml file contains the structure underneath the project folder, which consists of three types of items:

  1. Normal folders and files: these are simply copied across to the target location.
  2. A folder with a YAML file (having the same name as the folder): this represents dynamic content. For example, there may be a shot and shot.yml and when folders are created, this shot folder is the template used to generate a number of folders-one folder per shot.
  3. A file named name.symlink.yml which will generate a symbolic link as folders are being processed. [Symbolic links are covered later in this document] (#Symbolic%20Links).

The dynamic configuration setup expressed in the YAML files currently supports the following modes:

  • Shotgun Query folders: Dynamic folder names based on a Shotgun Database Query. For example, this mode can be used to create a folder for every Shot in a project.

  • Shotgun List Field folders: Dynamic folder names based on a Shotgun List Field. For example, this mode can be used to create a folder for every value in the Shotgun List field "Asset Type", found on the Asset Entity in Shotgun.

  • Deferred folders: Only executed when a second folder creation pass is requested via the create folders method of the Toolkit API, usually when an application (such as Maya) is launched. Typically, this method is executed by Toolkit's various application launchers just prior to starting up an application.

  • Current User Folders: A special folder, which represents the current user.

Let's dive deeper into these modes.

Shotgun Query Folders

For a dynamic folder which corresponds to a Shotgun query, use the following syntax in your YAML file:

# the type of dynamic content
type: shotgun_entity

# the shotgun entity type to connect to
entity_type: Asset

# the shotgun field to use for the folder name
name: code

# shotgun filters to apply when getting the list of items
# this should be a list of dicts, each dict containing
# three fields: path, relation and values
# (this is std shotgun API syntax)
# any values starting with $ are resolved into path objects
filters: [ { "path": "project", "relation": "is", "values": [ "$project" ] } ]
  • Set the value of dynamic content type field to be shotgun_entity.
  • The entity_type field should be set to the Shotgun entity from which we want to pull data from (e.g., "Asset", "Shot", "Sequence", "CustomEntity02", etc).
  • The name field is the name that should be given to each folder based on the data in Shotgun.
    • You can use a single field, like in the example above (e.g., name: code).
    • You can use multiple fields in brackets (e.g., name: "{asset_type}_{code}").
    • If you want to include fields from other linked entities, you can use the standard Shotgun dot syntax (e.g., name: "{sg_sequence.Sequence.code}_{code}").
  • The filters field is a Shotgun Query. It follows the Shotgun API syntax relatively closely. It is a list of dictionaries, and each dictionary needs to have the keys path, relation, and values. Valid values for $syntax are any ancestor folder that has a corresponding Shotgun entity (e.g., "$project" for the Project and "$sequence" if you have a sequence.yml higher up the directory hierarchy). For Shotgun entity links, you can use the $syntax (e.g., { "path": "project", "relation": "is", "values": [ "$project" ] }) to refer to a parent folder in the configuration-this is explained more in depth in the examples below.

Create With Parent Folder

In Shotgun, there is nesting within Shotgun data structures. This nesting can be referred to as a Parent to Child relationship, and vice versa. For instance, Sequences are typically parents to Shots in the file system, and likewise, Shots are typically Children to Sequences.

create_with_parent_folder

Note: This filesystem nesting relationship is independent from the Shotgun Hierarchy, and there is no connection between the two. They are configured completely independently.

A shotgun_entity type folder supports an optional flag to control whether the folder creation process tries to recurse down into it when a parent is created, so that the child will also be created. Flags are settings that can only have certain fixed values, in this case "true" or "false". To add this flag, use this example:

# recurse down from parent folder
create_with_parent: true

As mentioned, this setting is optional and set to false by default. If you set it to true, Toolkit create folders for any child entity it finds. To continue with our example, if you want Shots to be created whenever their parent Sequence is created, set create_with_parent to true for the Shot.

Note: the default setting is false, meaning that if you create folders for a Sequence, shot folders will not be created automatically. Also, you will need to add this flag to make it true. There will not be a flag in the shotgun_entity folder specifying false since false is the default behavior.

Optional fields

Typically, when you define the folder name (e.g., {code}_{sg_extra_field}), Toolkit requires all fields to have values in Shotgun. For example, if the sg_extra_field is blank, an error message will be generated. If you have a field that is sometimes populated and sometimes not, you can mark it as optional. This means that Toolkit will include the field if it has a value, and exclude it if the value is blank-without error.

You define optional fields using square brackets, like: {code}[_{sg_extra_field}]. This will generate the following folder names:

  • If the code is BECH_0010 and the sg_extra_field is extra, the folder name will be BECH_0010_extra.

optional_fields_BECH_0010_extra

  • If the code is BECH_0010 and the sg_extra_field is blank, the folder name will be BECH_0010.

optional_fields_BECH_0010

Note: optional fields can only be used to define part of the name of a folder in your schema. An entire folder can not be optional.

Examples

Below are a collection of examples showing how to use the filters syntax.

To find all shots which belong to the current project and are in progress, use the syntax below. Note that the Shotgun Shot entity has a link field called project which connects a shot to a project. We want to make sure that we only create folders for the shots that are associated with the current project. Since there is a project level higher up in the configuration file system, we can refer to this via the $syntax and Toolkit will automatically create to this Shotgun entity link reference. Remember, valid values for $syntax are any ancestor folder that has a corresponding Shotgun entity (e.g., "$project" for the Project and "$sequence" if you have a sequence.yml higher up the directory hierarchy).

entity_type: Shot
filters:
    - { "path": "project", "relation": "is", "values": [ "$project" ] }
    - { "path": "status", "relation": "is", "values": [ "ip" ] }

If you have a Sequence folder higher up the tree and want to create folders for all Shots which belong to that Sequence, you can create the following filters:

entity_type: Shot
filters:
    - { "path": "project", "relation": "is", "values": [ "$project" ] }
    - { "path": "sg_sequence", "relation": "is", "values": [ "$sequence" ] }

To find all assets use this syntax:

entity_type: Asset
filters: [ { "path": "project", "relation": "is", "values": [ "$project" ] } ]

Shotgun List Field Folders

Shotgun list field folders are useful if you want to create one folder for every asset type in Shotgun, for instance. Asset types are list fields in Shotgun, and this folder config type makes it possible to define a layer in the file system that reflects those asset type listings.

list_field_folders

Note: once folders have been created on disk, we strongly advise not to change the value (e.g., the asset type) on the associated data.

When you want a dynamic folder which corresponds to all the items in a Shotgun list field, use the following syntax in your YAML file:

# the type of dynamic content
type: "shotgun_list_field"

# the shotgun entity type to connect to
entity_type: "Asset"

# only create for values which are used in this project.
# this is optional and will be set to false if not specified.
skip_unused: false

# by default, list fields are only created if they are needed by a child entity node
# by setting the create_with_parent parameter to true you ensure that list field
# nodes are always created
create_with_parent: false

# the shotgun field to use for the folder name
field_name: "{sg_asset_type}_type"
  • Set value of dynamic content type field to be shotgun_list_field.
  • The entity_type field should be set to the Shotgun entity from which we want to pull data (for instance, "Asset", "Sequence", "Shot", etc.).
  • The field_name field should be set to the Shotgun field from which the data is pulled from and must be a list type field. You can use expressions if you want to add static text alongside the dynamic content. field_name: "{sg_asset_type}_type" This example expression includes text as well as a template key.

  • The optional skip_unused parameter will prevent the creation of directories for list type field values which are not used (as covered under the Optional Fields section above). Note: setting this to True may negatively affect folder creation performance. Also, the culling algorithm is currently crude and does not work in scenarios where complex filters have been applied to the associated entity.

  • The optional create_with_parent parameter forces the creation of the list_field node, even if there isn't a child entity level node that is currently being processed (see Create With Parent Folder section above).

Pipeline Step Folder

The Pipeline Step folder represents a Pipeline Step in Shotgun. Pipeline Steps are also referred to as Steps.

pipeline_step_folder

# the type of dynamic content
type: "shotgun_step"

# the shotgun field to use for the folder name. This field needs to come from a step entity.
name: "short_name"

You can use name expressions here, just like you can with the Shotgun entity described above. The node will look at its parent, grandparent, etc., until a Shotgun entity folder configuration is found. This entity folder will be associated with the Step and the type of the entity will be used to determine which Steps to create.

Note: If you want to create a top level folder with Pipeline Steps, just use the Shotgun entity node and set the associated type to step.

By default, the Step folder will try to create all the relevant Steps for a particular entity automatically. For example, if the folder creation is triggered for a shot which has five Steps (Layout, Animation, FX, Lighting, Compositing), Step folders will automatically be created for those five Steps (Layout, Animation, FX, Lighting, Compositing).

You can, however, turn this off by using the following syntax:

# recurse down from parent folder
create_with_parent: false

Adding this setting to the configuration means that no Step folders will be created when a Shot folder is created. Instead, Step folders will be created only when you run the folder creation on a Task. This can be useful if you want to configure user sandboxes and other structures which are created just before work starts.

Different file system layouts for different pipeline steps

Imagine you want to have one folder structure for Lighting and Comp and one for everything else. If you want to have different file system layouts for different Pipeline Steps, you can achieve this by adding a filter clause to your config. This filter allows you to scope which Pipeline Steps will be covered by a particular Step's configuration. In our example, you can create two configuration files: step_lightcomp.yml and step.yml. In the first one, you would add the following filter:

filters: [ { "path": "short_name", "relation": "in", "values": [ "Light", "Comp"  ] } ]

The above syntax will only be used when Step folders of the type Light or Comp are being created. For the other file, we want to create a rule for everything else:

filters: [ { "path": "short_name", "relation": "not_in", "values": [ "Light", "Comp"  ] } ]

Now you can define separate sub structures in each of these folders.

Advanced: Specifying a parent

As part of the folder creation, Toolkit needs to associate a Pipeline Step with an entity (e.g., "Shot", "Asset", etc). Toolkit does this by default by looking up the folder tree and picking the first Shotgun entity folder it finds. For example, if you have the hierarchy Sequence > Shot > Step, the Step folder will automatically be associated with the Shot, which is typically what you want.

However, if you have a hierarchy with entities below your primary entity, for example Sequence > Shot > Department > Step, Toolkit will, by default, associate the Step with the Department level, which is not desired. In this case, we need to explicitly tell Toolkit where to look. We can do this by adding the following to the Step configuration:

associated_entity_type: Shot

Shotgun Task Folder

The Task folder represents a Task in Shotgun. By default, the Task folder will not will not be created with its parent. For example, if the folder creation is triggered for a Shot which has a Task node associated, the Task folders will not be created automatically. Instead, Task folders will only be created when the folder creation is executed for the Task (e.g., launching a Task from Shotgun).

task_folder

# the type of dynamic content
type: "shotgun_task"

# the shotgun field to use for the folder name. This field needs to come from a task entity.
name: "content"

You can, however, turn on creation so that Tasks are created with their parent entity by using the following syntax:

# recurse down from parent folder
create_with_parent: true

Similar to a Step, you can also optionally supply a filter parameter if you want to filter which Tasks your folder configuration should operate on.

Once again, you can use name expressions, just like you can with the Shotgun entity described above, where static text can be used alongside dynamic content so that you can create a name that has both dynamic and static context.

name: "task_{content}"

The node will look at its parent, grandparent etc., until a Shotgun entity folder configuration is found. This entity folder will be associated with the task and will be used to determine which task folders to create.

Advanced: Specifying a parent

As part of the folder creation, Toolkit needs to associate a Task with an entity (e.g., a Shot, an Asset, etc.). Toolkit does this by default by looking up the folder tree and picking the first Shotgun entity folder it finds. For example, if you have the hierarchy Sequence > Shot > Task, the Task folder will automatically be associated with the Shot, which is typically what you want.

However, if you have a hierarchy with entities below your primary entity (e.g., below Shot), like Sequence > Shot > Department > Task, Toolkit would by default associate the Task with the department level, which is not desired. In this case, we need to explicitly tell Toolkit where to look, similarly to how we updated this with Steps in the previous section. We can do this by adding the following to the Task configuration:

associated_entity_type: Shot

Workspaces and Deferred Folder Creation

Deferred folder creation means that creation will only be executed when a second folder creation pass is requested via the optional engine parameter in the create folders method of the Toolkit API. Typically, this method is executed by Toolkit's various application launchers just prior to starting up an application. Most folder types support a deferred flag, which is false by default. To make deferred folder creation true, you can add this flag:

# only create this folder when tk.create_filesystem_structure is
# called with tk-maya, tk-nuke or any-custom-string.
defer_creation: ["tk-maya", "tk-nuke", "any-custom-string]

# create this folder when any application launches, but not when normal folder
# creation runs
defer_creation: true

This flag makes it possible to split the folder creation in half-one part that runs in a first "global" pass and a second pass that runs at a later point. Typically, the second pass is associated with the engine launching (although it does not happen automatically since the default is false) and allows for a user to create folders just before engine startup. This allows for two primary workflows:

  1. Workspaces: Application specific folder setups. Folders can be created just before an application launches.
  2. A common workflow for this is to have a Pipeline Step that might require Houdini, Maya, and another Engine, depending on what the shot requires and how an Artist chooses to tackle it. The Artist can create maya/, houdini/, and other directories for that Pipeline Step initially, but if the Artist on a given shot only ever works in Maya, empty folders for Houdini and any other Engine are unnecessary. So, if you defer the folder creation to happen at the time of the launch of individual engines, then if an Artist never uses Houdini, the houdini/ folder will not be created for that shot.
  3. User folders: A user folder is created just before application launch. The user folder config construct (described above) is deferred by default.
  4. This can happen so that instead of basing a user folder on the assigned user in Shotgun, you can create a folder for the current user whenever they launch an Engine. For instance, if you start working on a shot, and you launch Maya, a username folder will be created for you (based on your username in Shotgun), and you will not interfere with anyone else's work.

Tip: If you prefer a normal, static folder to be created when an application (like Maya) launches, just create a config YAML file named the same as the folder and add the following:

# type of content
type: "static"

# only create this folder for maya
defer_creation: "tk-maya"

:::yaml
# type of content
type: "static"

# only create this folder when tk.create_filesystem_structure is
# called with any-custom-string.
defer_creation: "any-custom-string"

Current User Folder

The current user folder is a special construct that lets you set up work areas for different users. A common scenario is if you have multiple artists from a department working on the same shot. User folders can be used so that artists can store their workfiles in their own directories and be able to filter just for their files in the Workfiles App. In this case, the configuration file needs to include the following options:

# the type of dynamic content
type: "user_workspace"

name: "login"
  • Set value of type field to be user_workspace.
  • The name field is the name that should be given to a user folder. It must consist of a combination of fields fetched from People in Shotgun (HumanUser in Shotgun).
  • You can use a single field, like in the example above (e.g., name: login).
  • You can use multiple fields in brackets (e.g., name: "{firstname}_{lastname}").
  • If you want to include fields from other linked entities, you can use the standard Shotgun dot syntax (e.g., name: "{sg_group.Group.code}_{login}").

The current user folder is created as a deferred folder by default, meaning that it will only be executed when a second folder creation pass is requested via the optional engine parameter in the create folders method of the Toolkit API.

Static folders

Static folders (and files) are the most simple type. You can drop them into the configuration structure, and they will automatically get copied across when the folder creation process executes. Here are some examples of static folders (https://github.com/shotgunsoftware/tk-config-default/tree/master/core/schema/project) in the default configuration (note that static folders do not have a corresponding YAML file).

Often, you will not need to go beyond this for static folders; however, Toolkit does support some more advanced functionality for static folders. It is possible to define dynamic conditions to determine if a static folder should get created. For example, you may want to have special static folders that only get created for Pipeline Steps of the Editorial type. In this case, you need to add a YAML configuration file next to the static folder and give it the same name, with the extension "yml". Then, use the following syntax:

# the type of dynamic content
type: "static"

# pick one of the shotgun folders that are above this folder
# in the folder hierarchy. In this case it is a parent folder
# named step that we want to look at when deciding if this
# static folder should be created or not.
constrain_by_entity: "$step"

# we can now define constraints for this step. Constraints are simple
# Shotgun queries, following the same syntax as the other shotgun filters
# shown in previous sections.
#
# In our example, if the parent step matches the constraints given
# in the filter below, the static folder will be created. If not,
# it (and its children) will be ignored by the folder creation process.
constraints:
    - { "path": "short_name", "relation": "is", "values": [ "edit" ] }

By default, static folders will automatically get created together with their parent folder. There may be cases where this is not desirable, and in those cases you can add a special flag to indicate that the static folder should not be created together with its parent:

# do not recurse down automatically
create_with_parent: false

Symbolic Links

You can create symbolic links (symlink) as part of the dynamic folder creation. If you want to create a symbolic link with the name artwork, create a file in your schema configuration named artwork.symlink.yml. This will be identified by the system as a symbolic link request and will not be copied across, but will instead be processed.

The artwork.symlink.yml file must, at the very least, contain a target key:

# Example of a .symlink.yml file

# A target parameter is required.
target: "../Stuff/$Project/$Shot"

# Additional parameters will be passed to the hook as metadata
# so you can for example include permission hints or other stuff
# that you may need for advanced customization
additional_param1: abc
additional_param2: def

If the target parameter contains $EntityType tokens such as $Asset, $Shot, or $Project, these will attempt to be resolved with the name of the folder representing that entity (Asset, Shot, Project, etc.). Toolkit will look up the filesystem tree for these values and if they are not defined higher up in the tree, an error will be reported.

Symlink creation happens (like all input/output, or I/O) inside the folder processing hook. A special symlink action is passed from the system into the hook, and you will get the name of the symlink, the fully resolved target, and all the YAML metadata contained within the definition file along with this request. For our artwork example above, we create the folder under the Shot like this:

 {'action': 'symlink',
  'path': '/mnt/projects/chasing_the_light/Sequences/AA/AA001/artwork'
  'target': '../Stuff/chasing_the_light/AA001',
  'metadata': {'target': '../Stuff/$Project/$Shot', 'additional_param1': 'abc', 'additional_param2': 'def'}
  }

Ignoring files and folders

Files that are placed in the schema scaffold will be copied across into the target area as part of the folder creation. This copy process is handled by a core hook, so for example, permissions handling can be customized for a project or studio.

Note: There are more details on this kind of handling in the Customizing I/O and Permissions section Customizing I/O and Permissions section under Simple Customization. We have a process_folder_creation core hook (https://github.com/shotgunsoftware/tk-core/blob/master/hooks/process_folder_creation.py#L62-L71) that handles a lot of folder setup. You can add chmod calls into this hook (and/or set permissions as you mkdir), thereby setting permissions for the folders you are creating.

Sometimes it can be useful to exclude certain files and folders from being copied across as part of the folder creation. For example, if you store your folder creation configs in Git or SVN, you will have .git and .svn folders that you will not want to copy to each Shot or Asset folder. If there are files which you do not want to have copied, a file named ignore_files can be placed in the config/core/schema folder inside the project configuration. This file should contain glob-style patterns to define files not to copy. Each pattern should be on a separate line:

# This is a good example of a standard ignore_files file

.svn                # no svn temp files to be copied across at folder creation time
.git                # no git temp files to be copied across at folder creation time
.DS_Store           # no mac temp files to be copied across at folder creation time

You can also use wildcards. For example, if you need to exclude all files with the TMP extension, just add a *.tmp line to the file.

# This is a good example of a standard ignore_files file

.svn                # no svn temp files to be copied across at folder creation time
.git                # no git temp files to be copied across at folder creation time
*.tmp           # no files with tmp extension to be copied across at folder creation time

Customizing I/O and Permissions

Shot and Asset folders often need to be created with special permissions and parameters. Sometimes this is as simple as setting permission bits during the folder creation, and sometimes it may be as complex as sending a remote request to a special folder creation server that will create the folders with the appropriate credentials, groups, and permissions.

It is also common that folders on different levels in the file system tree need to have different permissions; a work area folder is typically writeable for everybody, whereas a shot folder may have much stricter permissions.

Toolkit allows for customization of the folder creation via a single hook. This is a core hook and it is named process_folder_creation.py. As the folder creation API call is traversing the folder configuration and deciding which folders should be created, it builds up a list of items that could be created. These items can be both files and folders. As the final step of the folder creation, this list is passed to a hook to handle the actual folder processing. You can examine the default process_folder_creation core hook here (https://github.com/shotgunsoftware/tk-core/blob/master/hooks/process_folder_creation.py#L62-L71).

Data passed to the hook

The folder creation hook is executed just once for each folder creation request. All the folder creation data is passed in a list to the hook and the hook typically loops over this and creates the folders according to the parameters passed from the Core API.

The data in the list is always a depth first recursion, starting with the top level folders and files and then traversing deeper and deeper. Here is an example of what the data passed to the hook may look like:

[

 {'action': 'entity_folder',
  'entity': {'id': 88, 'name': 'Chasing the Light', 'type': 'Project'},
  'metadata': {'root_name': 'primary', 'type': 'project'},
  'path': '/mnt/projects/chasing_the_light'},

 {'action': 'folder',
  'metadata': {'type': 'static'},
  'path': '/mnt/projects/chasing_the_light/sequences'},

 {'action': 'entity_folder',
  'entity': {'id': 32, 'name': 'aa2', 'type': 'Sequence'},
  'metadata': {'entity_type': 'Sequence',
               'filters': [{'path': 'project',
                            'relation': 'is',
                            'values': [<tank.folder.folder_types.FilterExpressionToken object at 0x10ca04c90>]}],
               'name': 'code',
               'type': 'shotgun_entity'},
  'path': '/mnt/projects/chasing_the_light/sequences/aa2'},

 {'action': 'entity_folder',
  'entity': {'id': 1184, 'name': 'moo87', 'type': 'Shot'},
  'metadata': {'entity_type': 'Shot',
               'filters': [{'path': 'sg_sequence',
                            'relation': 'is',
                            'values': [<tank.folder.folder_types.FilterExpressionToken object at 0x10ca04b10>]}],
               'name': 'code',
               'type': 'shotgun_entity'},
  'path': '/mnt/projects/chasing_the_light/sequences/aa2/moo87'},

 {'action': 'copy',
  'metadata': {'entity_type': 'Shot',
               'filters': [{'path': 'sg_sequence',
                            'relation': 'is',
                            'values': [<tank.folder.folder_types.FilterExpressionToken object at 0x10ca04b10>]}],
               'name': 'code',
               'type': 'shotgun_entity'},
  'source_path': '/mnt/software/tank/chasing_the_light/config/core/schema/project/sequences/sequence/shot/sgtk_overrides.yml',
  'target_path': '/mnt/projects/chasing_the_light/sequences/aa2/moo87/sgtk_overrides.yml'},

 {'action': 'create_file',
  'metadata': {'entity_type': 'Shot',
               'filters': [{'path': 'sg_sequence',
                            'relation': 'is',
                            'values': [<tank.folder.folder_types.FilterExpressionToken object at 0x10ca04b10>]}],
               'name': 'code',
               'type': 'shotgun_entity'},
  'content': 'foo bar',
  'target_path': '/mnt/projects/chasing_the_light/sequences/aa2/moo87/automatic_content.txt'},

 {'action': 'symlink',
  'path': '/mnt/projects/chasing_the_light/Sequences/AA/AA001/artwork'
  'target': '../Stuff/chasing_the_light/AA001',
  'metadata': {'target': '../Stuff/$Project/$Shot', 'additional_param1': 'abc', 'additional_param2': 'def'}
  },

]

The data is a list of dictionaries. Each dictionary has a key called action. This key denotes the type of I/O item that is requested. If you are implementing the folder creation hook, you need to add support for the following different actions:

  • entity_folder: A folder on disk which is associated with a Shotgun entity.
  • folder: A folder on disk.
  • copy: A file that needs to be copied from a source location to a target location.
  • create_file:- A file that needs to be created on disk.
  • symlink: A symbolic link should be created.

Each of the actions have a different set of dictionary keys. For example, the entity_folder action has an entity key which contains the details of the entity that it is connected to. The create_file has a source_path and a target_path key which inform the hook which file to copy and where.

All actions also have a key called metadata. This key represents the YAML configuration data that comes from the associated configuration file in the schema setup. You can see in the example above how the metadata key for a Shotgun folder contains all the filter and naming information that is set up within the schema configuration. For example, here is the metadata for the Shot folder in the example above:

{'action': 'entity_folder',
  'entity': {'id': 1184, 'name': 'moo87', 'type': 'Shot'},
  'metadata': {'entity_type': 'Shot',
               'filters': [{'path': 'sg_sequence',
                            'relation': 'is',
                            'values': [<tank.folder.folder_types.FilterExpressionToken object at 0x10ca04b10>]}],
               'name': 'code',
               'type': 'shotgun_entity'},
  'path': '/mnt/projects/chasing_the_light/sequences/aa2/moo87'}

...which corresponds to the shot.yml schema configuration file:

# Copyright (c) 2013 Shotgun Software Inc.
#
# CONFIDENTIAL AND PROPRIETARY
#
# This work is provided "AS IS" and subject to the Shotgun Pipeline Toolkit
# Source Code License included in this distribution package. See LICENSE.
# By accessing, using, copying or modifying this work you indicate your
# agreement to the Shotgun Pipeline Toolkit Source Code License. All rights
# not expressly granted therein are reserved by Shotgun Software Inc.

# the type of dynamic content
type: "shotgun_entity"

# the shotgun field to use for the folder name
name: "code"

# the shotgun entity type to connect to
entity_type: "Shot"

# shotgun filters to apply when getting the list of items
# this should be a list of dicts, each dict containing
# three fields: path, relation and values
# (this is std shotgun API syntax)
# any values starting with $ are resolved into path objects
filters: [ { "path": "sg_sequence", "relation": "is", "values": [ "$sequence" ] } ]

Note that the dynamic token $sequence has been resolved into an actual object at runtime.

Passing your own folder creation directives to the hook

In addition to the various configuration directives required by Toolkit, you can also define your own configuration items as part of the schema configuration. These are passed into the hook via the metadata key described above, and can be used to drive folder creation.

For example, if you had the following structure in your schema setup:

# the type of dynamic content
type: "shotgun_entity"

# the shotgun field to use for the folder name
name: "code"

# the shotgun entity type to connect to
entity_type: "Shot"

# shotgun filters to apply when getting the list of items
filters: [ { "path": "sg_sequence", "relation": "is", "values": [ "$sequence" ] } ]

# user settings
studio_permissions_level: "admin"

...the data passed via the folder creation hook would be:

{'action': 'entity_folder',
  'entity': {'id': 1184, 'name': 'moo87', 'type': 'Shot'},
  'metadata': {'entity_type': 'Shot',
               'filters': [{'path': 'sg_sequence',
                            'relation': 'is',
                            'values': [<tank.folder.folder_types.FilterExpressionToken object at 0x10ca04b10>]}],
               'name': 'code',
               'type': 'shotgun_entity',
               'studio_permissions_level': 'admin'},
  'path': '/mnt/projects/chasing_the_light/sequences/aa2/moo87'}

Now the special parameter studio_permissions_level is passed into the hook and you can use that, for example, to control file permissions. You can also pass arbitrarily complex data structures using this method. A typical usecase for this would be to control permissions at a very detailed level.

Adding custom configuration to static folders

Typically, when you create a folder inside the folder schema configuration, and it does not have a corresponding YAML file, Toolkit will assume that it is a static folder and will simply create it.

If you would like to associate custom configuration metadata with a static folder, you have to create a YAML configuration file with the static type. For example, let's say you have a static assets folder just under the project root and would like to group together assets and add custom configuration metadata. To achieve this, create the following assets.yml file:

type: static
studio_permissions_level: "admin"

The configuration data passed to the hook will then contain the following:

{'action': 'folder',
 'metadata': {'studio_permissions_level': 'admin', 'type': 'static'},
 'path': '/mnt/projects/chasing_the_light/assets'},

Again, arbitrarily complex data can be passed from the YAML configuration file into the hook in this way.

Simple customization of how folders are created

A simple folder creation hook could look something like this:

class ProcessFolderCreation(Hook):

    def execute(self, items, preview_mode, **kwargs):
        """
        The default implementation creates folders recursively using open permissions.

        This hook should return a list of created items.

        Items is a list of dictionaries. Each dictionary can be of the following type:

        Standard Folder
        ---------------
        This represents a standard folder in the file system which is not associated
        with anything in Shotgun. It contains the following keys:

        * "action": "folder"
        * "metadata": The configuration yaml data for this item
        * "path": path on disk to the item

        Entity Folder
        -------------
        This represents a folder in the file system which is associated with a
        shotgun entity. It contains the following keys:

        * "action": "entity_folder"
        * "metadata": The configuration yaml data for this item
        * "path": path on disk to the item
        * "entity": Shotgun entity link dict with keys type, id and name.

        File Copy
        ---------
        This represents a file copy operation which should be carried out.
        It contains the following keys:

        * "action": "copy"
        * "metadata": The configuration yaml data associated with the directory level
                      on which this object exists.
        * "source_path": location of the file that should be copied
        * "target_path": target location to where the file should be copied.

        File Creation
        -------------
        This is similar to the file copy, but instead of a source path, a chunk
        of data is specified. It contains the following keys:

        * "action": "create_file"
        * "metadata": The configuration yaml data associated with the directory level
                      on which this object exists.
        * "content": file content
        * "target_path": target location to where the file should be copied.

        """

        # set the umask so that we get true permissions
        old_umask = os.umask(0)
        folders = []
        try:

            # loop through our list of items
            for i in items:

                action = i.get("action")

                if action == "entity_folder" or action == "folder":
                    # folder creation
                    path = i.get("path")
                    if not os.path.exists(path):
                        if not preview_mode:
                            # create the folder using open permissions
                            os.makedirs(path, 0777)
                        folders.append(path)

                elif action == "copy":
                    # a file copy
                    source_path = i.get("source_path")
                    target_path = i.get("target_path")
                    if not os.path.exists(target_path):
                        if not preview_mode:
                            # do a standard file copy
                            shutil.copy(source_path, target_path)
                            # set permissions to open
                            os.chmod(target_path, 0666)
                        folders.append(target_path)

                elif action == "create_file":
                    # create a new file based on content
                    path = i.get("path")
                    parent_folder = os.path.dirname(path)
                    content = i.get("content")
                    if not os.path.exists(parent_folder) and not preview_mode:
                        os.makedirs(parent_folder, 0777)
                    if not os.path.exists(path):
                        if not preview_mode:
                            # create the file
                            fp = open(path, "wb")
                            fp.write(content)
                            fp.close()
                            # and set permissions to open
                            os.chmod(path, 0666)
                        folders.append(path)

                else:
                    raise Exception("Unknown folder hook action '%s'" % action)

        finally:
            # reset umask
            os.umask(old_umask)

        return folders

Part 2 - Configuring File System Templates

The Toolkit templates file is one of the hubs of the Toolkit configuration. There is always one of these files per project and it resides inside the config/core folder inside your pipeline configuration.

configuration

This file contains definitions for templates and their keys.

A key is a dynamic field we defined. It can be a name, a version number, a screen resolution, a shot name etc. Keys are configured with types, so we can define that a key should be a string or an int for example. They are also formatted, so we can define that a string should only contain alpha numeric characters, or that all integers should be padded with eight zeroes.

A template is a dynamic path. An example of a template is shots/{shot}/publish/{name}.{version}.ma. This template could for represent maya publishes for a shot - the bracketed fields are keys.

The templates file is divided into three sections: keys, paths and strings.

The Keys Section

Keys define what values are acceptable for fields. In the template config file keys are defined in the form:

key_name:
   type: key_type
   option: option_value
   option: option_value

Key type is either str, int, or sequence. Str keys are keys whose values are strings, int keys are keys whose values are integers, and sequence keys are keys whose values are sequences of integers.

In addition to specifying the type, you can also specify additional options. The following options exist:

  • default: default_value - Value used if no value was supplied. This can happen if you are using the Toolkit API and trying to resolve a set of field values into a path for example.

  • choices: [choice1, choice2, etc] - An enumeration of possible values for this key.

  • exclusions: [bad1, bad2, etc] - An enumeration of forbidden values for this key. If key is of type sequence, frame spec values cannot be invalidated with this setting.

  • length: 12 - This key needs to be of an exact length.

  • alias: new_name - Provides a name which will be used by templates using this key rather than the key_name. For example if you have two concepts of a version number, one is four zero padded because that is how the client wants it, and one is three zero padded because that how it is handled internally - in this case you really want both keys named "version" but this is not really possible since key names need to be unique. In this case you can create an alias. See one of the examples below for more information.

  • filter_by: alphanumeric - Only works for keys of type string. If this option is specified, only strings containing alphanumeric values (typically a-z, A-Z and 0-9 for ascii strings but may include other characters if your input data is unicode) will be considered valid values.

  • filter_by: alpha - Only works for keys of type string. If this option is specified, only strings containing alpha values (typically a-z, A-Z for ascii strings but may include other characters if your input data is unicode) will be considered valid values.

  • filter_by: '^[0-9]{4}_[a-z]{3}$' - Only works for keys of type string. You can define a regular expression as a validation mask. The above example would for example require the key to have four digits, then an underscore and finally three lower case letters.

  • format_spec: "04" - For keys of type int and sequence, this setting means that the int or sequence number will be zero or space padded. Specifying "04" like in the example will result in a four digit long zero padded number (e.g. 0003). Specifying "03" would result in three digit long zero padded number (e.g. 042), etc. Specifying "3" would result in three digit long space padded number (e.g. " 3"). For keys of type timestamp, the format_spec follows the strftime and strptime convention.

  • strict_matching: true - Only works for keys of type type int. This settings means that the field will only match numbers that have been properly formatted. For example, given "003" and strict_matching set to true, we would match "002", "12345" and "042", but not "00003" or "2". If you need the matching to be less strict, set strict_matching to false. The default behavior is to strictly match.

  • shotgun_entity_type - When used in conjunction with the shotgun_field_name option, will cause contexts to query shotgun directly for values. This allows using values from fields not seen in the folder structure to be used in file names.

  • shotgun_field_name - Only used in conjunction with shotgun_entity_type.

  • abstract - Denotes that the field is abstract. Abstract fields are used when a pattern is needed to describe a path - for example image sequences (%04d) or stereo (%V). Abstract fields require a default value.

  • subset and subset_format - Extracts a subset of the given input string and makes that the key value, allowing you to create for example an initials key from a full username or a key that holds the three first letters of every shot name.

For technical details about template keys, see the API reference.

Example - An alphanumeric name

A name that defaults to "comp" and that is alphanumeric:

name:
    type: str
    default: "comp"
    filter_by: alphanumeric

nuke_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/nuke/{name}.v{version}.nk

Example - Version number

A version number that would match numbers such as 002, 102, 034, 12341

version:
    type: int
    format_spec: "03"

A version number that would match numbers such as 002, 102, 034, 12341, but also 0002, 2 and 0102

version:
    type: int
    format_spec: "03"
    strict_matching: false

Example - A stereo eye

A typical stereo eye setup. The eye field is either L or R, but when used in software, it is often referred to in a generic, abstract fashion as %V. Since %V does not really refer to a file name but rather a collection of files, we set the abstract flag. Abstract fields need to have a default value that is pulled in whenever the abstract representation is being requested.

eye:
    type: str
    choices: ["L", "R", "%V"]
    default: "%V"
    abstract: true

nuke_shot_render_stereo: sequences/{Sequence}/{Shot}/{Step}/work/images/{Shot}_{name}_{eye}_v{version}.{SEQ}.exr

Example - Image sequences

Image sequences are abstract by definition and they have a default value set to %0Xd unless otherwise specified. The below sequence spec would identify frame numbers such as 0001, 1234 and 12345.

SEQ:
    type: sequence
    format_spec: "04"

nuke_shot_render_stereo: sequences/{Sequence}/{Shot}/{Step}/work/images/{Shot}_{name}_{channel}_{eye}_v{version}.{SEQ}.exr

Example - Two fields both named version via an alias

Two definitions of version number that can both be used by code that expects a key which is named "version". This is useful if you have two Toolkit apps that both need a version field but you want these version field to be formatted differently.

nuke_version:
    type: int
    format_spec: "03"
    alias: version
maya_version:
    type: int
    format_spec: "04"
    alias: version

# nuke versions are using numbers on the form 003, 004, 005
# the nuke publish app requires a field called {version}
# however {nuke_version} is a valid replacement for {version}
# because it has an alias defined
nuke_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/nuke/{name}.v{nuke_version}.nk

# maya versions are using numbers on the form 0004, 0005, 0006
maya_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/maya/{name}.v{maya_version}.ma

Example - Timestamp

A timestamp that defaults to the current local time and is formatted as YYYY-MM-DD-HH-MM-SS.

now:
    type: timestamp
    format_spec: "%Y-%m-%d-%H-%M-%S"
    default: now

A timestamp that defaults to the current utc time and is formatted as YYYY.MM.DD.

year_month_day:
    type: timestamp
    format_spec: "%Y.%m.%d"
    default: utc_now

A timestamp that defaults to 9:00:00 and is formatted as HH-MM-SS.

nine_am_time:
    type: timestamp
    format_spec: "%H-%M-%S"
    default: "09-00-00"

Example - Shotgun mappings

This is useful when you would like to to add Shotgun fields to a file name, for example. Let's say we would like to include the user name in a file name- we'd use the following definition:

current_user_name:
    type: str
    shotgun_entity_type: HumanUser
    shotgun_field_name: login

nuke_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/nuke/{current_user_name}_{name}.v{version}.nk

When a Toolkit app populates all the context fields (via the context.as_template_fields() method, it will populate the higher level fields Shot, Sequence and Step automatically. It will also scan through all fields which have shotgun_entity_type defined (like our current_user_name field above). If the Shotgun Entity is defined in the context, it will be able to automatically resolve the value. The current user is always tracked in the context, and in the above example, it would also be possible to pull data from fields on Shot, Sequence and Step since these are defined as part of the higher level path and therefore part of the context. However, trying to refer to an Asset entity in a field wouldn't work in the above example since Toolkit would have no way of knowing which asset in Shotgun to pull the data from.

Example - String field with two valid values

Often times a studio will have a project that needs to save out ASCII and Binary Maya files. In this scenario, a string value with two valid values looks like:

maya_file_extension:
    type: str
    choices: ["ma", "mb"]

Note: the default apps use either .ma or .mb based on what's configured in the templates.yml. So, for example, if you want to change the work files app to save .mb instead of .ma in a project, you can change these three templates (for Shots):

maya_shot_work: '@shot_root/work/maya/{name}.v{version}.ma'
maya_shot_snapshot: '@shot_root/work/maya/snapshots/{name}.v{version}.{timestamp}.ma'
maya_shot_publish: '@shot_root/publish/maya/{name}.v{version}.ma'

If you instead end them with .mb, then the apps will save out as Maya binary:

maya_shot_work: '@shot_root/work/maya/{name}.v{version}.mb'
maya_shot_snapshot: '@shot_root/work/maya/snapshots/{name}.v{version}.{timestamp}.mb'
maya_shot_publish: '@shot_root/publish/maya/{name}.v{version}.mb'

Check out The Paths Section below for more details.

Example - Disallowing a value

A string field for which the value "assets" is not allowed. This is useful if you for example have two a folder which contains folders for all the sequences for a project alongside with a single "assets" folder where all the assets are kept:

project
  |--- sequence1
  |--- sequence2
  |--- sequence3
  \--- assets

In order for Toolkit to correctly understand that the assets folder is not just another sequence, we can define that "assets" is not a valid value for the sequence template.

sequence:
    type: str
    exclusions: ["assets"]

The exclusions field above allows us to define two templates that both correctly resolves:

sequence_work_area: {sequence}/{shot}/work
asset_work_area: assets/{asset}/work

Example - Substrings

The following example extends a previous example and shows how to prefix filenames with a user's initials.

user_initials:
    type: str
    shotgun_entity_type: HumanUser
    shotgun_field_name: login
    subset: '([A-Z])[a-z]* ([A-Z])[a-z]*'
    subset_format: '{0}{1}'

nuke_shot_work: sequences/{Sequence}/{Shot}/{Step}/work/nuke/{user_initials}_{name}.v{version}.nk

The Paths Section

The Paths section specifies where work will be saved. All paths consist of at least a name and a definition, where the definition is a combination of key names in brackets interspersed with non-key values representing a path. For example, a definition for a shot work file might look like:

shot_work: sequences/{Sequence}/{Shot}/{Step}/work/{Shot}.v{version}.ma

With Sequence, Shot, Step and version being keys defined in the same template file.

Note: If a string key's name matches the entity type of a dynamic schema folder that has an associated Shotgun entity, then that folder name will be substituted in for the token. For example, if you are using a {Sequence} template key of type 'string' like the above snippet, and in your schema, you have a dynamic folder named "sequence", and in its corresponding sequence.yml file, it's defined to be of type shotgun_entity, and is connected to the "Sequence" entity type in Shotgun. Toolkit will recognize that your template key corresponds to this dynamic folder's entity type (in that they are both Sequence). So, Toolkit will take the resulting folder name (i.e., the name of the specific sequence in question), and substitutes that in for the template key.

This form is required if any optional attributes need to be defined. Currently, the only optional attribute is root_name, which can be used to specify a project root for a path in a project that has multiple roots. Multiple roots are used when you'd like to add a new storage root to store some of your project files.

root_name: name_of_project_root

For example, it may look like this:

shot_work:
  definition: sequences/{Sequence}/{Shot}/{Step}/work/{Shot}.v{version}.ma
  root_name: primary

You need to use the above format if you want to use another storage root than the primary one. In this example, using this simple format implies that you are using the primary root for all entries.

The Strings Section

Strings are similar to paths in that they must include a name and definition, which can be supplied in the simple form:

string_name: string_definition

String definitions are templates consisting of key names and other values which together resolve to a string rather than a file system path. An example might the name used in Shotgun for a publish:

maya_publish_sg_name: "Maya publish, {name}, v{version}"

With name and version as key names defined in the same file.

Using Optional Keys in Templates

Optional keys in templates are useful for a number of reasons. One common case is when {SEQ} is optional for rendered images. In this example, there can be a set of exrs that that are comprised of frame numbers, like /path/to/render/shot.101.exr (and 102, 103, etc), while you are also able to use the same template for quicktime movies, like /path/to/render/shot.qt. Another more common case is when you are rendering stereo images. If you are in a studio where the convention is: left eye: file.LFT.exr, right eye: file.RGT.exr, stereo image: file.exr?, you can make {eye} optional.

Optional sections can be defined using square brackets:

shot_work: sequences/{Shot}/work/{Shot}.[v{version}.]ma

The optional section must contain at least one key. If the path is resolved with no value for the key(s) in an optional section, the path will resolve as if that section did not exist in the definition. The example above can be thought of as two templates baked into a single definition:

shot_work: sequences/{Shot}/work/{Shot}.v{version}.ma
shot_work: sequences/{Shot}/work/{Shot}.ma

As you pass in a dictionary of fields, Toolkit will choose the right version of the template depending on the values:

>>> template = tk.templates["shot_work"]
>>> template.apply_fields({"Shot":"ABC_123", "version": 12}
/project/sequences/ABC_123/work/ABC_123.v12.ma
>>> template.apply_fields({"Shot":"ABC_123"}
/project/sequences/ABC_123/work/ABC_123.ma
Follow

25 Comments

  • 0
    Avatar
    Manne Öhrström

    Hello Asi! There is a core hook that let's you do exactly this. You can both process the data as it is coming in from Shotgun and validate it to make sure it works with your studio file and folder naming conventions. The default hook converts spaces to dashes, but there is room in that hook to implement pretty complex logic! Check out the core hook in github here: https://github.com/shotgunsoftware/tk-core/blob/master/hooks/process_folder_name.py

  • 0
    Avatar
    Asi Sudai

    Question about sanitizing entities name before we're creating folders using those names.

    could we have a way to sanitize those names for folder creation by replacing none [a-zA-Z0-9_] characters? 

    Something like:

    force folder name to be lowercase:

    lowercase: true

    # sanitize name re.sub( '\W', '_', name ):
    sanitize_name: true

     

    Which will do the following:

    entity_name   = "Dev 100_010" # or "My Unique Asset Name"

    folder_name =  re.sub( '\W', '_', entity_name )

    folder_name = folder_name.lower() 

     

    any thoughts on that?

  • 0
    Avatar
    Morgan McDermott

    Probably missing something obvious but when I edit the templates.yml file with the example where name defaults to "comp" I get this error:

    ERROR: A general error was reported: while scanning for the next token

    found character '\t' that cannot start any token

    in "Y:\settings\shotgun\stk100\config\core\templates.yml", line 42, column 1

    Should this be done somewhere other than the templates.yml file?

  • 0
    Avatar
    Morgan McDermott

    Ok, well this was user-error. Forgot that whole tab settings 'replace for spaces' option in Notepad++.

     

  • 0
    Avatar
    Dave Lajoie

    Hello Manne. Great document again. I have been reading it over and over, since I am creating a kick ass folder schema. however I hit a problem.

    Under Current User Folder (user_workspace), I think it would be great to have this support 'defer_creation: false'. currently, this is implicitly the folder directive user_workspace is always defer_create: true. I would need to bypass and have it behave since I want to enforce creating directories for current user or all users.This is useful for populating the path_cache.db if it gets corrupted.

    Also I would suggest having the ability to externally modify the folder yml directive file, with an env var or some sort of folder schema global.

    let say need a human_user directory in my schema, I would like to optionally have the filter switch from "current user" to "all users".

    Current user:

    type: "user_workspace"
    name: "login"
    defer_create: false

     

    All users:

    type: shotgun_entity
    entity_type: HumanUser
    name: login
    defer_creation: false
    defer_creation: []
    create_with_parent: true
    filters: [ { "path": "projects", "relation": "is", "values": [ "$project" ] } ]

    Is there a way to do all this?

  • 0
    Avatar
    Manne Öhrström

    Hello there Dave!

    Those are great feature requests! There is not currently a way to create workspace folders for all users, only the current one. Also, there is no way to include environment variable values in folder names. I will bring these suggestions back to our product team for triage and prioritisation!

    Thanks for the feedback! It all helps making toolkit a better and more fully featured product.

  • 0
    Avatar
    Benoit Leveau

    Hi Manne,

    I have a question about process_folder_name. I'm using it now to put the sg_asset_type to lowercase (Prop => prop). How does toolkit then knows that the folder prop is associated with Prop? I mean, when it reads a path of a published file, how does it find back where it's coming from?

    Cheers,

    Benoit

  • 0
    Avatar
    Manne Öhrström

    Hello Benoit!

    Good question! Prop in this case is an enum value in Shotgun (Asset.sg_asset_type), so it's basically just a string. There is no way to "link" to this asset type really, from a toolkit perspective it is just as if it was a string field where someone had filled in the value Prop. So for enums, there is no connection from the file system to shotgun. However, with Shotgun entities, we track the entity id inside the path cache so that it is possible to go both ways between shotgun and the file system! The path cache knows both the computed folder name and what entity it came from so this is what toolkit uses to do its lookups. In v0.15 of the core API, we are moving these entries from the path cache file into Shotgun (the Filesystem Location entity which can already been seen on the admin menu) so that the mapping will be very transparent. (We will then cache this shotgun table locally on disk for performance reasons, so we don't need to talk to shotgun all the time!)

  • 0
    Avatar
    Benoit Leveau

    It took me a bit but I think I now get it. If I try to extract the fields from a path, then toolkit will know that the token "sg_asset_type" in the template has the value "prop" in this case, and if I then use these extracted fields to construct another template everything will work, which is what all the apps (especially tk-multi-publish) are doing and why they are working as they never have to go from "prop" in the filesystem back to Shotgun's "Prop" enum value. So nothing is ever trying to link between the token/value (eg. sg_asset_type/prop) and the actual field in shotgun. If I was changing the folder name on disk the apps would happily read and use a "prop2" instead.

  • 0
    Avatar
    Manne Öhrström

    Yes exactly! Thanks so much for elaborating and clarifying! And apps sort of operate in "file system space" so they always use the template system, which is all about the file system. The role of the context is to handle the transition from "Shotgun space" to "file system space" so that you can get the right file system location for a Shot for example. We have been trying to keep the coupling between the file system and Shotgun as loose and simple as possible, without completely disconnecting the two!

  • 0
    Avatar
    David van Heeswijk

    Hi Manne,

    Is it also possible to generate a different folder structure per Task/Step?

    Example: Under lighting we only have folders for Maya and Houdini and under Comp we only have the bare minimum for Nuke Comps. So basically it only generates the folders that are necessary for the given task.  In the default config it creates the same structure for every step.

    Cheers,

     

    David

  • 0
    Avatar
    Manne Öhrström

    Hello David!

    Yes, that's totally possible! There is an example in the docs - check it out! That example shows how to do it with a step, but the same exact workflow applies to tasks too!

  • 0
    Avatar
    Louis Vottero

    maya_shot_work: '@shot_root/{Step}/wip/ann{Shot}_{Step}[_{name}].v{version}.ma'

    is there a way to format just {Shot} in this case?   Basically I need the number to be 100_0100 instead of 100-0100.  Our shot folders get named 100-0100, but I would like files to be named with 100_0100

  • 0
    Avatar
    Ryan Mayeda

    Hi Louis!

    Sorry for missing the question here, as I just answered it in the email ticket you sent in.  Just checking, do you actually want your Shot folders to be 100-0100 (i.e. with a '-') or do you want it to also be 100_0100 (i.e. with a '_')?  Based on the other ticket you sent in, I'm assuming you want them both to be '_'.  I pasted in my answer from the other ticket here, which goes through how to control the non-word character substitution character.  Let me know if you actually want folders as '-' but files as '_' and we can take it from there...

    The handling for non-word character substitution is in a core hook. Here is the spot in the code to dive into:

    https://github.com/shotgunsoftware/tk-core/blob/master/hooks/process_folder_name.py#L91

    Our default hook replaces non-word characters with a '-', which is why you're getting the behavior you have questions about. :)

    You can change it to a '_' here:

    https://github.com/shotgunsoftware/tk-core/blob/master/hooks/process_folder_name.py#L108

    if isinstance(src, unicode):
            # src is unicode so we don't need to convert!
            return exp.sub("_", src)  # CHANGE THIS FROM '-' TO '_'
        else:
            # assume utf-8 encoding so decode, replace
            # and re-encode the returned result
            u_src = src.decode("utf-8")
            return exp.sub("_", u_src).encode("utf-8")  # CHANGE THIS FROM '-' TO '_'
    

    Finally, here is our doc on how to override a core hook, which you'll need to do in order to get your customized behavior in place:

    https://support.shotgunsoftware.com/entries/95442818

    Hope this helps!

    Ryan.

  • 0
    Avatar
    Sebastian Thiel

    Hi there,

    Thanks for the great docs. A quick question though:

    Given a project named A * , and a primary root at *$PRIMARY all created directories and files for A will automatically go into $PRIMARY/A . However, the company I work for is used to having the option for configuring the top-level directory using a custom field on the A project entity, effectively specifying that files and folders should go to $PRIMARY/Z if Z is the value in the custom field.

    How would I achieve that with tank ?

     

    Thanks a bunch,

    Sebastian

  • 0
    Avatar
    Sebastian Thiel

    One possible solution would certainly be to use the process_folder_name or process_folder_creation hooks to intercept what would be created, and do a simple substitution on them. The earlier this happens, the better.

    But maybe such functionality is built in already, and I am missing something obvious here.

  • 0
    Avatar
    Sebastian Thiel

    It turns out that the top-level directory is determine by the link_field of the project entity, which is hardcoded to be tank_name (see https://github.com/shotgunsoftware/tk-core/blob/master/python/tank/folder/folder_types.py#L1687) .

    Is there a way to configure the link_field or name, similar to the implementation of the other entities ? Maybe this is not the forum for this, and I'd better file an issue/ask a question.

  • 0
    Avatar
    Sebastian Thiel

    I couldn't find an issue tracker, so I post it here for now.

    Following in Asi's footsteps, I just wanted to implemented a 'lowercase = 1' field. The idea is to use the 'project_folder_name' hook default, and lower case it if the field says so.

    However, it seems the 'sgtk.get_hook_baseclass()' doesn't work for core hooks. This might be due to the special case core does, looking in project folders first.

    It's surprising that tank encourages mixed-cases by making it hard to lower case names.

    Importing the core hook implementation the old-fashioned and standard python way doesn't work, as hooks are neither in the python path, nor in any valid package.

    The only options left are to copy-paste the code of the original implementation, or to somehow manually import the original implementation.  The latter seems like the more maintainable choice, so here is what I came up with:

    dn = os.path.dirname

    base_hook_file = os.path.join(dn(dn(dn(sys.modules[Hook.__module__].__file__))),

    'hooks',

    'process_folder_name.py')

    Base = loader.load_plugin(base_hook_file , Hook)

    It's clearly ugly.

    Now I realized that none of my additional flags, like lowercase: 1 made it into the hook. Apparently, those are only given to the process\_folder\_creation.py hook. The latter is a core hook as well, which will force me into the base implementation dance once again. However, it should be possible to alter the items dict in place, and see everything work just fine, after all.

    Sorry for the formatting, but I see no help about what this forum can do, nor is there a preview function.

  • 0
    Avatar
    Manne Öhrström

    Hello Sebastian!

    Thank you for your great questions! Let me try to explain how the project name works!

    The project name is a bit more special than say for example a Shot name - it's special in the sense that it has a particular field (Project.tank_name) where its name is stored. This is so that the project name can vary fairly independently from the the display name for the project which is shown inside Shotgun. 

    This tank_name field is set when you run the tank setup_project command. The command will ask you what you want your project folder to be called and it defaults to a all lower case version where spaces have been replaced with underscores (the code is here if you are interested!). But that formula is _not _setting the value, it is merely populating the suggested value that is presented to the user in the setup_project process. So you could change it to abc123 if you wanted. Because you have complete control over the field at project_setup time, we don't do any real processing at folder creation time. This is different from a shot for example, where we only have access to the display name by default and therefore have to transform that (via a hook) at folder creation time to ensure that the display name is turned into something file system friendly. Does that make sense? 

    Is controlling the name at project setup an option for you at all?

    Somewhat related, it is also worth mentioning that we have a special hook, allowing you to customize the suggested value that a user is presented as part of the setup_project process. This makes it possible for studios to implement complex project naming conventions, so that users can simply press enter and accept the default value when they are setting up projects. Read more about it here!

    Hope this answers your question!

    Manne

  • 0
    Avatar
    Sebastian Thiel

    Hi Manne,

    Thanks for the reply.

    Apparently it's easiest if I change code on our end to also consider the tank_name field of the project.

    I will use the community forums for some other questions that turned up.

    Best,

    Sebastian

  • 0
    Avatar
    Dave Lajoie

    Hello Manne,

    I was wondering if the optional keys can be grouped such it avoid the explosion of variations

    shot_work: sequences/{Shot}/[{token1}/][{token2}/]work/{Shot}[.{token1}][.{token2}].ma

     

    I want token1 ( directory and filename ) to be dealt as a single variation.

     

    Currently it will compute the following variations:

    sequences/{Shot}/work/{Shot}.ma

    sequences/{Shot}/{token1}/work/{Shot}.ma
    sequences/{Shot}/{token1}/{token2}/work/{Shot}.ma
    sequences/{Shot}/work/{Shot}.{token1}.ma
    sequences/{Shot}/work/{Shot}.{token1}.{token2}.ma
    sequences/{Shot}/{token1}/work/{Shot}.{token1}.ma
    sequences/{Shot}/{token1}/work/{Shot}.{token1}.{token2}.ma
    etc.

     

    What I was expecting is that token1 be grouped such that when it appears, it is present in both directory/filename

    Afaik nested optional tokens are not supported, and grouping directives are not supported as well

    Let me know if there are any workarounds for this one.

    Tx

    Dave.

  • 1
    Avatar
    Patrick Macdonald

    Just a quick question.

    Is it possible to create different templates on a per step basis? I know I can customise the folder structure, but I want to have a different max_shot_work template and max_shot_publish template for anim/lighting/asset/etc.

    This is specifically in the case of using the tk-multi-workfiles app 'save-as' dialog, which to my understanding doesn't provide hooks for reconfiguring the fields used with the above templates. (which would have been the other way to do things; eg to remove the fields I don't want on particular steps)

    Here are some example formats.

    lighting : PROJECT_SEQUENCE_SHOT_ARTIST_DESCRIPTION_VERSION.max

    animation : SEQUENCE_SHOT_ARTIST_DESCRIPTION_VERSION.max

    modelling  : CHARACTER_ARTIST_DESCRIPTION_VERSION.max

     

  • 0
    Avatar
    Ashley Retallack

    Hi is there an example of using create_file anywhere?

    I'd like to make a hidden  stub file at each entitiy location that contains the basic shotgun information for that entity.

    Cheers

     

    Ash

  • 0
    Avatar
    Diogo Girondi

    Newbie question for sure, but is there any reason for using the method you guys are using on the process_folder_name.py _replace_non_alphanumeric() function over something like lets say:

    from unicodedata import normalize

    text = "praça santo antônio"
    joinchar = "_"
    legal = normalize('NFKD', text.decode('utf-8')).encode('ASCII', 'ignore')
    print joinchar.join(legal.split())
    >> "praca_santo_antonio"

    Because getting "pra?_a_santo_ant?_nio" as a result doesn't look that pretty.

  • 0
    Avatar
    Darragh D

    It would be great to get further info with examples on this page regarding:

    associated_entity_type: Shot
    

    For more information about this advanced use case, you can always contact our support team.

    Rather than having to get in touch with support team.

Please sign in to leave a comment.