Project configuration

Managing your Project Configuration

This document explains best practices for how to safely roll out changes to your production pipeline. It explains how you can create a staging sandbox, which is a copy of your production configuration, update this sandbox and do testing and then finally push your changes to the production config.

Table of Contents:

Introduction

Basics of configuration management

      Cloning your Configuration

            Creating a Cloned Configuration

            Pushing changes from your staging sandbox to Primary

            Refreshing an old cloned configuration

            Deleting a cloned configuration

      Getting latest apps and engines

      Upgrading the Toolkit Core API

Managing the Project Lifecycle

      Inheriting the config from your previous project

      A studio configuration in git source control

            Setting up your studio config repository

            Creating a new project from git

            Making changes to a project

            Updating a project to have the latest version

            Advanced git usage: Branches

            Advanced git usage: Batch deployment

      A global configuration using includes

            Best practices when setting up a global config

            Managing a global configuration

Introduction

This document outlines how to manage your Toolkit configuration. Toolkit contains several utilities and tools that lets you safely manage your configuration and test upgrades and changes in a safe and sandboxed environment before rolling them out to the entire production.

In this document, we'll describe how to:

  • Safely upgrade the Toolkit Core API.
  • Upgrading your Apps and Engines.
  • Various ways to manage your config across multiple projects.
  • Go through Toolkit's Clone and Push functionality that allows you to safely test upgrades and changes without disrupting production.
  • How to work with git source control and Toolkit.

Basics of configuration management

Each Toolkit Project has one or more configurations associated with it. The configuration contains all the settings that the project needs: file system locations (templates), engines apps etc. In toolkit, each project is independently configurable. This means that different projects can live side by side safely: upgrading one project will not affect another.

If you are working with a large number of projects, this may become cumbersome and we offer several ways to make this process easy, safe and streamlined.

In Shotgun, each project has a number of Pipeline Configurations. When a project is first set up with Toolkit, a Pipeline Configuration called primary is created. The pipeline configuration entity in Shotgun points at a location on disk where the toolkit configuration can be found.

During the course of a project, you often need to make changes to the configuration. This can be tweaks to the configuration, or perhaps you need to add additional apps or engines. We also release new app versions frequently and we recommend that you use the latest versions if possible.

While it is possible to upgrade your primary project configuration straight away, this can be risky: Since this configuration is used by everybody on the project, introducing a problem will affect everyone. A better approach is to create an isolated version of the configuration that one a select group of people have access to. In this safe environment, upgrades, configuration changes and development can happen without affecting the rest of the production. Once the changes have been tested, they can be safely and confidently pushed to the primary configuration.

This process is called cloning and means that you make a personal copy of the primary configuration that only you (and other people you invite) have access to the clone. In here you can make changes safely and once you are happy you can push these changes back to the primary configuration.

Cloning your Configuration

Once you have set up toolkit and configured a project, the setup will look something like this:

There is a studio install which holds the Core API for all projects. This studio location also contains a tank command and a toolkit python API you can use to access any of your toolkit-enabled Shotgun projects.

In addition to this, there is a configuration folder for each project. This folder contains all the settings for that project. In addition to this, it also contains a tank command (and a Python API) which specifically operates on this configuration. When you are using this tank command or API code, you can strictly only operate on this configuration.

When a new project is set up, a Primary configuration is created. This is the configuration that toolkit will use by default for the project. In addition to the primary configuration, you can create additional configurations for a project. These can exist in parallel and are useful if you for example want to privately test some modifications, upgrade some apps or do development without impacting the entire team. Additional configurations are created by a process called cloning, a process where a configuration is copied to a new location.

Once you have cloned your configuration, your setup may look something like this:

In addition to the studio level tank command and your primary project configuration tank command, you now have a new pipeline configuration which has its own tank command. If you run this tank command, you will operate exclusively on the configuration located in the staging sandbox. So if you want to test out some new things in Maya, you can simply navigate to your cloned sandbox, run ./tank Shot xyz launch_maya and the Shotgun menu which appears in Maya will reflect the configuration inside of your staging sandbox rather than your Primary configuration.

Note that the studio level tank command always uses the Primary config, so the only way to access a cloned configuration is by navigating to its location and using the tank command that is located in that folder. In Shotgun, you can assign a specific group of users to a pipeline configuration entry, and any users that are associated with a configuration will new see menu entries appear in addition to those coming from the Primary config:

Creating a Cloned Configuration

In order to create a staging sandbox where you can safely test your upgrades, navigate to the pipeline configurations for your project in Shotgun and right click on the Primary configuration. An option to clone your configuration will appear:

Once you click it, you'll be presented with a dialog where you specify the path on disk where you want this new configuration to be located. If you are running a multi OS setup, make sure to put in paths for all relevant platforms.

When you press ok, toolkit will copy the configuration across and set up the clone. It will also associate your Shotgun user with this configuration, so as soon as you have created your clone, you will see new menu items appearing inside Shotgun.

Pushing changes from your staging sandbox to Primary

Once you have applied the relevant updates and run any testing that you deem is necessary, you can push back your changes into the production configuration by executing the tank push_configuration command. This will basically transfer all the changes you have made in your staging sandbox to your Primary configuration.

Please note that your current configuration is moved to a backup location when you run the push_configuration command. If you accidentally push or if there is a problem with the push, you can roll back simply by taking the content in the backup folder and copying into the config folder.

By default, this command will copy a collection of files into the config folder in the target pipeline configuration. If you are using unix and would like a more atomic update, you can add a --symlink flag to the push_configuration command. This will turn the config folder in the target pipeline configuration into a symbolic link which makes it easier to upgrade without running the risk of having configuration mismatches in currently running sessions.

Refreshing an old cloned configuration

If you have a old dev or staging sandbox set up, but it is old and you need to sync its contents with the latest production configuration, you do this by running the push_configuration command for the primary configuration:

tank push_configuration

Welcome to the Shotgun Pipeline Toolkit!
For documentation, see https://support.shotgunsoftware.com
Starting Toolkit for your current directory '/tmp'
- Using configuration 'Primary' and Core v0.14.52
- Setting the Context to Empty Context.
- Running command push_configuration...

----------------------------------------------------------------------
Command: Push configuration
----------------------------------------------------------------------

This command will push the configuration in the current pipeline configuration
('Primary') to another pipeline configuration in the project.

Your existing configuration will be backed up.

The following pipeline configurations are available to push to:
 - [324] Staging Sandbox (/my/staging/sandbox)

Please type in the id of the configuration to push to (ENTER to exit): 324
Hold on, pushing config...
Checking if there are any apps that need downloading...
Push Complete!

Your old configuration has been backed up into the following folder:
/my/staging/sandbox/config.bak.20140108_093218

Note how we are pushing from the primary project config to the staging sandbox. We do this by running the primary configuration's tank command. If you have multiple sandboxes set up, it is also possible to push data between those.

Deleting a cloned configuration

If you want to delete a cloned configuration, simply delete the entry in Shotgun and remove the folder from disk.

Getting latest apps and engines

Inside your staging sandbox (or in any other config), you can run the tank updates command in order to check if there are any app updates available. This command has got rudimentary filters that you can use if you only want to check certain areas of your configuration:

----------------------------------------------------------------------
Command: Updates
----------------------------------------------------------------------

This command will go through your current configuration and check if there are
any updates available. If there are updates, you will be asked if you want to
perform an upgrade. If settings has been added to the new version that you are
installing, you may be prompted to specified values for these.

Running this command with no parameters will check all environments, engines
and app. This may take a long time. You can also run the updater on a subset
of your installed apps and engines.

General syntax:
> tank updates [environment_name] [engine_name] [app_name]

The special keyword ALL can be used to denote all items in a category.

Examples:

Check everything:
> tank updates

Check the Shot environment:
> tank updates Shot

Check all maya apps in all environments:
> tank updates ALL tk-maya

Check all maya apps in the Shot environment:
> tank updates Shot tk-maya

Make sure the loader app is up to date everywhere:
> tank updates ALL ALL tk-multi-loader

Make sure the loader app is up to date in maya:
> tank updates ALL tk-maya tk-multi-loader

Upgrading the Toolkit Core API

This section explains how you can use a clone staging sandbox configuration to safely upgrade the Toolkit Core API. If you haven't got a staging sandbox prepared yet, just follow the instructions in the previous section!

If your staging sandbox was cloned from a pipeline configuration using a shared studio Core API, you'll want to update your sandbox to use it's own unique Core API code. This is called "localizing" the core and can be done by navigating to your staging sandbox and running tank localize. This command will copy the Core API from the studio install, into your sandbox, making it possible to run and test a different version of the Core API later on.

The default behavior in Toolkit is to localize the core by default. If you haven't explicitly created a shared studio core previously, it's safe to assume your core is localized already.

cd /my/staging/sandbox
./tank localize

Welcome to the Shotgun Pipeline Toolkit!
For documentation, see https://support.shotgunsoftware.com
Starting Toolkit for your current directory '/my/staging/sandbox'
- Using configuration 'Staging Sandbox' and Core v0.14.52
- Setting the Context to Empty Context.
- Running command localize...

----------------------------------------------------------------------
Command: Localize
----------------------------------------------------------------------

This will copy the Core API in /mnt/software/shotgun/studio into the Pipeline
configuration /my/staging/sandbox.

Do you want to proceed [yn]y

Localizing Core: /mnt/software/shotgun/studio/install/core -> /my/staging/sandbox/install/core
Copying Core Configuration Files...
Localizing Apps: /mnt/software/shotgun/studio/install/apps -> /my/staging/sandbox/install/apps
Localizing Engines: /mnt/software/shotgun/studio/install/engines -> /my/staging/sandbox/install/engines
Localizing Frameworks: /mnt/software/shotgun/studio/install/frameworks -> /my/staging/sandbox/install/frameworks
The Core API was successfully localized.

Localize complete! This pipeline configuration now has an independent API. 
If you upgrade the API for this configuration (using the 'tank core' command), 
no other configurations or projects will be affected.

Now we are no longer sharing the Core API with the studio location but are running our own, independent version. We can now go ahead and perform a standard Core API upgrade, again using our local tank command:

cd /my/staging/sandbox
./tank core

Toolkit will check if there is a new version available and offer you to download and install it.

Once you have updated the Core API, make sure to test the installation. Launch some apps, either using the tank command in the sandbox or using the special menu entries in Shotgun. Do a basic run-through of your pipeline and perform the tests you deem necessary.

Finally, once you are happy, it is time to go ahead and update the studio version of the Core API. Note that in the typical toolkit setup, the Core API is shared between all projects, so by running the tank core command from your studio location tank command, you are updating the Core API for all projects.

Managing the Project Lifecycle

Each toolkit project contains an independent configuration which holds all the settings for that project. If you are managing a lot of projects, this means that you need to manage a lot of configurations. For this reason, Toolkit contains a number of different tools and ways to handle configurations, both for individual projects and for the studio life cycle and configuration evolution. This section outlines some of those best practices and shows practical steps how to set up your config.

Depending on the needs of your studio, different levels of complexity may be relevant. Toolkit offers three different approaches and we'll explain each one of them in detail:

  • The most straight forward approach is to basically copy the config from the previous project when you set up a new project. This is good if you are a small studio and don't have a massive amount of projects.
  • If you have a higher project turnover and if you run more than one project in parallel, the next level of integration that we recommend involves git version control. Toolkit has got native support for git and once you are up and running with a git based workflow you have a single configuration for your studio and are tracking all the changes you are making to that configuration over time. Each project can safely pull in configuration changes as and when they need to.
  • If you are running a large scale facility, it may be worth considering a setup where a single configuration is directly connected to all the different currently running projects in the studio. A single change to this configuration will have an immediate effect on all the projects.

In the following sections we'll describe the different approaches in detail.

Inheriting the config from your previous project

This is the simplest approach to configuration management. For the very first time you set up a project with Toolkit, you download our default configuration. You then set up that first project to match your pipeline settings, file system locations etc. When the project is running, you run commands such as tank updates to update the app versions you are running.

When your second project comes around, you don't want to start with the default config again - instead, during the project setup phase, pick the config from your first project as the starter for your new project. This means that all the changes, tweaks and improvements you did for the first project will be taken advantage of in the second project as well. For the third project, use the second project's config etc.

This is a very simple way to gradually evolve the configuration over time. Changes and improvements will flow from project to project in an ad hoc fashion. The first time you run the setup_project command, just hit enter when the setup process prompts for the configuration to use. This will download and install the default configuration.

For your second project, you will get presented with a list of paths to configurations for previous projects. Choose one of this paths and enter that when the setup process prompts for a config. This will copy that configuration to the new project:

Welcome to the Shotgun Pipeline Toolkit!
For documentation, see https://support.shotgunsoftware.com
- Running command setup_project...

----------------------------------------------------------------------
Command: Setup project
----------------------------------------------------------------------

Welcome to the Shotgun Pipeline Toolkit Project Setup!

Connecting to Shotgun...
Connecting to the App Store...

------------------------------------------------------------------
Which configuration would you like to associate with this project?

You can use the configuration from an existing project as a template for this
new project. All settings, apps and folder configuration settings will be
copied over to your new project. The following configurations were found:

   My very first project: '/mnt/software/shotgun/first_project/config'

If you want to use any of the configs listed about for your new project, just
type in its path when prompted below.

You can use the Default Configuration for your new project.  The default
configuration is a good sample config, demonstrating a typical basic setup of
the Shotgun Pipeline Toolkit using the latest apps and engines. This will be
used by default if you just hit enter below.

If you have a configuration stored somewhere on disk, you can enter the path
to this config and it will be used for the new project.

You can also enter an url pointing to a git repository. Toolkit will then
clone this repository and base the config on its content.

[tk-config-default]: /mnt/software/shotgun/first_project/config

A studio configuration in git source control

Limitations with the first approach include the fact that the projects are not connected to each other. If you have 10 projects and you all need to update them because a critical bug fix has been released, you would have to manually go through each project and run the tank updates command.

One way to resolve this is to create a master configuration and store it in git source control. Whenever you create a new project, simply type in the path to this git repository in the setup project dialog and toolkit will clone it for you. Now all the projects are connected to the same "studio master" config. If you have made some good changes to a project configuration, you can commit them and push them to the studio master. Other projects can then easily pull these down. You also retain a history of all your changes via git.

What is Git?

Git is a so called source control system that software engineers and others use to carefully track changes to code and to be able to collaborate in large project environments. You can read about it here: http://git-scm.com. Several 'development community' sites exist, all based around git. The idea with these sites is to make sharing and collaboration around code easy and open. https://github.com is one of the larger such sites and you can find all the toolkit code managed here under https://github.com/shotgunsoftware.

Git is a very powerful tool but can be hard to understand if you are not already familiar with it. The rest of the git documentation here assumes that you know some basics about source control and git.

The basic idea is that you set up a git repository which holds the git configuration. Whenever you run tank setup_project, you specify the git url to this repository (for example username@someserver.com:/studio_config.git) and the setup process will clone the repository so that the new project basically becomes a repository connected to the main studio repository. Once they are connected you can push and pull changes, and work in branches for finer granularity.

Setting up your studio config repository

Before you do anything else, you need to create a studio config repository. This section shows how to take an existing toolkit configuration and creating a git repository from that.

First, you need to go to your git server and create a repository. This process may be different, depending on how you run things. If you are using something like github, you would start a web browser and navigate to github. If you have access to the server you may do something like git init --bare. In our example, we assume that the git repository you create is called username@someserver.com:/studio_config.git.

Now move the config folder of the project you want to use to seed your repo with into a config.bak location:

cd /project_configs/studio_config
mv config config.bak

Clone your initialized git repository into the config location of your project that you want to base the studio config on. Once you have run the clone command, you will have an empty config folder which is also a git repository:

cd /project_configs/studio_config
git clone username@someserver.com:/studio_config.git config

Copy all the files from your config.bak location back into the config folder. Once done, you can delete the empty config.bak folder. Your config files are now inside the git repository and we need to add them, commit them and push them to the server. But before doing that, we need to do some house keeping to handle some toolkit system files correctly. In the config folder, create a .gitignore file and add the following lines to it:

install_location.yml
pipeline_configuration.yml

Now git will ignore those two system files. We can go ahead and add the rest of the files in the configuration, commit them and push them to the studio repository.

git add --all
git commit -am "initial commit of our studio config!"
git push

Creating a new project from git

When you create a new project, simply specify a valid git url when the setup process prompts you to enter the path to the configuration to use. Following our example above, we would enter username@someserver.com:/studio_config.git. As part of the project setup process, Toolkit will clone this repository into the config folder of your new project configuration. This means that you can later on go into this config folder and run git commands. Note that any cloned pipeline configurations will also clone the git repository and will work seamlessly.

Making changes to a project

Whenever you have made changes to your primary config, you can simply go to your config folder and then run git diff, git commit and git push. This would push your changes to the studio configuration that you cloned your project config from.

Updating a project to have the latest version

Alternatively, if you have updated your studio level config with some changes and you want to pull those down to your project, just go to your config folder and run a git pull. Important: Note that once you have done this, make sure you run a tank cache_apps to ensure that all the app versions that your changed config requires are present in the system!

Advanced git usage: Branches

The default behavior in toolkit is a simple clone operation when a new project is created. The project will be using the git master branch. For a more advanced setup, if you for example want to keep certain configuration changes for a project private and don't share them with other projects, you can set up a git branch after the project has been created. You then commit to this branch. The changes you want to push to the studio configuration can be merged with the local master branch for the project and then pushed.

Advanced git usage: Batch deployment

since toolkit keeps a list of all the different configurations for a shotgun site via its pipeline configuration entity, it is possible to iterate over this list and execute a git pull command for each of the items, effectively running a batch update on all projects. This functionality is not currently part of Toolkit. If you need assistance, don't hesitate to reach out to our technical support for more information.

A global configuration using includes

The git based approach above handles independent project configurations which are connected via git: Updates are not automatically reflected across projects but will have to be pulled and pushed.

For a fully centralized configuration, where the configuration truly resides in one place and where a single change immediately reflects a group of projects, you will need to make use of the @include functionality in the Toolkit configuration. This makes it possible to create references so that each project configuration points at a central location where the actual configuration is being kept.

The @include syntax allows you to chain together multiple files. For example, if you have a file /tmp/stuff.yml, which contains the following content:

# paths to maya
maya_windows: 'C:\Program Files\Autodesk\Maya2012\bin\maya.exe'
maya_mac: '/Applications/Autodesk/maya2012/Maya.app'
maya_linux: 'maya'

# a configured toolkit app
file_manager:
  allow_task_creation: true
  file_extensions: []
  hook_copy_file: default
  hook_filter_publishes: default
  hook_filter_work_files: default
  hook_scene_operation: default
  launch_at_startup: false
  launch_change_work_area_at_startup: false
  location: {name: tk-multi-workfiles, type: app_store, version: v0.4.8}
  saveas_default_name: scene
  saveas_prefer_version_up: false
  sg_entity_type_filters: {}
  sg_entity_types: [Shot, Asset]
  task_extra_display_fields: []
  template_publish: null
  template_publish_area: null
  template_work: null
  template_work_area: null

As you can see above, you can create include definitions at several different levels - in the case above, we have an app definition and three strings values. These can then be referenced from an environment file:

includes: ['/tmp/stuff.yml']

engines:

  tk-maya:

    # First all our app definitions
    apps: 
        # normally, we would have the entire set of configuration parameters at this point.
        # because we are using an include, we can reference an entire sub-section of configuration
        # using the @ keyword:  
        tk-multi-workfiles: '@file_manager'

        # alternatively, for simple values, we can use them as parameter values for apps:        
        tk-maya-launcher:
          mac_path: '@maya_mac'
          linux_path: '@maya_linux'
          windows_path: '@maya_windows'
          location: {name: tk-multi-launchapp, type: app_store, version: v0.2.19}

    # and after the apps section follows specific configuration options for the engine
    location: {name: tk-maya, type: app_store, version: v0.4.1}
    menu_favourites:
    - {app_instance: tk-multi-workfiles, name: Shotgun File Manager...}
    template_project: null
    use_sgtk_as_menu_name: false

Furthermore, you can read in several include files, one after the other. If the same include definition exists in two different files, the most recently read file will take precedence. We could extend our example environment above:

includes: 

    # first include a global config file which contains all the studio level app definitions
    - '/studio/configurations/studio_apps.yml'

    # now we can include an include containing overrides, for example based on project type.
    # if the global config contains an entry named file_manager and the vfx_apps.yml contains
    # the same, the vfx_apps definition will be used.
    - '/studio/configurations/vfx_apps.yml'

engines:

  tk-maya:    
    apps: 
        tk-multi-workfiles: '@file_manager'

    location: {name: tk-maya, type: app_store, version: v0.4.1}
    use_sgtk_as_menu_name: false

With the approach just shown, it is possible to have a set of studio defaults which can be overridden by project type defaults which in turn can be overridden by specific project settings. You can either do it on the app level, as shown in the example above, or an engine level, as shown in the next section.

Best practices when setting up a global config

There are several ways to set up a global configuration. Our recommended best practices approach for setting this up breaks the configuration down on a per engine basis. Each environment file is completely empty and references engines (and apps) defined in separate files. This makes it easy to tweak and reconfigure things - one engine at a time.

Each of these include files are in a standard form, named after the engine. For example, if you have a Maya engine, an include file would contain just the engine and its apps. It's top level entry would simply be named maya:

maya:
    apps:
      tk-maya-breakdown:
        hook_multi_update: default
        hook_scan_scene: default
        location: {name: tk-maya-breakdown, type: app_store, version: v0.2.13}
      tk-multi-loader2:
        action_mappings:
          Maya Scene: [reference, import]
          Photoshop Image: [texture_node]
          Rendered Image: [texture_node]
        actions_hook: '{self}/tk-maya_actions.py'
        entities:
        - caption: Assets
          entity_type: Asset
          filters:
          - [project, is, '{context.project}']
          hierarchy: [sg_asset_type, code]
        filter_publishes_hook: '{self}/filter_publishes.py'
        location: {name: tk-multi-loader2, type: app_store, version: v1.0.6}
        menu_name: Load
        publish_filters: []
        title_name: Loader
      [... additional app entries here ...]

    debug_logging: true
    location: {name: tk-maya, type: app_store, version: v0.3.10}
    menu_favourites:
    - {app_instance: tk-multi-workfiles, name: Shotgun File Manager...}
    - {app_instance: tk-multi-snapshot, name: Snapshot...}
    - {app_instance: tk-multi-workfiles, name: Shotgun Save As...}
    - {app_instance: tk-multi-publish, name: Publish...}
    template_project: shot_work_area_maya
    use_sgtk_as_menu_name: false

# each include file has a framworks section containing all the frameworks needed for
# all apps in the file
frameworks:
  tk-framework-shotgunutils_v1.x.x:
    location: {name: tk-framework-shotgunutils, type: app_store, version: v1.0.8}

In your studio, you most likely don't have a single maya setup, but may have a number of different ones for different departments and types of things. We recommend that you set up a maya include file for each of these ones, organized in a file hierarchy. Each of these files has a top level maya entry just like the file above.

Now each project will contain a number of environments. Each of these environment files will be a list of engine includes, linking that specific environment to a collection of engine and app setups. You can set up one (or several) default project configurations for your studio, all containing includes like this, thereby ensuring that the actual app and engine payload is completely contained within the include files and therefore global. If you make a change to your global include files, all projects will be affected. With this setup, your environment files will then be on the following form:

includes:     
    - '/studio/configurations/maya/asset.yml'
    - '/studio/configurations/nuke/asset.yml'
    - '/studio/configurations/houdini/generic.yml'

engines:
  tk-maya: '@maya'
  tk-nuke: '@nuke'
  tk-houdini: '@houdini'

# we don't need any frameworks here because there are no apps or engines defined
frameworks: null

If you wanted to break out of the above setup and start defining some project specific entries, you would simply replace @maya with a series of app and engine definitions in the environment file itself.

Managing a global configuration

Managing a global configuration is more involved than managing a normal one. Because you have effectively combined a number of projects into a single file structure, running the tank updates command from any project and choosing to update an app version will affect all other projects, so proceed with some caution here.

Furthermore, the standard clone workflow won't work out of the box, since what you are cloning is the project configuration, which now only contains includes.

For safe testing and maintenance, we recommend storing the global configuration in source control (e.g. git) and do updates in a separate test area with a special test project. Once the quality control passes, commit the changes and pull them down into the actual global configuration.

Big Buck Bunny - footage courtesy of (CC) Blender Foundation, www.blender.org

Follow

33 Comments

  • 0
    Avatar
    Toke Jepsen

    Just to confirm, if I wanted to make an existing configuration into a studio config, I only need to make the config folder into a git repo?

  • 0
    Avatar
    Manne Öhrström

    Hey Toke!

    Great question - I think that is just about possible but I am not sure that's what I would recommend! I think a better workflow may be to create a bare repo on your git server, then use this as your "studio config". You then use this git repo when you run setup_project, toolkit will automatically clone it and your project configuration will automatically be a git repo. You can then send changes back to the studio config easily by pushing. Likewise you can fetch any changes that have happened in master by doing a git pull.

    We have been getting a bunch of really good feedback on the git related workflows so I will update the docs shortly based on this! Hopefully once the updates are out things should be clearer! Thanks for contributing to these improvements!! :)

  • 0
    Avatar
    Toke Jepsen

    the problem is that we have been developing our configuration on a project, so if I setup a project with the bare git repo I would lose the connection to our current projects config and not be able to push any configuration?

  • 0
    Avatar
    Manne Öhrström

    Could you do something like this?

    Create bare repo (e.g.  username@someserver.com:/studio_config.git )

    Move the config folder of that studio config project to a config.bak 

    cd /project_configs/studio_config
    mv config config.bak

    Clone repo into config location

    cd /project_configs/studio_config
    git clone username@someserver.com:/studio_config.git config

    Now copy all config files back

    cd /project_configs/studio_config/config.bak
    cp -r * /project_configs/studio_config/config

    Finally, add all the config files to git, commit and push!

    When you want to update the studio template, just commit and push. When you want to create a new project, just specify git bare repo path in setup project.

    Do you think that would work? Or am I missing some important aspect of your workflow?

     

  • 0
    Avatar
    Toke Jepsen

    Cheers Manne! That should work.

  • 0
    Avatar
    Genaro Rocha

    Hello. I am trying to make the setup_project command clone an existing repository on bitbucket that contains code for an existing configuration. However, when the setup_project command prompts me to put in the path the path to the configuration, it says its invalid if I use the repository's url as location. For example:

    If you have a configuration stored somewhere on disk, you can just enter the

    path to this config it will be used for the new project.

    [tk-config-default]: https://genarorg@bitbucket.org/boxelinc/tk-config-default-boxel.git

    ERROR: File path https://genarorg@bitbucket.org/boxelinc/tk-config-default-

    boxel.git does not exist on disk!

    Is there a specific way in which this url needs to be typed in so that the script clones the repository as mentioned in the documentation?

    Thanks!

     

  • 0
    Avatar
    Genaro Rocha

    Never mind, I had an old version of the Core API. Updating introduced this functionality. Thank you anywas

  • 0
    Avatar
    Max Persson

    Hi,

    Is it possible to switch an existing project over to getting it's configuration from git? Is it just a matter of making a git checkout and renaming the folder to the current config folder name along with keeping the old install_location.yml and pipeline_configuration.yml?

    Anything more to think about?

    Cheers,

    Max Persson

  • 0
    Avatar
    KP

    Hey Max,

    Assuming you want to convert your existing config to be managed in a new git repo,  did you see Manne's reply from Jan 21? Seems like that might cover what you're trying to do. How does that look?

  • 0
    Avatar
    Max Persson

    That is basically what I've already done, created tk-config-mfx on Github containing the config that all current projects use. My question is what needs to be done to switch the current projects over to use this shared git version of the config and not their own, even if they are the same? In the future I want to be able to pull from this config in each project instead of the rsync from a special "master config" project we have now. In detail what I wonder is which files are unique for each project and can't be pulled from git, hence my question about keeping "install_location.yml" and "pipeline_configuration.yml". Are these the only project unique files?

  • 0
    Avatar
    KP

    Hey Max,

    Sorry for the late reply but the answer is basically yes, config/core/install_location.yml and config/core/pipeline_configuration.yml are the config-specific files. You should add those to a .gitignore file when initially setting up your git repo.

    Then to convert an existing project to use the git repo:

    Backup your existing config:

    cd /projects/shotgun/your_project

    mv config config.bak

    Checkout your studio config:

    cd /projects/shotgun/your_project

    git clone git@github.com:mfx-cgi/tk-config-mfx.git  config

    Copy your  config/core/install_location.yml and config/core/pipeline_configuration.yml files back in: 

    cd /projects/shotgun/your_project

    cp config.bak/core/install_location.yml config/core/

    cp config.bak/core/pipeline_configuration.yml config/core/

    And obviously, if you've made any other changes to your project config that differ from the studio config you cloned from the repo, you'd need to re-apply them.

    You may want to be safe and ensure your apps are all up to date for the config you cloned (in case there are differences you weren't aware of):

    cd /projects/shotgun/your_project/config

    tank cache_apps

    That should be it!

    FYI, I've updated the docs above to make things a little clearer on this whole process. Let us know if you have any more questions! 

     

  • 0
    Avatar
    Max Persson

    Thanks for the answer. That is basically what I ended up doing on my own, just wanted a confirmation of the ignored files. Good that it made it to the docs!

  • 0
    Avatar
    Philip Scadding

    Hi 

    Im trying to make my project managed by a global configuration. I'm currently not getting the hooks to be globally managed.

    in your code above I see it has: 

    filter_publishes_hook: '{self}/filter_publishes.py'

    What is "{self}" in this context and how should I go about using my template projects hooks? is it a case of specifying the hook location?

     

    Thanks

    Phil

  • 0
    Avatar
    Manne Öhrström

    Hello!

    You can read more about the hooks syntax here: https://support.shotgunsoftware.com/entries/95442678

    Basically, the** {self}** keyword refers to the hooks that are distributed with each app.

    If you want to put hooks in a global location on disk and make sure all projects reference these, you could set up an environment variable GLOBAL_HOOK_PATH, containing the path to your global hooks folder, and the configure all hooks to be on the form {$GLOBAL_HOOK_PATH}/my_hook.py. Do you think that would work?

    Thanks!

    Manne

  • 0
    Avatar
    Addison Bath

    I have completed transition of our sgtk setup to a global configuration and have everything for the env configs working smoothly.  I have a few questions remaining:

    1.  How should the template inheritance work?  Your documentation mentions this is possible, but I could not find any example.  I tried implementing it the same as the env yml config files, but could not successfully get it working.  I included a studio level templates file, then use a @studiokeys.   I receive the following error:  ValueError: dictionary update sequence element #0 has length 1; 2 is required

     

    1.  It seems I cannot use env variables inside the yaml include tags like I can in the location path tags.  {$ENV_VAR} does not seem to resolve correctly inside include tag.  This sadly forces me to hard code all the include paths.

     

    Please let me know if you need any additional clarification.

     

  • 0
    Avatar
    Manne Öhrström

    Hello!

    Template inheritance works like the environment inheritance, with the exception that the included file needs to have the three distinct sections keys, paths and strings. Essentially, it follows the same resolution mechanism as the environment includes except it is collecting data in the three sections keys/paths/string sections in parallel as it traverses files. After all things have been loaded in, the total data structure is validated.

    Essentially, it runs the following algorithm:

    1. Set up three empty "buckets" for keys, strings and paths.
    2. For each included file in top-down order, add their respective keys, strings and paths to the buckets. This is done recursively allowing you to have file inside files.
    3. Add the keys, strings and paths from the main templates.yml file.
    4. Validate and ensure that all templates exists etc.

    Here's a quick example. Here is the main templates.yml file: 

    include: ./global_defs.yml
    keys:
    name:
    type: str
    filter_by: alphanumeric
    version:
    type: int
    format_spec: "03"
    paths:
    maya_shot_work: '@shot_root/work/maya/{name}.v{version}.ma'

    And here is the contents of the included file global_defs.yml:

    keys:
    Sequence:
    type: str
    Shot:
    type: str
    Step:
    type: str
    paths:
    shot_root: sequences/{Sequence}/{Shot}/{Step}

    It is recommended good practice (but may not always be appropriate) to keep the keys and the paths together, so that each file contains all the pieces it needs. We don't seem to have a lot of documentation about this so I will add a section to the documentation explaining this! 

    As for the second question, we don't currently support environment variable syntax for the includes. This sounds easy to add. I'll create a ticket on our end and will try to get it into the next core release!

  • 0
    Avatar
    Manne Öhrström

    Hello Addison!

    I have just released Core v0.14.56 which includes support for environment variables in includes. Read more about it here. Here's an extract from the docs: 

    files that are not recognized are skipped, so you can put paths for

    # windows and linux next to each other for multi platform support:
    includes: ['/foo/bar/hello.yml', 'z:\foo\bar\hello.yml']
    # you can use environment variables inside of full paths too
    includes: ['$STUDIO_ROOT/foo/bar/hello.yml', '%STUDIO_ROOT%\foo\bar\hello.yml']

  • 0
    Avatar
    Addison Bath

    Great Manne!

    I will let you know if I hit any further problems. This should give me everything I need to maintain a simple, flexible hierarchy of configuration.

  • 0
    Avatar
    Oscar Juarez

    Hello shotgunners,

    A couple of post above you say that "config/core/install_location.yml" and "config/core/pipeline_configuration.yml" are the only files config specific but apparently "tank_configs.yaml" is also config dependent. Should I also exclude it from my Git repository or that is a bad idea. 

  • 0
    Avatar
    Manne Öhrström

    Hello Oscar,

    Yes you are quite right - that tank_configs.yml file is also a system file which is changed by the system (although it typically resides in the project location rather than in the configuration folder). If you are storing your config in source control, it is probably a good idea to exclude this file!

    Thanks!

    Manne

  • 0
    Avatar
    Oscar Juarez

    Hello Manne,

    Thanks for the answer, in our case it is inside the config folder, I guess because we are using the old structure for tank, where the tank folder is inside the project folder?

  • 0
    Avatar
    Fran Zandonella

    Hi Manne,

      Is there any way to update to a specific Core API version that is not the latest?  While the latest is the greatest, I want to update to v0.14.67 so I can test those changes before doing the testing needed for v0.14.69.

      I don't see any examples like that in the docs.

  • 0
    Avatar
    Daniel Goertz

    Hi Manne,

    I got a problem (see image) during "tank setup_project" for a new project.

    The GitHub-Repo which should be clone into the right place in the new project,

    is setup as in documentation below. So I have no idea how to fix this problem?!

    Thx.

  • 0
    Avatar
    Marthinus Wessels

    Hi,

    So I followed the guide, I cloned my primary pipeline configuration in Shotgun, waited for it to tell me that it was completed. Did some app dev tinkering in my clone version and when I want to push the configuration it tells me that I only have one pipeline configuration for the project:

    ERROR: Only one pipeline configuration for this project! Need at least two configurations in order to push. Please start by cloning a pipeline configuration inside of Shotgun.

    Any ideas why this is happening? I have to mention, in the past I did close the shotgun pipeline config window before the cloning process was completed. This creates the clone on disk but skips the entry in shotgun which I used to create manually until I discovered that I need to wait for the prompt that the clone process is complete. I don't know if this left some scars that is messing with the push now?

  • 0
    Avatar
    Manne Öhrström

    Hi!

    Yes, this would be happening because there isn't a corresponding pipeline config entry in Shotgun. Try creating one manually and hopefully the clone command should start working!

  • 0
    Avatar
    Marthinus Wessels

    Thanks Manne. In the end I levelled everything and started over. Push_configuration now works!! But, for some bizarre reason it doesn't push the "interpreter_Windows.cfg" or any of the other OS.cfg files in the config/core folder. I then copy them over manually from the clone. Any ideas why?

  • 0
    Avatar
    Manne Öhrström

    That's actually a classic "a feature, not a bug!" but it may seem a little odd. Traditionally, most toolkit configs have been using a shared core, where the toolkit core and all of the apps, engines etc are in a single location on disk and shared by all projects. This shared core location looks similar to a config structure, but with a different set of config files, including interpreter_xxx.cfg. If you localize your config, you copy all those files into your project config.

    We created the localize feature originally so that people could try out a new version of the core API in an isolated fashion before upgrading lots of projects. The workflow would be that you'd clone a config, then localized it to detach it from the shared location. Now that you had a fully independent cloned config, you could upgrade the core API and run tests etc - disconnected from the rest of the pipeline.

    Now, we localize things by default in the desktop, so a lot more configs are localized. This is one of the reasons it takes so long to clone them - configs with a shared core on the other hand are super quick to clone!

    So to answer your question - the interpreter_xxx.cfg config files belong to the "core" portion of the config and not the traditional config - therefore, it's not copied over when you push your config. This made a lot more sense when most configs were shared - now it can appear strange why it isn't happening! Hope the explanation makes sense!

    Thanks!

    Manne

  • 0
    Avatar
    Marthinus Wessels

    I appreciate the explanation, it makes sense. Since my project wizard got stuck on 9% while downloading the softimageqt framework I had to force close Shotgun Desktop. I'm guessing the wizard never got to the localizing part and all of the apps etc. folders had a "placeholder" file in them. I just ran "tank localize" to get everything out of %AppData% onto the server location.

    To be sure, it is normal for the interpreter_windows.cfg file to be deleted when you push_configuration from the clone to the primary every time? In that case all good, it's easy enough to quickly copy the files over manually.

  • 0
    Avatar
    Seth Lippman

    Would it be possible to use environment variables in the icon path? I'm currently implementing a global configuration and from what I can tell, only absolute paths and path relative to the config can be used. That might be OK if the path was relative to the yaml file that was referencing the icon, but it in fact seems to be relative to the pipeline configuration install location. As I am rolling this out across platforms, having the ability to do something like:        

      display_name: Obj File, name: obj_file, icon: {$GLOBAL_ICON_PATH}/obj_file.png,

    would be ideal.

     

    Thanks!

     

  • 0
    Avatar
    Barbara Laigneau

    Hey! I noticed that the "include" syntax doesn't work as expected: in the documentation, it says we can include an optional file and if the file doesn't exist, it's not a problem. But for now, the file must be created otherwise there is a mistake and the environment file doesn't load. It would be great to have this functionality! (I talked with KP about it in a ticket but I don't know if you discuss about it internally)

     

    Cheers,

Please sign in to leave a comment.