Developing Apps

Getting Started with App Development

This is a brief introduction to App development and how the Shotgun Pipeline Toolkit can make things easier for you when you develop new tools.

Introduction

The Shotgun Pipeline Toolkit is not only a collection of Apps and Engines maintained by Shotgun Software - it is also a framework and a development platform which makes it easy to quickly build custom pipeline tools! This document outlines the workflows in both these cases and also explains some of the basics around app development.

All apps and engines maintained and released by Shotgun Software are open source and can be found in github: https://github.com/shotgunsoftware

The Starter App Template

When starting a new app, we recommend that you head over to https://github.com/shotgunsoftware/tk-multi-starterapp and take a look at our starter app! This is a "blank" app designed to be used as a starting point when you build new apps. By using this app you get all the standard toolkit boilerplate code set up for you.

Forking an existing app

Because all the Shotgun Toolkit Apps are open source, it is also possible to use their source code as a base for further app development. If you find that one of our default apps is missing a crucial feature, you can clone it in github and continue your own development! We love pull requests, so if you feel you have made a change that can benefit others, don't hesitate to feed it back to us as pull request. We may then fold it back into the main version of the App.

If you think that the feature that you feel is missing is something that would be generally useful, we would love if you would reach out to us via support@shotgunsoftware.com to hear your thoughts - perhaps this modification is something that we already have planned to do, or know that some other client has done, or maybe there is another way to achieve the same behaviour! We love to know what clients feel are missing from the current versions of our apps.

Creating a new app from scratch

This section goes through how to create a new app using our starter template. We assume that you are familiar with github and git workflows but please note that you can do Toolkit development even if you are not using git as your source control solution.

Create a development sandbox

Before you do anything else, we recommend that you set up a development sandbox by cloning the project configuration. This will result in a separate configuration where you can do development and make changes (and break things!) without affecting anyone one else on the production. You can find detailed instructions about configuration cloning in the following Document:

Forking or downloading the starter app repository

The easiest way to get started is to fork our starter app. You can find it here: https://github.com/shotgunsoftware/tk-multi-starterapp

If you don't want to use git source control just yet, you can just download the files from github as a zip, and unzip them somewhere on disk. It is easy to add it to git (or other source control packages) at a later point. If you already know that you need a git repository, you can also fork the starter app repo to your own git server and then check out the contents into your local dev area. Either way, the goal is to have the starter app code in your local development area so that you can start doing development.

Adding parameters and frameworks

At this stage, if you know that you will be using frameworks or have specific manifest parameters that you need, this is a great time to get these set up! This is because in the next step, when we install the app into an environment, the system will take care of all the configuration for you automatically.

For example, if you plan to use the widget and shotgun utils frameworks for your app, add the following to the info.yml manifest:

# the frameworks required to run this app
frameworks:
    - {"name": "tk-framework-shotgunutils", "version": "v2.x.x"}
    - {"name": "tk-framework-qtwidgets", "version": "v1.x.x"}

Minimum required framework versions

If there is a required minimum version for a framework, the minimum_version setting can be used in the info.yml manifest:

# the frameworks required to run this app
frameworks:
    - {"name": "tk-framework-shotgunutils", "version": "v2.x.x"}
    - {"name": "tk-framework-qtwidgets", "version": "v1.x.x", "minimum_version": "v1.5.0"}

The above will ensure that v1.5.0 of tk-framework-qtwidgets is available for the app to use. If it is not, the app will not be loaded at startup time and an error will be printed to the console. When the app is updated using tank updates, any configured frameworks not meeting their required minimum versions will be automatically updated along with the app.

For more information about frameworks and how they can be useful, check out the following links:

Add your new repository to an environment

Now let's get this App up and running inside of Toolkit!

With your development sandbox there will be a specific tank or tank.bat binary that you can use to address this particular configuration. Open a terminal and navigate to your dev sandbox. Using the local tank command, run the install_app command. When you install an app, you need to choose an environment and an engine to install it into. The engine is the application you will be running, so either tk-maya, tk-nuke etc. The environment is a collection of tools that you want to run against a specific work area. In our default configuration, when working with Shots, the environment is called shot_step and when working with Assets, the environment is called asset_step.

For example, if you have your new app code located in /Users/manne/dev/tk-multi-mynewapp, you run the following to install it:

> cd /your/development/sandbox
> ./tank install_app shot_step tk-maya /Users/manne/dev/tk-multi-mynewapp

Any frameworks that you app requires will be automatically installed by the install command. Now if you start up maya for the environment you just installed into, you should see the starter app appearing on the menu!

Anatomy of the Template Starter App

The template starter app found over at https://github.com/shotgunsoftware/tk-multi-starterapp contains the following basic structure:

  • The app entry point and menu registration can be found in the app.py file. This is where you typically set up your classes and get things initialized and get menu items registered.
  • The manifest defines all the different settings that this app requires when it is installed. Settings are often useful if you want reusable apps and you don't want to hard code any values in the app itself.
  • The dialog.pycontains the logic and event callbacks etc. that produces the main app window.
  • The dialog.py loads in an auto generated widgets file residing in the UI folder. This file contains all the base layouts and properties of the dialog UI.
  • In the resources folder, the dialog.ui file is a QT Designer file which you can open up and use to rapidly design and define the look and feel of the App. Once you have made changes you have to execute the build script which will convert the ui file to python code.

Do development

You are now all set to do development! As a convenience, as soon as you have one more more dev items in your configuration, Toolkit will automatically add a Reload and Restart option to the menu inside of maya, Nuke etc.. Clicking this will reload your configuration and code and then restart your engine. This means that you can iterate quickly: Start maya once, then each time when you have made code or configuration changes that you want to try out, simply hit the reload and restart button and your changes will be pulled in. Note that if you have any UIs active on screen, these will not automatically update, but you have to go in and re-launch the UIs from the menu.

Do testing

When you want to test your code, you can easily invite other users to your dev sandbox by adding them to the pipeline configuration entry in Shotgun. As soon as you have added a user, they will see new entries on their menus inside of Shotgun.

They can then go in and for example click the launch maya menu entry associated with your dev sandbox. When maya starts up they will be running it from your configuration and seeing your in progress work, just the way you are seeing it. Note: Make sure they have access to see your app code too, otherwise the app will not load.

Preparing your first release

When you feel ready to create your first release, if you haven't already, we recommend that you start using source control to make sure that changes are tracked. Toolkit supports git (http://git-scm.com/) out of the box and we strongly recommend using this. The rest of this document assumes that are familiar with git and have it set up on your machine.

Now you may already have a git repository set up for your app. If you haven't, it is time to create that and push your first commit. Toolkit git repos need to contain just a single app -- basically your git repo should have the same structure as the starter app repository where you got the original code from.

When you do development, toolkit uses a dev location in the environment file, something looking like this:

location: {path: /Users/manne/dev/tk-multi-mynewapp, type: dev}

This instructs toolkit to load the app code directly from disk in the given location, which is great for development, where you want to change the code all the time.

For released software, we want to ensure that things are versioned so that they can be upgraded safely and easily. All Toolkit's built-in apps uses the toolkit app store to track updates and releases, and they will have a location tag that looks something like this:

location: {name: tk-multi-setframerange, type: app_store, version: v0.1.7}

This allows toolkit (for example the tank updates command) to check when updates are available, update and maintain configurations in a very safe way. Whenever a new version is available, toolkit will automatically download the code and place it in a local "app cache" on disk and ensure that users have access to it.

For in-house tools, you can use the same principle, but instead of using the toolkit app store you can use tags in git. So for your new app, create and push a new tag to git. Note that if you have cloned our starter app, there are already tags defined. We strongly recommend that you use Semantic Versioning when creating your tags ( see http://semver.org ). Toolkit will use these version numbers to try to determine which version is the most recent, and by following the convention vX.Y.Z your app versioning will be compatible with Toolkit.

Installing changes into the production config

Once you have created your first tag in git (eg. v1.0.0), we can safely install this in your production config. This means that everyone on the production will get access to the new tool. Installing the app is done using the install app command. Make sure that you are using the tank command that belongs to the primary production config when you run the install command:

> cd /your/primary/tank/configuration
> ./tank install_app shot_step tk-maya user@remotehost:/path_to/tk-multi-mynewapp.git

This will find the highest version tag in git and install this into the environment. Once installed, you can simply run tank updates, and if new tags have been created, you will be prompted if you want to upgrade. The workflow is now identical to the one which happens with official app store apps.

Modifying an existing app

Rather than starting from an empty starter template, it is sometimes necessary to add a minor feature to an existing app, for example one of Shotgun Software's standard Apps. When you work with a modified version of an app, you typically want to 'track' against the source app and make sure to regularly pull in changes and bug fixes.

When you do this type of development, you pick up the parent code, then apply some of your changes, then release this to your pipeline. Your release effectively consists of the base version of the app PLUS your applied local changes. We recommend that you add a version suffix to the existing version number. This will work seamlessly with Toolkit and is relatively straight forward. The following workflow illustrates how to proceed:

  • You for the parent app and create your own repository. With the fork you get all the git tags. The latest tag is called v0.2.12 and the master branch basically equals the contents in this tag.
  • You apply your changes and commit to your master. You now basically have v0.2.12 PLUS your changes. When you release this to production you need to create a tag. Name the tag v0.2.12.1, to indicate that your code is based on v0.2.12 and it is the first release.
  • Now someone finds a bug in your modifications. Fix the bug and tag up and release v0.2.12.2.
  • A number of important bug fixes have been released in the parent repository. Pull them down to your repository. The most recent tag is now v0.2.15 due to releases that have happened in the parent repository. Now merge your changes with master and test. You now basically have parent app v0.2.15 merged with your changes. Tag up v0.2.15.1.

The tagging scheme outlined above guarantees that the toolkit updates work correctly and makes it easy to quickly see what code each tag in your fork is based on.

Documentation Resources

Apart from this document, we also have a number of other documentation resources available that can be useful when doing app development:

The Core API reference documentation contains all the commands that you use to manipulate the context, the file system templates and other core mechanisms in the system.

The Platform API reference documentation covers the Engine and App base classes and contains detailed information about what your options are when you are writing App code.

The Apps Configuration reference documentation outlines the various ways you can set up parameters and configuration for your app.

Interactive Access to Toolkit

If you are simply interested in playing around with Toolkit in a more interactive fashion, you can easily do this inside of for example Nuke or Maya once you have started the Toolkit engine and can see the Shotgun menu. You can access all of toolkit via the currently running engine, which you can retrieve like this inside Maya or Nuke's Python shell:

import sgtk
engine = sgtk.platform.current_engine()

Once you have access to the engine you can reach most other aspects of the currently loaded environment:

# A Shotgun API instance
engine.shotgun

# The toolkit API instance
engine.sgtk

# The current work area object
engine.context

# All the apps currently loaded
engine.apps

If you are in a command shell with access to the tank command, you can open up an interactive Python session easily by running the tank shell command. Toolkit will try to start up as much of the system as possible for you, but depending on your context, you may get a limited set of accessors:

> tank Shot ABC123 shell

Welcome to the Shotgun Pipeline Toolkit!
For documentation, see https://support.shotgunsoftware.com
Will search across all Shotgun Projects.
- Found Shot ABC123 (Project 'Chasing the Light')
- Using configuration 'Primary' and Core v0.14.214
- Setting the Context to Shot ABC123.
- Started Shell Engine version v0.3.4
- Environment: /mnt/software/tank/chasing_v2/config/env/shot.yml.
- Running command shell...

----------------------------------------------------------------------
Command: Shell
----------------------------------------------------------------------

Welcome to Shotgun Pipeline Toolkit Python!
2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)]
Running on darwin

- A tk API handle is available via the tk variable
- A Shotgun API handle is available via the shotgun variable
- Your current context is stored in the context variable
- The shell engine can be accessed via the engine variable
>>>

The Toolkit shell is handy when you want to do quick debugging or run little script snippets.

Big Buck Bunny - footage courtesy of (CC) Blender Foundation, www.blender.org

Follow

5 Comments

  • 0
    Avatar
    Toke Jepsen

    possibly one missing piece of information could be how to make/edit the changelog, although not entirely nessesary when you have the git commit log.

  • 0
    Avatar
    Manne Öhrström

    Hello Toke!

    We actually don't have a standard for release notes at the moment outside our standard apps. The standard apps take advantage of the github wiki feature and this wiki page is processed and translated into the toolkit app store version page. Here is an example of a github version history and here is the equivalent version history in the app store.

    With a project managed purely in git, it is harder to settle on a standard for storing the version history. Github provides both a wiki and their (semi-)new release feature, which looks really cool and which we may start utilizing a bit more. But for a studio git server, neither of those facilities would be present. 

    Perhaps we should start having a file as part of each app which contains the version history and move all our release note wikis pages into this file. Do you think this would be a good solution?

    Thanks for the feedback!!

  • 0
    Avatar
    Toke Jepsen

    Hey Manne,

    Think using the wiki pages on github is great, and super easy to update so in my opinion that is enough. Although I added a page release notes page for an app, and tried updating through the tank shell, it didnt seem to get the information. Are there some special formatting in the release notes wiki that the shell updater reads?

  • 0
    Avatar
    Manne Öhrström

    Hello!

    The process which converts the release notes wiki page into an update summary when you run tank updates actually only currently works for toolkit app store apps (and involves a couple of steps!). I really like this idea of supporting / moving release notes in a file and not just on a github wiki page -  I think thin is probably the way to go for the future! We really want to encourage clients to write apps, and we want the same set of features available to them as is available to us when we write and deploy apps! I will log a feature requests and pass it on to our product team for triage and consideration!

  • 0
    Avatar
    Andrew Cuthbert

    Hey,

     

    I was just looking for a way to submit a changelog for my own apps. Did you get any further with this thought?

     

    Cheers

Please sign in to leave a comment.