How can we test a pipeline configuration before deployment?

We're getting started with Shotgun, coming from a continuous integration (e.g. Jenkins CI) background. In other words, we're used to running automated tests (and maybe manual tests) before performing a deployment.

It looks like the tank tool is the preferred/de facto deployment mechanism for pipeline configurations (`tank move_configuration`?) and it includes an automated test command (`tank validate`). Is this the only built-in test mechanism? Is it possible to invoke things like unit tests from the validate tool? Basically, we want to reduce manual testing of our pipeline configuration, app tweaks and new apps wherever possible.


  • 0
    Manne Öhrström

    Hi David!

    We have unit tests and run CI on the toolkit core code and some of the app APIs, but we don't have a way to test a pipeline configuration itself. Most of our clients are doing manual validation after updates, e.g. run nuke after they have grabbed the latest nuke updates. 

    But this is a really interesting topic - what sort of end goal did you have in mind? Looking for syntactical problems in the config? In this case, the tank validate should be a good starting point and something we could potentially extend to return a data structure when executed in the tank command API mode (you can run most tank commands using the API). Testing beyond that is potentially harder, a lot of it I imagine would go into the realm of UI driven test automation (which is something we are experimenting with internally for our QA process, but it's tricky!). But by running validate, you have at the very least made sure that all apps and config is valid and has been wired up correctly.

    btw you can find the code here: https://github.com/shotgunsoftware/tk-core/blob/master/python/tank/deploy/tank_commands/validate_config.py - if you have something specific in mind, feel free to send us a pull request!

  • 0
    David Rix

    Ok, that gives us some clarity on the limits of what's easy to do. Thanks! If we think up anything cool to contribute, we'll send it through :)