Shotgun Utilities

Shotgun Utilities Framework

A Collection of Qt objects and utilities to simplify app development and Shotgun connectivity.


By: Shotgun Software
Latest Version: v5.2.1 (prod)
For our version numbers, we follow the Semantic Versioning standard.
System Name: tk-framework-shotgunutils

Overview

A collection of Shotgun related Helpers and utilities to make App development easier. Includes a Shotgun (MVC) Model which makes it easy to connect a Shotgun query with a standard QT View Widget such as a list view, tree view or spreadsheet. Includes a settings module which makes it easy to manage user preferences in apps and an asynchronous (background) Shotgun data retriever.

Documentation

The API Reference documentation for this Framework can be found here:

http://developer.shotgunsoftware.com/tk-framework-shotgunutils

Installation and Updates

Updating to the latest version

If you already have this item installed in a project and you want to get the latest version, you can run the update command. You can either navigate to the tank command that comes with that specific project, and run it there:

> cd /my_tank_configs/project_xyz
> ./tank updates

Alternatively, you can run your studio tank command and specify the project name to tell it which project to run the update check for:

> tank Project XYZ updates

Collaboration and Evolution

If you have access to the Shotgun Pipeline Toolkit, you also have access to the source code for all apps, engines and frameworks in Github where we store and manage them. Feel free to evolve these items; use them as a base for further independent development, make changes (and submit pull requests back to us!) or simply tinker with them to see how they have been built and how the toolkit works. You can access this code repository at https://github.com/shotgunsoftware/tk-framework-shotgunutils.

Special Requirements

  • You need Shotgun Pipeline Toolkit Core API version v0.18.45 or higher to use this.

Configuration

No Configuration Needed!

This item does not have any options to configure!

Release Notes

Welcome to the release notes for this Framework. Below you will find an overview of all the changes we did for each release. We try to be as detailed as possible and include all bugs we have fixed, features we have added and things that may have changed. If you have questions about a particular release, don't hesitate to contact our support team!

v5.2.1

2017-May-16

Adds include_root option to hierarchy model for displaying root item

v5.2.0

2017-May-03

Hierarchy related additions and changes

Details:

  • Adds data retriever support for nav search methods. If used with an older core/shotgun API which does not have the nav search methods accessible, it degrades gracefully and returns an empty data set.
  • Adds async expansion/fetch_more methods to the hierarchy model via the async_item_from_entity and async_item_from_paths methods.
  • Upgrades to 0.18 logging and bumps minimum required core to 18.45.
  • The hierarchy model _load_data method has a breaking interface change. Previously, it would take a path argument, now it takes a root argument instead in order to avoid using path strings to drive the API.

v5.1.3

2017-Feb-28

Documentation updates

v5.1.2

2017-Feb-07

Downgraded log messages regarding extra cache files from 'warning' to 'debug' in ShotgunDataRetriever._get_thumbnail_path(). Removed old thumbnail cache path logic that was replaced in March 2015. If any of the old thumbnails still exist, they will now be recached into the new directory structure.

v5.1.1

2017-Feb-01

Properly retrieve the application palette for both Qt 4 and Qt 5 when initializing the Hierarchy model.

v5.1.0

2017-Jan-26

Added ShotgunDataRetriever methods to download entity thumbnail source files

Details:

  • Added ShotgunDataRetriever.download_thumbnail_source(entity_type, entity_id, bundle) and Shotgun DataRetriever.request_thumbnail_source(entity_type, entity_id) methods that retrieve the original file uploaded as an entity thumbnail.
  • If the source file has already been cached, a path to it will be returned instantly. Otherwise, it will be downloaded from Shotgun and placed in the standard cache location on disk.
  • The full path to the downloaded file is returned.

  • Updated all download calls to use the newly introduced use_url_extension parameter of sgtk.util.download_url() which derives the filetype extension for the cache file from the resolved url, instead of hard-coding it to ".jpeg". If the use_url_extension functionality is not available, the previous behavior of using ".jpeg" for all cache files will be envoked.

v5.0.1

2016-Dec-19

Optimization and deferred loading.

Note: This is a major revision bump to indicate a changed behavior.

As of v5, the ShotgunModel and HierarchyModel classes implement a new, optimized data backend to better handle large datasets. The models now load and save significantly faster and can handle orders of magnitude more data.

The trade-off in order to achieve this speed increase is that the data set is loaded into memory in a deferred fashion. If you visualise a tree of tasks, grouped by step, shot, sequence and project, the data will be loaded as you click the expand icon in the UI. Previously, the entire model was loaded at initialization.

This is a change in behaviour and means that certain operations will not work the same way in v5 as they did in v4 of the Shotgun Model. One example is filtering using the QSortFilterProxyModel, which assumes a model where everything is loaded. In order to achieve the same behavior with ShotgunModel v5, the entire model needs to be loaded prior to filtering. This can be achieved via the helper method ShotgunModel.ensure_data_is_loaded

In addition, the model has been updated to correctly retain state when data is updated or deleted. Previously, these operations would force a model reset. Now, when new shotgun data arrives, the model correctly handles all update cases, so if an item has been updated or deleted, the model is no longer cleared but instead updated. Any selection that exists in associated views will be retained.


v4.5.2

2016-Dec-14

Bug fixes around performance improvements

v4.5.1

2016-Nov-30

Minor tweaks and bug fixes

v4.5.0

2016-Nov-24

Optimizations, refactor and deferred loading.

v4.4.15

2016-Nov-19

All schema cache methods that accept entity type and field now handle bubble field notation

v4.4.14

2016-Nov-02

API Documentation updates only, no code changes.

v4.4.13

2016-Nov-01

Including hierarchy model docs.

v4.4.12

2016-Sep-23

Updated to fallback gracefully with older core

v4.4.11

2016-Sep-20

Adds additional fallback for python-api without hierarchy support

v4.4.10

2016-Sep-12

Adds sg to timestamp data cleanup back into sg models

v4.4.9

2016-Sep-09

Bug fix for checking cached schma

v4.4.8

2016-Sep-08

Allows for graceful failure when Hierarchy API is not available

v4.4.7

2016-Sep-02

Shotgun Hierarchy Model

v4.4.6

2016-Sep-01

Added missing entity dark icons

v4.4.5

2016-Aug-15

Ensures that cached schema and status data is loaded from disk when requesting that data from a project that isn't the active context.

v4.4.4

2016-Aug-08

Reduces noise from the background task manager debug messages and resolves a schema_cache bug related to per-project lookup of type schemas.

v4.4.3

2016-Jul-22

Updates field editable/visible to use project id based schema caches

v4.4.2

2016-Jul-19

Allow the editor widgets to be used with the shotgunmodel

v4.4.1

2016-Jul-19

Fixed variable reference issue in shotgun globals, causing warnings to be printed in some cases.

v4.4.0

2016-Jun-27

Adds the ability to download attachments via the ShotgunDataRetriever.

Details:

In the same way that the data retriever can be used to asynchronously download thumbnail images, it can now be used to download data associated with Attachment entities in Shotgun.

v4.3.1

2016-Jun-14

Changes a log_warning into a log_debug to cut down on noisy output.

v4.3.0

2016-Jun-09

The schema and status caches are now stored in memory on a per-project basis.

Details:

Schema and status information can be queried for a given project, even if it is outside of the current context. By default the behavior remains as it always has and the current project is queried. If a project id is provided and the version of tk-core being used does not support looking up cache paths on a per-project basis, the current project will be used instead as a means of remaining backwards compatible with older core releases.

v4.2.3

2016-Jun-08

Fixes a bug with tooltip generation in the Shotgun Model

Details:

The Shotgun Model can now generate a tooltip for model items that reference a field with more than two period in their name, e.g. entity.Shot.sg_sequence.Sequence.sg_status_list.

v4.2.2

2016-Jun-08

Optimization - settings module no longer requires an initialized shotgun API handle at initialization.

v4.2.1

2016-May-25

Adds utility methods to cached schema api

v4.2.0

2016-May-18

Adds support for an "additional_filter_presets" optional argument for ShotgunModel when loading data. Also adds support for detecting and handling bubbled fields when getting field data types.

v4.1.2

2016-Apr-20

Introduces an alternate event dispatching scheme for events from the background manager's to avoid a deadlock bug between the GIL and PySide.

Details:

Signalling between two different threads in PySide is broken in several versions of PySide. There are very subtle race conditions that arise when there is a lot of signalling between two threads. Some of these things have been fixed in later versions of PySide, but most hosts integrate PySide 1.2.2 and lower, which are victim of this race condition.

The background task manager and its workers does a lot on inter-threads communications and therefore can easily fall pray to these deadlocks that exist within PySide.

Therefore, the worker threads now use a different event dispatching mechanism based on QMetaObject::invokeMethod.

v4.1.0

2016-Mar-22

Adds support for columns in the Shotgun data model.

Details:

In addition to column support, bug fixes related to Toolkit operating in a "site" context plus better handling of null thumbnails on entities are included.

v4.0.4

2016-Jan-15

ShotgunStandardItem now displays the entity's type in its tooltip instead of on its label.

v4.0.3

2015-Dec-15

Added data retriever support for find_one

v4.0.2

2015-Dec-09

Optimized tree rebuilding speed in the ShotgunModel.

Details:

  • Rewrote the tree building algorithm of the ShotgunModel which provides speed boosts from 2.5x to 6.5x depending on the number of nodes in the tree.
  • The ShotgunModel now advertises that is going to be reset before clearing the tree..

v4.0.1

2015-Nov-23

Fixed incorrect python imports

v4.0.0

2015-Nov-23

Bumped major version to 4. Deprecated ShotgunOverlayModel in favour of new view adapter object.

v2.5.1

2015-Oct-20

Improved cache handling for advanced use cases.

Details

This improves how cache files are partitioned when using the seed query parameter and the schema_generation parameters are being used. The documentation around these parameters have also been improved. Cache files containing thumbnails and the ones without are also separated in order to allow non-thumbnail caches to be loaded in multi-threaded scenarios without getting QT pixmap warnings.

v2.5.0

2015-Oct-13

Converted documentation to use sphinx. Added a shotgun_globals module with access to shotgun schema and other site-global data.

v2.4.3

2015-Oct-05

Adjusted code imports to avoid dependency cycles.

v3.0.4

Pre-release v3.x for testing with File Manager v2

Details

  • Code tidy up and code-review fixes
  • Fully merged changes from master 2.x.x branch

v2.4.2

2015-Aug-06

Added support for capped queries via a limit parameter.

Details:

You can now define a model which holds a maximum of X entries by passing a limit parameter to the data load command. This is then passed to the Shotgun API find() call to limit the results returned from the server. In conjunction with ordering your shotgun resultset, this can be used to effectively cap the data set that the model is handling, allowing a user to for example show the twenty most recent notes or similar.

v3.0.3

Pre-release v3.x for testing with File Manager v2

Details

  • PyQt fixes

v2.4.1

Moved some data sanitation methods into the utilities part of the Shotgun Model module.

Details:

This makes it easy for other model code outside of the shotgun utils framework to take advantage of these methods.

v2.4.0

Optimizations and bug fixes.

Details:

  • Adds a new bg_load_thumbs flag to the ShotgunModel constructor. This is false by default. Setting the flag to True will request that the sgdata async fetcher not only downloads the thumbnail but also loads it up into a QImage object. This means that most I/O will happen in the worker thread rather than in the main UI thread. It seems to lead to much better behaviour, especially when you have lots of images.
  • When bg_load_thumbs is set to True, the method _populate_thumbnail_image() is called from the base class instead of the normal _populate_thumbnail() method. Code subclassing the Shotgun model needs to implement this _populate_thumbnail_image() method if they want to take advantage of the new behaviour.
  • Thumbnails are downloaded faster from S3. Instead of doing a find prior to every S3 download, this is only done on failure. Thumbs queued up several times (not an uncommon situation) are only dowloaded once.
  • Improved debugging output to aid troubleshooting.
  • Fixed issue with S3 logic inside attachment entities, causing queries pulling down attachment fields with uploads to always invalidate and refresh.
  • Fixed an issue with callbacks firing when thumbnails are removed on the server and not returned Shotgun connections are now created JIT - for optimal performance of initial thumbnail download tasks.

v2.3.0

Added additional methods for async data access and retrieval

Details:

  • Schema method now combines both schema_read and schema_entity_read
  • execute_find() call in sg data fetcher now supports arbitrary arguments that are being dispatched to sg.find()
  • Added delete, update and create methods to async data fetcher
  • Added method which allows for the execution of an arbitrary method in the async data fetcher
  • Fixed an issue where the wrong request_type was being returned for thumbnails.

v2.2.0

Added methods for better overlay access and schema retrieval.

Details:

  • The shotgun asynchronous data fetcher now supports Shotgun schema retrieval via a new get_schema() method.
  • The overlay model now exposes spinner access via a _show_overlay_spinner() method.

v2.1.4

Fixed a race condition in SgDataRetriever.

Details:

Fixed a race condition in SgDataRetriever. This happens if you try to set the Shotgun connection after starting the thread.

v2.1.3

Various changes and tweaks to the ShotgunModel class.

Details:

  • Signals are now blocked during ShotgunModel destruction to avoid instability
  • Changed a couple of members/methods to be protected rather than private
  • Added an method to easily get a QModelIndex from an entity
  • Exposed an property that returns the list of entity ids in the model

v3.0.1

Initial pre-release v3.x for testing with File Manager v2

v2.1.2

Memory related bug fixes and improved cache path structure.

Details:

Improved workarounds for a known pyside/qt issue where clearing and resetting of models will result in dangling pointers on the PySide side. This change implements close() and reset() for the Shotgun Model in the interest of stability - anyone trying to call reset() will now get an exception raised instead of a (highly likely) scenario with stability issues or crashes. close() is reimplemented in order to avoid the default C++ implementation and instead use the hand crafted recursion based clearing that ensures that all objects both on the C++ and Python side are removed.

Thumbnail and cached query paths are restructured to better handle long path name as we were running into issues with MAX_PATH on windows. We had to deal with taking an S3 guid and shard that into the file system (which wasn't done nicely before) so now all thumbnail paths that shotgun generate are first hashed and then sharded on disk. This should result in an even distribution of thumbnails on disk.

Paths logic for shotgun caches have been updated as well in order to avoid MAX_PATH on windows (260 char limitation). Warnings are emitted if the paths are getting close.

The code attempts to move thumbnail files from the old to the new path cache structure if possible in the interest of saving disk space and bandwidth.

v2.0.6

Improved storage logic for cache files.

Details:

While this update will work with pre-015 cores (where cache files will be stored in the pipeline configuration cache directory), it is really intended as a 0.15 followup, attempting to resolve some things related to the cache storage:

  • sg data caches are now stored in the official bundle cache location, allowing clients to customize the location of cache data. In addition, it also creates a more clear, standardized structure and resolves potential buildup of cache files in temp storage, which sometimes is highly limited by design.

  • we have seen potential evidence of md5 cache collisions with the current implementation, where a large number of parameters are squeezed into a single string that is md5 hashed. This change breaks out the structure further and hopefully no collisions should appear.

  • improved I/O handling around cache writes.

v2.0.5

Fixed a bug causing some thumbnail comparisons to fail.

v2.0.4

Minor administrative adjustment: This framework now requires core v0.14.58 or greater to correctly handle installation in certain setups.

v2.0.3

Fixed a bug which was preventing some I/O warning messages from being emitted.

v2.0.2

Fixed a bug relating to unicode conversion.

Details: The recursive unicode conversion method was designed in such a way that it only worked on shotgun multi-entity list data structures, not normal lists of simple data items. This could cause it to fail for any field which returns a list of things (or a customization which relies on this).

v2.0.1

Minor bug fixes and adjustments.

Details: - Cache now correctly responds whena thumbnail is updated. Previously thumbnail updates were ignored. - Added improved trapping of use case where cache comparisons fail due to strange values in cache. This attempts to resolve of at least work around a long standing issue where NULL errors would appear infrequently when the loader was retrieving its content. - Multi entity values are now formatted as a comma separated list. Previously these were printed out in their "raw" form, as a list of link dictionaries. - Items with their name field set to None are now displayed as "Unnamed Asset" rather than just "Unnamed". - Cache writer now explicitly closes its cache file handle.

v2.0.0

Moved UI related modules into a separate widgets framework.

Details: Please note that this is a breaking change - hence the major version number increase! This release has removed the shotgun_view, overlay_widget and help_screen modules. These are now located in tk-framework-qtwidgets instead for better separation. In addition to this, the following smaller changes were carried out: - A shotgun_data module was added. This was previously a private part of the shotgun model, but has now become a first class citizen of the framework. This module makes it easy to do asynchronous shotgun queries and also manages caching and retrieval of shotgun thumbnails. - Changed the framework icon - Improved some documentation relating to time zone management.

v1.0.10

Data refreshing signal and bug fix for simpleshotgunmodel.

Details: New data_refreshing signal emitted when data starts refreshing (thanks Tony!) Bug fix for how simpleshotgunmodel passes on hierarchy.

v1.0.9

Bug fixes for PyQt4 and fixed an interface typo.

Details: Fixed an issue in the widget delegates framework where the QWidget.render() method would work correctly on some pyqt distros but not all. Fixed a spelling error in the set_shotgun_connection() method name in the public interface.

v1.0.8

Improved exception handling in the settings module.

v1.0.7

Improved documentation.

v1.0.6

Updated framework icon.

v1.0.5

Fixed a bug causing the help screen transitions not to show on Linux.

v1.0.4

Bug fixes and tweaks.

Details: - Help screen module now hides arrow buttons on the first and last page. - Fixed issues with help screen object parenting, causing problems on earlier versions of PySide/Shiboken. - Fixed an issue with object deallocation which was causing crashes on earlier versions of PySide/Shiboken (Nuke 6.3).

v1.0.3

Changed initialization order so that all data population happens prior to adding objects to the model.

Details: This ensures that by the time signals are emitted by the model, the item data is fully initialized.

v1.0.2

Fixed a bug causing models to behave erratically when items where nested in a hierarchy.

Details: For more information, see https://github.com/shotgunsoftware/tk-framework-shotgunutils/commit/a4ab39c0bbea8b3f06f7a17c1ba34f9f5ece20cb

v1.0.1

Minor tweaks to default values formatting.

v1.0.0

First version of public interface.

Follow

9 Comments

  • 0
    Avatar
    Philip Scadding

    I'm trying to follow this, but I can't get the import_framework to work. I have an app that is launched through the shell. but I keep getting:

    import_framework: <Sgtk App 0x02c6ccf8: asset\_manager, engine: <Sgtk Engine 0x03689ac8: tk-shell, env: project>> does not have a framework tk-framework-shotgunutils associated!

    I'm not really sure how to proceed, can you use this on an app that is launched from the tk-shell engine?

    thanks Phil

  • 0
    Avatar
    Manne Öhrström

    Hello!

    We are working on a sample app to make it quick and painless to get started. In the meantime, here are some pointers!

    • Make sure your app has the framework requirement set up in its manifest.
    • In the frameworks section of your environment, the framework needs to be registered. This is done automatically at app install but if you do dev you may have to add it yourself. Once you have added this, run tank cache_apps to ensure that all required binaries exist on disk.

    Hopefully this should get you started! Any problems, let us know!

    Thanks! :) Manne

  • 0
    Avatar
    Philip Scadding

    Thanks, that's sorted it, it was the info/manifest file I needed to append. Funny enough it dawned on me that's where I was going wrong late last night when I should have been sleeping.

  • 0
    Avatar
    Manne Öhrström

    Let us know if you need more help with the API specifics! And hopefully no more sleepless nights because of the shotgun-utils framework! :)

  • 0
    Avatar
    Philip Scadding

    Hi Again

    I've had a play and got the help pop up window working nicely, but I've gone back to looking at the hello world example again, and I'm getting an error.

    I have this code for time being in the in the init of my AppDialog class. I've copied it the same as you have written it up in the example, with the exception I'm using sgtk instead of tank and replaced "parent widget" with self. I wonder if its because I'm using self, but the class is derives from QtGui.QWidget.

    But I get the following error:

    <code> File "Z:\shotgun\software\sgtk\studio\install\frameworks\app_store\tk-framework-shotgunutils\v1.0.9\python\shotgun_model\shotgunmodel.py", line 375, in _refresh_data fields = list(set(self.__hierarchy + self.__fields + ["image"]))

    cannot concatenate 'str' and 'list' objects

     

  • 0
    Avatar
    Manne Öhrström

    Hey there Philip!

    I know we reached out to you separately, but just wanted to followup here for completeness; This was a bug and we deployed a fix for it as part of v1.0.10 of the utils framework!

    Thanks!

    Manne

     

  • 0
    Avatar
    Alon Gibli

    Is there an easy way to add columns to the model? For example, say I want to display various shotgun fields in separate columns. I can add data roles with _populate_item/_finalize_item, but to add columns I'd need a way to return a list of QStandardItems back to the ShotgunModel base class for each row.

     

    Thanks!

    • Alon
  • 0
    Avatar
    Manne Öhrström

    Hello Alon!

    Column support is something that is on our todo-list. For the initial uses of the shotgun model (loader, desktop app, other small tools) we didn't really need columns so we didn't add support for it just yet! It has come up in conversation several times over here and I suspect it is something we'll add in the not too distant future! I am not totally sure about the scope of this change - i don't think it is a huge addition to the code! If you do end up implementing something, we'd love to fold it into the official framework!

  • 0
    Avatar
    Alon Gibli

    Thanks for the reply! I'll let you know if we end up implementing something for this in ShotgunModel. For giggles, I tried to figure out ways to implement this behavior in a subclass, and ended up with absurdities like monkey-patching each items' appendRow method in _finalize_item... probably not a solution you'd recommend, haha.

Please sign in to leave a comment.