1

How does Toolkit work with external artists?

Hi,

After reading the documentation I'm unclear as to how Tank would work with external artists, who don't have access to our internal network.

Tank offers version control, can it also be integrated into a version control system like SVN or Perforce, that would allow us to synchronize
files with external artists?

Thanks,
Ben

댓글 26개

  • 0
    Avatar
    Jon Jones

    Has anyone successfully deployed the publishing tools yet? How would all this work?

  • 0
    Avatar
    Ryan Mayeda

    It would be interesting to hear if anyone has already experimented with this type of workflow with Tank, but from our standpoint, we still have a bit of core work to do before we can reasonably cover the requirements.  That said, we are making progress!

    Tank v0.13 introduced the concept of a Pipeline Configuration, which allows a Project to have more than one configuration that can be isolated to specific users.  While the main initial target use case is for developers doing testing outside of production, our thinking is that it will also be applicable to the outsourcing/remote artist use case as well.  For example, you could have one or more Pipeline Configurations tailored to external artists, each with customized templates, environments, Maya/Nuke application launch paths, etc.:

    Screen_Shot_2013-05-29_at_8.07.10_PM.png

    These configurations would then only show up for the external artists in menus:

    Screen_Shot_2013-05-29_at_8.02.30_PM.png

    The Tank install, containing the core/engine/app/framework code can also be "localized" to a Pipeline Configuration, which makes them self-contained, and thus in theory easier to package up for an external artist as well.

    As you can see, most of the work we've done in this use case's direction has to do with managing the configuration for an external artist.  That said, our sense is that there is still a bit more to do on this component, namely:

    • Tank's path cache makes some assumptions that this use case would break (we are going to do a spike on this work in our next sprint though).
    • The "Primary" Pipeline Configuration would still show up for external artists, but they would get an error because they can't access the "Primary" (aka in-studio) filesystem (the external configuration would still work though).
    • It would be nice if there were convenience utilities for an external artist pulling the configuration down and installing it locally for themselves (although this could also be done by you guys).
    • I'm probably leaving something out, but hey...  :)

    Once we get thing squared away with external artists running Tank itself, the next step is to think about the data transfer/synchronization aspect.  To that end, our hope is that the hooks we provide in Tank apps can be leveraged to do the heavy lifting.  For example, in our default setup, when something is published there is a hook that copies the file to the publish location.  For an external artist, this hook could be overridden to instead push something to Perforce or upload to an FTP site, say.  Alternately, the publish location could just be a Dropbox or Box or other sync filesystem setup that would then get the files back to the studio.

    So far, we're hearing that people are using a lot of different transfer approaches, so it would be great to pin down a favorite (or maybe a couple favorites, at least) to focus on initially.  With any approach, there may also need to be a bit of bookkeeping work to make sure that paths created by an external artist map back to the studio properly for when an internal artist wants to use something generated by an external artist, and vice versa.

    In any case, we're really excited about this workflow, and we think there is a lot of potential for Tank to help make this easier for clients.  We'd love to start getting into more specifics with clients who are interested in this workflow so we can start fleshing out our plans and sharing them with the group!

    Ryan.

  • 0
    Avatar
    Jon Jones

    That's awesome! I can't wait to dig into this. But one key question -- what files\plugins\tools do I have to send the external vendors to set up Tank? Do they run the same install as I did when I was setting up Tank for the first time, or something else?

     

    Thanks!

  • 0
    Avatar
    Ryan Mayeda

    As far as the files that would actually be required to run Tank (i.e. independent of where publishes are sourced from and are written to) are concerned, my initial thought would be to have the process be more like cloning a Pipeline Configuration within a Project and localizing it.  That way, all of the installed components (Tank core, engines, apps, frameworks) and configuration could be "bundled" together and external artists wouldn't necessarily need to manage both their own Tank install as well as a Project-specific configuration.  Also, an external artist might be working with more than one studio, each with a different Tank setup.  With that in mind, it would probably be a lot simpler to just have each Project operate as a "localized" setup that doesn't have to account for what any other Project is doing.

    This is starting to get into future development, but ultimately, we would want/need to have a way to "bundle" up a cloned Pipeline Configuration and link it to a given external artist in Shotgun.  Maybe that just uploads the bundle to Shotgun and lets the external artist download it from there?  It feels like a bunch of options would be possible.  Then, the artist would download the "bundled" Pipeline Configuration and install it to a place of their choosing (or maybe they don't get to choose this?), which would also update Shotgun.

    In any case, it feels to me like it will be beneficial to allow for the Projects to have "localized" setups and not have to force an external artist to manage their install the way a TD or admin would want to within a studio, or at the very least make it so that stuff is optional.  How do you feel about this?

    Ryan.

  • 0
    Avatar
    Jeremy Hoey

    Long term, it would be fantastic to have a bundled configuration that can be installed by even the most non-tech-saavy remote artist. This configuration would, in my dream scenario, do the following:

    1. Give the remote artist a Tank-managed pipeline on their local system that contains only the shots assigned to the artist within Shotgun.
    2. Secure all the data in that environment with some form of disk- or file-level encryption to guard against possible hardware theft (this is not something that can be left to the artist to arrange independently - facilities need to have control over this to be able to comply with their contractual obligations to their clients).
    3. Have some form of built-in encrypted file transfer functionality to handle syncing of data between the remote artist and the facility. I envisage a choice between a custom VPN server/client combo with keys generated by Shotgun, or one of the common secure file transfer protocols. Things like Dropbox or straight FTP are unlikely to meet client data security standards, although it would certainly be handy to have those options for situations where data security isn't a big deal.
    4. In essence, this would create a closed, secure system that is entirely managed by the facility.

    I have no idea how feasible all of the above is - I'm not a developer - but I'm pretty sure that if you could somehow crack it, a lot of facilities would start to look very seriously at the possibility of employing a geographically-distributed workforce for at least some of their work. Which would be a bit of a game-changer, especially if it was nicely integrated with cloud rendering solutions like Zync...

  • 0
    Avatar
    Stephen Baker

    I would be just interested if artists who are in the studio can download/check out a version of a maya file to an external drive.  Go home work on it.  And then when they return to the studio they have the ability to save it back into the system as a new version.  Is this possible with the default configuration?  I guess I am asking how would someone add a newer version of the files to the server through Shotgun and file manager or some other method outside of just copying it into the mnt/projects/.../work/folder.

  • 0
    Avatar
    Ryan Mayeda

    Hi Stephen!

    For your go-home-and-work-from-an-external-drive use case, would this be the only way artists worked?  If so, that is potentially doable now, where you use a multiroot configuration to point the work area templates to the external drive, and the publish templates to your primary filesystem.  When artists would do work, it would all go to the external drive, and they wouldn't be able to publish until they returned to the office and could see the primary filesystem.  Even if they might do work at both places, that could potentially be handled by having multiple pipeline configurations, one that puts work areas on the primary filesystem and another that puts them on the external drive.

    What I'm describing is more on the "possible" side but would have some rough edges (stuff we hope to address soon!).  For example, an artist could try to publish from home, but would get an error because the primary filesystem wouldn't be accessible.  If there is a separate pipeline configuration, an artist working at home might still get an ignorable error about the primary pipeline configuration not being accessible.

    That said, it could be an interesting experiment.  Let me know if you want to give this a shot and I can whip up a sample config for you that tries to work this way.  Your use case is a little bit easier in that the artists will at least initially have access to the primary storage.  But hopefully we will be able to solve that soon as well...!

    Ryan.

  • 0
    Avatar
    Jonas Drehn

    Hi Guys,

    We're also very insterrested in how we could share work outside the studio and get it back into tank with screening rooms QT's etc.

    We outsource work all the time - but mostly to small studios or induviduals who has there own pipleine. So instead of getting them a full Tank / SG intergration, we are looking for a good way to get work in and out our pipeline. I was thinking a little more simple.

    As a begining connecting SG with our FTP. So when I assign tasks to a external artist / studio - it will make the plates availible to them via Symlinking the folders where Tanks put the plates/footage to a FTP login.

    But what about getting renders back in ? Would it be easy to do a tank App - that checks a directory for files (framestacks) and add each sequence it finds to a shot > version in SG (based on the folder naming convention) ? 

    I also considered letting external artist / studio's add versions via the RV submit tool - but then the Path to frames has to be changed somehow by a script after the files has been transfered to our ftp server.

     

    best

    jonas

  • 0
    Avatar
    Ryan Mayeda

    Great to see so much interest in this - it really helps validate our thinking as well!  Jonas and James, I'm going to consolidate my reply to both of you, hope that's ok...

    Jonas:

    Much of what you're describing does feel doable with some additional apps.  Perhaps something like:

    • An app that lets you right-click on plate publishes in Shotgun and say "FTP to ____ Vendor" and the app would then go out and create the symlinks so that the data would be accessible on your FTP to the relevant vendor?  The app could conceivably have other options for doing an actual upload, uploading to some other service that isn't FTP, etc.  If you're using event triggers, you could also have those create the symlinks and then update Shotgun so you know what has been made accessible to your vendor.
    • For bringing renders back in, a watch folder app could crawl for uploads to your FTP site and then create the publishes in Shotgun as it found stuff.  Watch folders can sometimes be tricky with image sequences though (how do you know when all the frames have been uploaded?), so a modified RV submit tool could do the Shotgun creation but then also facilitate the FTP upload and populate the fields with the FTP path?  I've also heard a few requests along these lines where the users actually want to know what's come in, review the list, then have an app that brings things in (kinda like the reverse of the plate distribution use case).

    Admittedly this is a bit handwavy so I'm mainly trying to get a few more details out there so we can figure out the right scope!

    James:

    Yes, we would definitely have to account for the use case where a person works both in the office and out, and we would need mechanisms (like your "working remote" flag idea) to help dictate what should and shouldn't be available based on what someone is trying to do.  This is the kind of polish we will aim to do as we also look to ironing out all of the technical details as well.

    You also touch on a key area of how we manage what Tasks happen where (i.e. in the office or at home), as well as what data is where.  I've been working on a proposal for some Shotgun schema additions to help track this, and I'm hoping to have it presentation-ready in the next couple weeks.  The idea here is to lay the groundwork for people to track if a file is at a given location as well as which locations it exists.  Then it will be down to the file sync to do its thing and update Shotgun accordingly.  It would be great to get your eyes on the schema design and our hope is that it will be another big step in helping us make this workflow easier to manage.

    Ryan.

  • 0
    Avatar
    James

    Wouldn't external-Ryan always have that pipeline configuration though? What about a person working inhouse that can work remotely at anytime too?

    A flag in the user dashboard to set 'working remote' ?

    Definitely being able to pull down a bundled config / project tank setup would be best, directly from shotgun would be great

    As for the file syncing...

    It's either user managed,  or ;

    Correct me if I'm wrong, but you'd need to track the submits / publishes to the external primary locations back to the `local' inhouse primary locations. Even if the user was flagged as remote at the time of publish, the publishes would be still updating the db so the shotgun pages would update, but nothing in the inhouse file system would, so  you'd have files being reported as existing but failing to load on application launches in house.

    If there was a way to mark those as remote submissions, awaiting files in house, say an `internal '  'external' column in the files areas,  it'd then be down to the file sync itself?

    As suggested some kind  of custom transfer app that connects into the internal file system (both have the same paths, but the internal has placeholders), that then transfers the user selected files back up to the main file server (have fun with that one).If you could boast a file sync system of some kind that doesn't rely on an external sync tool like dropbox, it'd just be another notch in the belt really..

  • 0
    Avatar
    Jonas Drehn

    Hi Ryan,

    Thanks for your reply ... I like the idea about right-click on a plate publish and say FTP to ___ Vendor - We have our vendors as a disabled person entity, so I guess it would be easy to to have a field here, with the path to there directory on the FTP. For getting stuff back in, you're right about the watchfolder could be tricky with framestacks - maybe it would require the vendor to submit it through a Vendor login or a modified RV submit with upload functionality or something like that. But I also understand that a more manually workflow could work where we, on our side, would just take the uploads into RV and submit them from there. 

    We really wanna jump start this workflow, so I guess it would be good to know if you're planing for some overall vendor / external artist apps in the nearest future. I guesss we could easily do the FTP to ____ Vendor ourself - one question in this regard : would a Tank app be the best way to do this - and how would we be able to ask which vendor to send to from within the SG webinterface ? Our should we do this via eventstream and have a field in the publishes with a link to the Vendor (people entity)

    cheers

    jonas

     

  • 0
    Avatar
    James

    Hey ya

    Would be great to look over the scehma.

    Def great to see this being looked into :)

  • 0
    Avatar
    Nils Lerin

    Hi

    Very interesting topic. I'm trying to set up and develop this exact workflow, our team consists exclusively of external artists. I've managed to hook in to the publisher app to seamlessly download the required file from our ftp and then when the artist is done, upload it upon publish. Also they can render from maya/motionbuilder and upload a version to shotgun for feedback.

    I'm a bit stuck though on a security issue. Since we need to give every artist the shotgun.yml file, which includes the api-key in order to communicate with shotgun, we give every artists all the permissions of the 'API Admin' permission role. Should the key get in the wrong hands that person will gain full access to all our shotgun data. Is there a way to restrict a user of the shotgun api to have the same permission rights as its shotgun user? Or at least only access to one project? Is there a way to create new permission roles which can only access one project and then use that role for a project exclusive script/api-key?

     

    Cheers,

    Nils

  • 0
    Avatar
    Ryan Mayeda

    Hi Nils.

    Great point!  This is something we've been thinking about as well.  On the Shotgun API side, the v3.0.15 introduces the ability to log in as a HumanUser via the API so you don't necessarily have to use an admin API key:

    https://github.com/shotgunsoftware/python-api#changelog

    This is something we're looking to leverage for Toolkit as a whole, where the user could "log in" to Toolkit + Shotgun and have the setup as a whole take on the permissions for that given user instead of the catch-all API key.  We're thinking that Shotgun Desktop (previewed in our last webinar) could be our vehicle into this, what are your thoughts on this approach?  Any potential gotchas with making it so the security options for Toolkit align with the permissions for a given user in Shotgun?

    Ryan.

  • 0
    Avatar
    Nils Lerin

    Hi Ryan!

    That's excellent news! Shotgun desktop looks amazing, something I've really been missing. I think it's absolutely the right approach to provide a login and launch interface like Shotgun desktop and I don't see any potential problems with having the permissions align with a given shotgun user, other than taken it into consideration when writing the apps.

    Does Shotgun Desktop also support updating the configuration and apps? Like checking for updates to a projects git repo and pull the changes?

    /Nils

     

  • 0
    Avatar
    Ryan Mayeda

    HI Nils.

    We definitely plan on Shotgun Desktop having the ability to update both the project configuration and the installed components (i.e. the Toolkit core, engines, apps, frameworks and of course the Desktop app itself).  But, it's not in the pilot build yet.  Our sense is that the feature to update the installed components will come first since we already have the "app store" mechanism to check for and distribute new releases of stuff.  With the configurations, we have to do a bit of homework on how those will be shared that we're hoping to share on the forums and/or in a webinar soon.  A Git repo, as you describe, has been discussed before so it's encouraging to hear you say that...!  :)

    Ryan.

  • 0
    Avatar
    denisTassenoy

    Hello Ryan,

    We're working with multiple studio on a full 3d feature film. I try to find the best way to share my master configuration to other studios.

    Currently, I've pushed all config files to a github repo and I'm wondering what's the next move. I tried to clone pipeline configuration but It's based on my local files. Then I cloned the git repo on a external local server (to testing) but there's no way to add a pipeline configuration and browse to the right local directory. Finally, I launch tank setup project and take git repo as [tk default_configuration] but I can't assign a new pipeline configuration to this new local copy. Do you have the right way to solve that issue?

    Thanks

     

    Denis

     

  • 0
    Avatar
    James

    At present Denis the way it appears is you have to make sure you replicate the setup pretty much exactly at each point.

    Essentially you're setting up a studio at each location.
    To that end your primary volumes should match, as should any mappings etc for your software/studio and software/config folders

    I've used github to sync the base configs but when I tried to add studio it was too big to sync and I really need to get more time in the day to rectify that aspect as it helps with base application updates etc.

    Anyway to that end what we have going on here is:

    Primary: I:\
    Secondary: K:\
    our dev/tank drive is T:\software

    So the studio and the configs reside in T just to keep them cleanly away from prod servers etc
    From memory I just downloaded and setup the first tank, then I githubed the configs to the repo (not studio atm) and made sure those git repos were directly in the T:\software
    Then from there I zipped up the studio folder and transferred that over to the new location
    Setup I: K: T: drives over there, put the updated studio into T:\software, then pull the repos into the T:\software correctly so the latest configs can be pulled directly to the servers on release

    we are working in a fairly basic setup so each dev artist works on a local T:\software setup (which unfortunately means they have to copy their studio folder from the server at this time to a local drive that is mapped at T on their dev boxes) but this means they can grab the repos from git hub and work locally and push to T:\software\configName_sandbox for testing. When we're all happy that the latest configName_sandbox is working okay and everyone has commited to the gihub repo, we can 
    a) Pull to the server production sandbox for testing or
    b) copy the sandbox over to the release config, clean up any pathing requirements etc and then release sandbox to the primary configuration and commit to github and pull down to the main servers T:\software\primaryconfig
    All locations are using the same pipeline configurations and script keys from the base project as well so it's pretty clean there too.


    This way anytime we need to roll out a release, as long as the base setup is done, once we commit a release to the main T:\software\primaryconfig all we have to do then is pull the repo to the main T:\ server drives at each location to remain in sync.
    The drama comes when you have to update tank core. And it runs through all the apps and alters all ya hooks etc :P But it's manageable.

    Anyways I figured I'd do a little dump in here just in case it helps ya :P

  • 0
    Avatar
    Ryan Mayeda

    Hi Denis and James!

    Very interesting stuff here!  I have a few follow up questions for both of you...

    Denis, are you sharing the same Shotgun site with the studios that you're partnering with on the feature film or does studio (or at least some of the studios) have their own Shotgun site(s)?  For the publish data itself, are you structuring things like James is, where all the different partners can see the same primary storage so their publishes go right to a consistent spot everyone can access them from?

    James, do you think it's a bad thing that the studio directory ends up being localized for everyone?  By that I mean, what if it was just easier to do that in a built-in way?

    One of the things we're excited about with Shotgun Desktop is the new ability it grants us to manage where configurations and installs live, potentially making it easier to localize them in a consistent way and place (ex. /Applications/Shotgun on Mac, C:\Program Files\Shotgun on Windows, etc.).  While this won't all be in the initial release of Shotgun Desktop, we're starting to think through the planning and design for this feature set and we touched on it a bit in the last webinar (around the 28:10 mark):

    https://support.shotgunsoftware.com/entries/95443087

    The idea here is that we can expand on the current install options to also support all-local setups as well as hybrid setups where some users are localized and others (maybe those who actually work in the office) use a shared setup.  The attached framegrab is the most relevant slide from that part of the webinar, showing how each user would have Shotgun Desktop, but some would point to the shared install and others would have their own install, with each sharing the same main data set (although this could also potentially be move-able as well).

    The current idea we have to try and better support what it sounds like James is doing would be to help share configurations, possibly through Github (as Denis is doing) or through Shotgun itself (or perhaps both can be options?).  Shotgun Desktop could help facilitate it, automatically grabbing any updates as they happen.  But, we'd only share the configuration, and any of the installed components (core/engines/apps/frameworks) would then be pulled down separately and dynamically as needed.  This way, we wouldn't need to always store a big mass of data (aka the studio folder), which can be a bear.

    Anyhow, hearing about what you guys are trying now and how you'd like things to work is great for us.  It really helps refine our design work, hopefully we'll have more stuff to share for feedback (or even testing) soon.

    Ryan.

  • 0
    Avatar
    James

    >>James, do you think it's a bad thing that the studio directory ends up being localized for everyone? By that I mean, what if it was just easier to do that in a >>built-in way?
    Yeah it can be a bad thing. Because the configs rely on applications from the app store, and updating those at each hub then runs through the configs at each hub and alters all the shot_step.ymls etc as necessary, the issue I found with this is that each location then commits those changes to github and it can potentially create issues with the repos.
    Studio I'm not githubbing atm due to the size limit and I need to look into this further when I get some time. But I think this would semi avoid that issue if I could push the studio apps through github and then the configs from one location.
    So essentially I do a core update / application update remote, then transfer any new applicatiosn manually to the other locations, do a local tank core update *no apps* then commit and push the one config from my location out to everyone else and then check to make sure they're all working okay with the latest applications, then do the release from there after problem solves etc.


    >>One of the things we're excited about with Shotgun Desktop is the new ability it grants us <SNIP>(core/engines/apps/frameworks) would then be pulled >>down separately and dynamically as needed. This way, we wouldn't need to always store a big mass of data (aka the studio folder), which can be a >>bear.

    Okay I've attached a couple of images as a means of a brain storm / discussion. The Current.png is the way I kinda see it atm, the Anticipated.png is my brain dump just ignoring any mitigating factors for now..

    I can def spend more time cleaning these up and making them clearer so apologies if they're a little obscure or not as thorough as poss, under the pump and all that atm..   :)

     

    cheers

  • 0
    Avatar
    Nils Lerin

    Ryan, will Shotgun Desktop be available on github in any sort of unofficial and unsupported way before you release it more officially? The things I'm developing at the moment would be very dependent on how shotgun desktop is working and it seems your test pilot pool is full :(

    Cheers

  • 0
    Avatar
    Bernhard Kimbacher

    I'm looking at the same kinda scenario, but maybe in a slightly simpler form:

    I'm thinking more in terms of remote artists and not so much sharing work/data between studios. I would imagine an artist to VPN into the studio and thus having access to all the data (and having all the behaviour as a internal workstation). In order to actually make it workable over the web I was imagining a 'localize' option for assets which would download the selected assets to the local disk.

    I could see the same functionality being useful within the studio environment (and have seen it at some studio pipelines) so you're less affected by network slowdowns...

    Has there any progress being made on the endeavours mentioned in your previous posts?

    -Bernie

     

  • 0
    Avatar
    Andriy Toloshnyy

    Hi guys!

    Just one question about the project root with the localized configuration - shall it be in the same place as defined in the website configuration or there's a way to customize it's location for an outsource work?

    Andriy

  • 0
    Avatar
    Abraham Levi Borbujo Carrasco
    Hi.
    I'm very interested in this case.
    Have you guys made any advance?

    Abraham
  • 0
    Avatar
    Atle

    We're looking at the same setup as Bernie. External user that has full access to our internal servers, but for speed improvements would like to work on files locally and the publish them to the main fileserver.

    -atle

  • 0
    Avatar
    Steven Quinones

    Anything new on this?

댓글을 남기려면 로그인하세요.