Diagnosing performance issues with hosted Shotgun sites

This page gives information about how to troubleshoot performance issues for hosted Shotgun sites. For urgent issues that requires immediate support, please open a ticket on Shotgun Support.

Types of performance issues

It can be difficult to identify the cause of a slow Shotgun site since there are many areas that could potentially have performance issues. In order to help our support team diagnose, we need to understand where you are seeing issues. Common areas where people might experience slowness include:

  1. Loading data across the whole of Shotgun: Where navigating across all areas of the UI seems slow.
  2. Loading specific pages in Shotgun: Where a single page or pages are noticeably slower to load than the rest.
  3. Playing back media in the Overlay Player or Screening Room: Where streaming media takes a long time buffer or you encounter stuttering during playback.
  4. Uploading and transcoding of media: Where uploading versions and media to Shotgun takes a long time or there is increased time between when the Version is uploaded and the transcoding is complete.
  5. Slowdowns at a specific time of the day: Where Shotgun performance across any of the above areas is noticeably slow during certain time periods, on a consistent basis, every day.
  6. Network wide issues that are not related to Shotgun: Where one of our services is experiencing network or connectivity issues which affects the Shotgun service.

A good first point of reference is to identify which of the above areas you are seeing slowness in, or any other areas not listed above. Then you can run some recommended tests, which we’ve outlined below, and send the relevant results to our support team.

What are 503 errors?

If more requests are made than the system can process, the request queue fills up and eventually, requests are rejected. For the end users, this is reflected by a 503 error, misleadingly letting users think the site is in maintenance. Should you encounter these, please contact support as soon as possible.

Diagnosing issues

The guide below will help you validate where you are seeing performance issues across Shotgun and help you provide our support team with the most relevant information.

Step 1: Validate Shotgun outages

The first step is to validate that Shotgun is not undergoing any maintenance, or reported downtime. You can check this via the Shotgun status page. If all systems are reporting as operational, then continue to the next step. Otherwise you can read a report on the type of incident.

Shotgun status page

Step 2: Validate network connection to Shotgun services

The repository at https://github.com/shotgunsoftware/shotgun-toolbox contains various tools that can be useful for hosted Shotgun Administrators. If you are experiencing any performance related issues, it is good to run the above scripts and send the output to our support team. The script will run multiple tests such as speedtests, ping, and traceroutes to all Shotgun end-points (load balancer, S3, CDNetworks).

For more validation tools, please try http://downdetector.com/status/level3 and https://www.dynatrace.com/capabilities/digital-experience-monitoring.

Step 3: Identify specific problem areas

Slowness on specific pages

If you are seeing slowness on specific pages, the first thing to do is send our support team the URL address of the page. Individual pages can often perform slower if they haven’t been optimized properly. Once a problematic page has been identified, you can use our guide on optimizing filters and scripts to make your pages faster.

Note: It is NOT normal for Shotgun pages to feel slow. If you notice slowness please contact our support.

Using the debug console

A browser's debug console can help identify which processes might be slowing down a page. To run this test in Chrome:

  1. Go to View > Developer > Javascript console.
  2. Select the Network tab, on the far left of the UI.
  3. Navigate to one of the slow pages.
    • You will see a number of calls to the server begin to appear. There may be one (or more) taking a very long time.
  4. Click the offenders, and then expand the height of the frame and either include that in your video, or take a screenshot of the header information and send it to us.

Slowness when playing back media

Hit the Amazon S3 link directly

If media is buffering or running slowly in the Overlay Player or Screening Room, then you can try the following test that accesses the media from S3 directly:

  1. Open up your browser’s debugger.
  2. Go to the Network tab.
  3. Begin playback of a version in the Overlay Player.
    • You’ll notice a load of calls being made, with one of them being the actual media file. If you inspect it you’ll get a direct S3 link to that piece of media.
  4. Copy that link and paste it into your browser, which should play the media directly.

Using this link essentially bypasses the Shotgun Overlay Player and talks to the server directly. If speeds are vastly different when playing off the server directly, then something in Shotgun could be slowing it down, but if speeds are more or less the same then it could be something on the network interfering.

You can also test the speed of AWS services with https://cloudharmony.com/speedtest-for-aws. Please note this test lasts 30 minutes.

Mobile network test

Since Shotgun doesn’t require a powerful connection to run smoothly, it can be run from a mobile data network. If your network allows for it, you can tether to a mobile 4G data network temporarily on a machine, and try running the Overlay Player on there.

Slowness when uploading and transcoding media

The first thing to validate is that your pure upload bandwidth speed is good. The results of the speed test above can be used to determine this. It is important to understand that if you have a 1 MBPS upload bandwidth, it will take about 14 minutes to upload a 100 MB file.

If you are experiencing slowness when uploading or transcoding media, it is important for our team to understand what type of media you are trying to upload. If you can, when submitting a support ticket, please include the specifications of the file you are trying to upload:

  1. Frame size
  2. Codec and container
  3. Frame rate
  4. File size in MB

If the problem persists, you can supply the problematic file to our support team to investigate for you.

Slowness reported at certain times of the day

While the above sections will help us narrow down where slowness might be coming from, understanding if it happens at a certain time of day can help us understand what process might be causing this.

Things to look out for include:

  • Are any big processes happening at that time of the day? For example, deliveries or renders at the end of the day.
  • Are any large scripts run at a certain time each day? This could have an impact based on what the script is doing.
  • Are the timestamps when the slowness is noticed consistent?

Please include this information to our support team to investigate.

Follow

0 Comments

Please sign in to leave a comment.