Fixing the SSL: CERTIFICATE_VERIFY_FAILED issues with the Python API

The Python API relies on a list of certificates that is bundled with the API and on your machine in order to connect to the various webservices Shotgun uses. Unfortunately, new certificate authorities can be released and those might not be bundled with the Python API or OS.

While our Python API comes with a very recent copy of the certificates, as of February 21st 2019, there's a bug right now that prevents the API from using those certificates for Amazon S3 uploads, even if you are using the latest version of the API. For background please see this AWS blog post. To remediate the situation temporarily, you can try the following solutions. Note that these are temporary workarounds and we're looking into a long-term solution.

Preferred Solution

Add required CA certificate to the Windows Certificate Store. Windows 7 users may have to first upgrade to PowerShell 3.0 in order to use this solution, or alternatively use certutil to add the required certificate.

  1. Start an elevated PowerShell by right-clicking Start and then left-clicking Windows PowerShell (Admin)
  2. Paste the following commands into the PowerShell window and then press Return to execute:
    $cert_url = "https://www.amazontrust.com/repository/SFSRootCAG2.cer"
    $cert_file = New-TemporaryFile
    Invoke-WebRequest -Uri $cert_url -UseBasicParsing -OutFile $cert_file.FullName
    Import-Certificate -FilePath $cert_file.FullName -CertStoreLocation Cert:\LocalMachine\Root
  3. If details of the added certificate bearing thumbprint 925A8F8D2C6D04E0665F596AFF22D863E8256F3F are displayed then the operation is complete and PowerShell can be closed

Alternative Solutions

If you are using the Python API only

1. Upgrade to the Python API v3.0.39

2a. Set SHOTGUN_API_CACERTS to /path/to/shotgun_api3/lib/httplib2/cacerts.txt

or

2b. Update your scripts and set the ca_certs=/path/to/shotgun_api3/lib/httplib2/cacerts.txt when instantiating the Shotgun object.

If you are using Toolkit

1. Upgrade to the latest version of the Toolkit API via the tank core command or by updating
the core/core_api.yml file of your pipeline configuration, depending on how you deploy Toolkit.

2. Download an up-to-date list of certificates at https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem

3. Set SHOTGUN_API_CACERTS to the location where you saved this file. Toolkit doesn't allow you to specify the ca_certs parameter when creating connections unfortunately like the Python API does.

If you can’t update the Python API or Toolkit

1. Download an up-to-date list of certificates at https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem

2. Set the SSL_CERT_FILE environment variable to the location where you saved this file.

Follow

7 Comments

  • 0
    Avatar
    Janice Barlow Collier

    I'm fairly sure my facility isn't experiencing this problem, but is there some means of testing it, same as we've been given test scripts for TLS and other Shotgun issues in the past?

    Also, assuming one's Shotgun instance did have this problem, are the steps listed above a mitigation until an upcoming Shotgun Desktop or API release, or are they intended to be permanent?

  • 0
    Avatar
    Jean-François Boismenu

    Hi Janice!

    No, it's meant to be temporary. We're looking into what is the best solution for us and our clients moving forward.

    I've updated the article to reflect that fact.

    Thanks for voicing your concern! :)

    Edited by Jean-François Boismenu
  • 0
    Avatar
    Ben Rhoades

    I followed both of the suggestions for fixing this issue when using the Shotgun Python API, and I'm still getting the cert verification error.

    If it helps, I'm using Windows 10, Python 2.7.14.

  • 0
    Avatar
    Scott Ballard
    Deadine 10.0 is distributing its own shotgun_api3 (3.0.35) in DeadlineRepository10\pythonsync\pythonsync.zip which also gets copied to the local render node under C:\Program Files\Thinkbox\Deadline10\bin\pythonsync.
     
    I replaced it with shotgun_api3 (3.0.39) and was able to upload to Shotgun through Deadline. I had to copy the shotgun_api3 to each render node, I wasn't sure how to get DL to sync them automatically.
     
    This was in addition to the recommendations above.
  • 1
    Avatar
    Christian Korneck

    Just to give some background (as this problem isn't specific to Shotgun): There's a common misunderstanding how SSL with Python works:

    - how Microsoft intends SSL on Windows to be used: There's the Windows certstore. On a "vanilla" Windows 10-1809 installation it contains *only* 12 essential RootCA certs (compared to ca. 134 in the Mozilla certstore / certifi). Then there's the Windows SSL library called SCHANNEL (the Windows equivalent to OpenSSL). Whenever an application makes a TLS/SSL call via SCHNANNEL (i.e. Internet Explorer) and the required RootCA cert isn't in the Windows certstore yet, the missing RootCA cert is automatically and on the fly downloaded from an online Microsoft server. The cert is then permanently stored in the Windows certstore (This is often referred to as "certificates via Windows Update" as it uses the same transfer system).

    - python "urllib2" / "urllib" (and others) : Uses Python's stdlib "ssl". This lib is using the Windows certstore in a non-ideal way: It queries the RootCA certs from the Windows certstore, but uses a bundled copy of openssl to do the actual TLS/SSL call (and not the native Windows SCHANNEL lib - which would be necessary to really get all RootCA certs as intended from Microsoft). So on a "vanilla" Win10 system, the Python "ssl" lib only trusts 12 RootCAs, which is not enough. Even commonly trusted and highly correctly set-up websites like https://google.com oder https://verisign.com are not trusted by the Python "ssl" lib. They only get trusted once the RootCA certs have been placed into the Windows certstore by other processes, i.e. Internet Explorer, Chrome, etc.

    -  python "requests" lib, pip (and some others): These libs bring their own bundle of trusted RootCAs (usually "certifi", based on Mozilla data).

     

    To get Python's "ssl" lib (and thus everything that uses urllib2 / urllib) reliably work on Windows, I would recommend to deploy the cacets from certifi ( https://certifi.io ) to the Windows RootCA store. That's conveniently possible on a large amount of machines with an Active Directory group policy.

    To do so, download the certifi CA bundle and convert it to a pfx file:

    wget -O certs.pem https://mkcert.org/generate/
    openssl pkcs12 -export -nokeys -out certs.pfx -in certs.pem

    Then create an AD GPO (Computer Config -> Windows Settings -> Security Settings -> Public Key Policies -> Trusted Root Certification Authorities) and import the PFX via the GPO GUI. This will import all certs at once.

    (For future updates, "certifi" has a security updates newsletter).

    Edited by Christian Korneck
  • 0
    Avatar
    Shervin Shahidi
    $cert_file = New-TemporaryFile

    On Win7, some of the commands, like the temporaryfile doesn't work in PowerShell v4.0 (and below). After updating to PowerShell v6, the Import-Certificate still doesn't work on win7.

    Edited by Shervin Shahidi
  • 0
    Avatar
    Christian Korneck

    @Shervin - I believe the above Powershell syntax only works with Powershell 3 and later. Win7 only contains PS 2 out of the box (unless you have updated the Windows Management Framework).

    To do the same in "pure Batch" (should work on a "vanilla" Win7 system in a .cmd file or cmd shell):

    @echo off
    setlocal
    bitsadmin.exe /transfer "%random%%random%%random%" https://www.amazontrust.com/repository/SFSRootCAG2.cer "%tmp%\SFSRootCAG2.crt"
    CERTUTIL -addstore -enterprise -f -v root "%tmp%\SFSRootCAG2.crt"
    exit /B %errorlevel%
    Edited by Christian Korneck
Please sign in to leave a comment.