Using Python 'requests' library with a custom SSL issuer certificate
Recently, my project manager asked me to create a small web service
and gave me an SSL certificate that each response should return
to allow client to verify the endpoint. We’ll refer to this certificate as the primary certificate.
He obtained this certificate from a well-known certificate authority we’ll just refer to here as “CA”.
Every SSL certificate contains an issuer field that points to another certificate needed to verify the primary certificate. Often, the issuer certificate is a root certificate, a self-signed certificate issued by a known and trusted certificate authority. Popular web browsers may be shipped with their own collections of trusted root certificates, or they may depend on a collection of certificates stored in a restricted directory in the local filesystem.
When my project manager requested the certicate from the CA customer web form, the form asked whether the certificate was intented to be used only within our company’s domain, or in the wild. He checked off that it would be used only within the company’s domain. In response, the CA mailed him a primary certificate to be returned by each server response and an issuer certificate to be distrubed to clients using the web service for verifying the primary certificate returned by each response. We’ll refer to this certificate as Issuer.crt.
I created the web server with unit tests, then wrote a small client application in Python with example requests. The Python library for HTTP requests is called (drum roll …) requests. requests depends on the certifi library that bundles a collection of root certificates in one file. Each HTTP request takes an optional named parameter, verify=, that allows me to provide it with an alternate directory or file of root certificates. requests also provides a Session object that can be configured with parameters including verify, then re-used for multiple requests.
When my example does:
Some browsers and http libraries will use the Authority Information Access (AIA) Extension. to attempt to resolve references to issuer certificates not already pre-installed in their collection by downloading them dynamically. Note that this introduces yet a new security vulnerability. Each new web request and response must itself be verified with SSL. This can result in a circular chain that can only be stopped by a timeout.
The authors of the Python requests library chose not make use of the AIA extension. If requests can’t resolve the chain of trust with whatever certificate collection it has been given, it will throw an SSLError exception.
The only way I could get the requests library to successfully use my web service was to append Issuer.crt to the collection of root certificates provided by certifi. This is not a permanent solution, as it would require every user of my web service to do the same, and the same modification would have to be made again each time a user updates the certifi library
on his local machine.
For due diligence, I wrote another example client app in Perl that send requests to the same web service with the Perl Rest::Client library. It had no problem completing HTTP requests to my web service. Nevertheless, my project manager and others who may use this web service are already using Python and I see no reason they should not be able to do all their work with whatever scripting language they prefer.
In response, I created the cert_session Python package.
and gave me an SSL certificate that each response should return
to allow client to verify the endpoint. We’ll refer to this certificate as the primary certificate.
He obtained this certificate from a well-known certificate authority we’ll just refer to here as “CA”.
Every SSL certificate contains an issuer field that points to another certificate needed to verify the primary certificate. Often, the issuer certificate is a root certificate, a self-signed certificate issued by a known and trusted certificate authority. Popular web browsers may be shipped with their own collections of trusted root certificates, or they may depend on a collection of certificates stored in a restricted directory in the local filesystem.
When my project manager requested the certicate from the CA customer web form, the form asked whether the certificate was intented to be used only within our company’s domain, or in the wild. He checked off that it would be used only within the company’s domain. In response, the CA mailed him a primary certificate to be returned by each server response and an issuer certificate to be distrubed to clients using the web service for verifying the primary certificate returned by each response. We’ll refer to this certificate as Issuer.crt.
I created the web server with unit tests, then wrote a small client application in Python with example requests. The Python library for HTTP requests is called (drum roll …) requests. requests depends on the certifi library that bundles a collection of root certificates in one file. Each HTTP request takes an optional named parameter, verify=, that allows me to provide it with an alternate directory or file of root certificates. requests also provides a Session object that can be configured with parameters including verify, then re-used for multiple requests.
When my example does:
s = requests.Session()
s.verify = Issuer.crt
s.get('https://my-web-service')
Some browsers and http libraries will use the Authority Information Access (AIA) Extension. to attempt to resolve references to issuer certificates not already pre-installed in their collection by downloading them dynamically. Note that this introduces yet a new security vulnerability. Each new web request and response must itself be verified with SSL. This can result in a circular chain that can only be stopped by a timeout.
The authors of the Python requests library chose not make use of the AIA extension. If requests can’t resolve the chain of trust with whatever certificate collection it has been given, it will throw an SSLError exception.
The only way I could get the requests library to successfully use my web service was to append Issuer.crt to the collection of root certificates provided by certifi. This is not a permanent solution, as it would require every user of my web service to do the same, and the same modification would have to be made again each time a user updates the certifi library
on his local machine.
For due diligence, I wrote another example client app in Perl that send requests to the same web service with the Perl Rest::Client library. It had no problem completing HTTP requests to my web service. Nevertheless, my project manager and others who may use this web service are already using Python and I see no reason they should not be able to do all their work with whatever scripting language they prefer.
In response, I created the cert_session Python package.
Comments
Post a Comment