At my last project, I wrote a small Python utility program to create and delete secrets in Hashicorp Vault. This tool was executed inside a docker container (and the container ran in a CI-pipeline). The container also talked to some other services that use an internal CA that was added to the system-CAs (that will be important shortly).
Back to Vault: there exists a great library called hvac that lets you interact with Vault from Python.
So I wrote the program, COPY’d it into the container, installed the hvac-library inside the container and let the program run. Vault secrets got created and deleted as needed, everything was fine. Except now the connection to the others service with the internal CA no longer worked - “failed to verify TLS connection”.
After verifying that the connection still worked outside the container I removed my program and the hvac-library from the container and the connection to the service was working again! Then I reinstalled the hvac-library and the connection stopped working. Clearly the library must be doing something strange with the CAs.
I then noticed that when I installed hvac, the certifi-library also got installed (as an optional dependency of the requests-library). And certifi “provides Mozilla’s carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts”. It does so by overriding all other CAs, thus the connection is no longer trusted. See here, here, here or here for similar problems, OR certifis documentation. :)
So what did I do to fix this problem? Easy: I installed hvac without its dependencies in the Containerfile:
RUN pip install --no-cache-dir --no-deps hvac
Note the --no-deps option from pip, that tells it to not automatically install any dependencies of the package.
This way, certifi was not installed and it could not change my system-CAs. Now creating secrets in Vault worked and talking to other services also worked.