PKI Infrastructure

Automated Certificate PKI Infrastructure In A Multi Region / Multi Cluster Istio Service Mesh

Intro

Public key infrastructure (PKI) is defined as a collection of tools and procedures for managing digital certificates and public keys. PKI is used to secure data transfers over the internet, and is built into most web browsers. Components of PKI include:

  • Certificate authority (CA) - A trusted third party that issues digital certificates. CAs also validate the identities of certificate holders.
  • Registration authority (RA) - Verifies the identity of the user or device requesting a digital certificate.
  • Digital certificates - Electronic credentials that establish trust.
  • Private keys - Unique keys that are used to decrypt messages. Only the owner of the private key knows it.
  • Certificate revocation list (CRL) - A list of certificates that have been revoked before their expiration date.

Back in 2022, as a Lead DevOps Engineer I was assigned as Tech Lead for an ambitious project to build and implement an Istio Service Mesh within our very large, enterprise scale multi-region Kubernetes Ecosystem. Along with the challenges of designing the high and low level architecture of the Service Mesh and integration within our Kubernetes ecosystem, the network ingress and egress traffic networking, overall network topology, network policies and access controls, and a redesign / reconfiguration of deployments within our CI/CD pipeline, came the pivotal difficulty of security. More specifically, how to implement mTLS across the entire Service Mesh.

The strategic challenge was not only how to implement mTLS, but automate it using Cert-Manager, which already existed and was being used in our clusters. For such a large Kubernetes Ecosystem, and for a multi-region / multi-cluster Service Mesh, this had never been done before. At the time, there were many examples, and many companies using Istio Service Mesh in production, but none pushing the technical envelope in this way. In Istio's documentation, only examples configuring mTLS for a single cluster were given (with manual certificate generation and renewal). Additionally, across the tech industry, incorporating Cert-Manager for automated certificate generation and rotation was only ever used on a single cluster Service Mesh.

So, the substantial undertaking was set. Building an industry first - a fully automated PKI infrastructure across a multi-region / multi-cluster Service Mesh, using an external cloud provider (GCP) for a Certificate Authority and to store a root certificate, using Vault as an Intermediate Certificate Authority and Vault's built-in PKI Infrastructure, and connecting that to Cert-Manager in each of the clusters, establishing a chain of trust that would be provided to Istiod (the Istio Control Plane) where it would finally act as another Intermediate CA, and mint workload certificates to inject into the sidecars running alongside every Pod, to establish mTLS connections Mesh wide.

The final architectural design and technical implementation. which I will go into more detail in this blog post, was ultimately done as follows:

  • Root Certificate Stored in Google CAS (Certificate Authority Service).
  • Intermediate Certificate Authority and Certificate Stored in Google CAS (With all Clusters in the Service Mesh in the CA Pool).
  • Intermediate Certificate Authority and Certificate Using Vault and Vault's Built-in PKI Infrastructure.
  • Cert-Manager Managed Cluster Certificate Minted from the Vault Intermediate CA, creating cacerts for Istiod.
  • Short-lived Workload Certificate Minted from Istiod Injected into Sidecars Fully Establishing A Chain of Trust and mTLS.

Istio CA Certificates

Before we discuss the details of the PKI Infrastructure, it is important to understand how this aspect of Istio works. To run an Istio Service Mesh across clusters, you must establish trust between workloads across cluster boundaries. This is where the cacerts comes in.

In Istio, "cacerts" refers to the collection of trusted root certificates used by the Istio service mesh to verify the identity of workloads within the mesh, essentially acting as a "certificate authority" (CA) where the Istio control plane generates self-signed root certificates to sign and trust other certificates within the mesh, enabling secure communication between microservices through mutual TLS (mTLS). These certificates are crucial for establishing trust between different services and components within the Istio mesh:

  • Root CA Certificate: Istio uses a root CA certificate to sign the certificates for all sidecars (Envoy proxies) and services within the mesh. This ensures that every communication channel is authenticated and encrypted.
  • Service Identity: Each service in the Istio mesh has a unique identity based on its certificate. This identity is used to enforce security policies such as access control and mutual TLS (mTLS) between services.
  • Trust Establishment: The cacerts in Istio includes the root CA certificate and any intermediate CA certificates that are needed to establish trust between services. This allows Envoy proxies and services to verify the authenticity of each other's certificates.
  • Security Controls: By managing cacerts, Istio provides administrators with the ability to configure and enforce security policies across the mesh, ensuring that only trusted services can communicate with each other.
In summary, cacerts in Istio refers to the set of CA certificates used for secure communication and identity verification within the service mesh, playing a critical role in establishing and maintaining trust between services.

There are (2) behaviors depending on architectural setup to discuss:

  • Default Behavior - First, the default behavior which you get out of the box with Istio.
  • "Productionalized" Setup - Second, the technical implementation we ultimately built, which is a more complex but more secure "productionalized" setup, of using an external certificate authority.

Istio CA Certificates

Default Behavior

Bootstrapping

The default behavior for the Service Mesh is letting Istio generate an intermediate CA. This is also referred to as "bootstrapping" the Istio installation. Bootstrapping of workload identity is how workloads get a signed certificate that proves their identity. For simplicity, using this default method Istio generates a CA to sign the certificates upon installation.

As the root CA, istiod uses the private root CA key to sign and issue leaf certificates to the workloads in the service mesh. The leaf certificates encode a workload’s identity and can therefore be used to enforce authentication and secure communication between workloads via mutual TLS (mTLS).

This generated CA is stored as a secret named istio-ca-secret in the Istio installation namespace (istio-system) and is shared with istiod replicas. Istio then stores the CA certificates in a secret named cacerts in the installation namespace istio-system, containing the following data:

  • ca-cert.pem — The intermediate CA’s certificate.
  • ca-key.pem — The intermediate CA’s private key.
  • root-cert.pem — The root CA’s certificate that issued the intermediate CA. The root CA validates the certificates issued by any of its intermediate CAs, which is key for mutual trust across clusters..
  • cert-chain.pem — The concatenation of the intermediate CA’s certificate and the root CA certificate that forms the trust chain.

Because all of the leaf certificates share the same root, trust can be established between the workloads in the service mesh. Similarly, if the root CA certificate is rotated, and the change is not reflected in all of the leaf certificates, communication between workloads starts to break until all leaf certificates are updated with the new root CA certificate.

Using an External Certificate Authority

Below is a list that shows all of the components built to enable Certificate management using an external CA. This is a good visual of all of the major and minor components that had to be integrated into this overall architectural solution, to work together to enable the desired outcome. Each list item plays an integral part, and when configured correctly yields an extremely reliable completely hands-off pipeline that secures the Service Mesh with mTLS certificates.

External Certificate Authority and Vault as External Intermediate CA

  • First - within Google Certificate Authority Service (CAS) a CA Pool, a Root CA, and Intermediate CA were created.
  • Second - a Vault mount and PKI path were created, which will utilize Vault's functionality to act an an Intermediate CA. On this mount path, permissions are set, the Kubernetes Auth Method is set, and the PemBundle from the GCP Intermediate CA Certificate is uploaded.
  • Third - Cert-Manager (running in each one of the Kubernetes Clusters) issues Certificate Requests (CR), requesting a Certificate. Cert-Manager uses a Cluster Issuer which is tied to a Service Account that specifically has the permissions to access the Vault mount path and secret (the Intermediate CA Cert) for signing and issuing Certificates.
  • Fourth - upon successful signing of that Certificate, the cacerts Secret is created.
  • Fifth - Istiod automatically pick up this Secret, and from the cacerts creates and issues its own workload certificates, which are updated in the ValidatingWebhookConfiguration, which injects the pemBundle into the Sidecars.
  • mTLs is established as all certificates have a common root of trust.

Below I've listed each component, as well as the sub components that relate, to try and provide a better visualization of the entire set up.
  • Google Certificate Authority Service (CAS)
  • CA Pool
  • Certificate Authority (In Pool)
  • Root CA Certificate
  • Intermediate CA Certificate
  • Vault
  • PKI Vault Mount (Path)
  • PKI Backend Secret Config
  • Vault Backend Role and Policy
  • Kubernetes Auth and Backend Config
  • Intermediate CA Certificate
  • Cert-Manager
  • CSR (Certificate Signing Request) / CR (Certificate Request)
  • Challenge and Order
  • Certificate
  • Secret (cacerts in istio-system Namespace)
  • Istiod (Pilot)
  • CSR (Certificate Signing Request) To Workloads
  • Injects CaBundle in ValidatingWebhookConfiguration which will establish common trust
  • Workload (Sidecar)
  • Holds CaBundle from ValidatingWebhookConfiguration
  • Intercepts all Ingress traffic and validates cert and trust chain for mTLS

Extra Notes / Recap

The root certificate lives in Google Certificate Authority Service (CAS), in which we created a Root CA and a CA Pool for the Certificates to live in. From the Root CA, we create a Google PrivateCA Certificate that will act as an Intermediate CA Certificate also in Google CAS.

For multi-cluster traffic, you can establish a shared root of trust by using a single root CA and an intermediate CA that is signed by the same root CA for each cluster. Enabling this functionality in our Clusters configures the entire Mesh to use Vault as an Intermediate CA, and Root CA in Google CAS. Then generate an Intermediate CA Certificate to be used by Istio on each cluster to verify and sign its workload certificates. To allow communication between workloads on different clusters, you must ensure that all workloads share the same root certificate and certificate chain so that trust can be established across clusters. Instead of treating istiod as the root CA, you can use your preferred private key infrastructure (PKI) provider to derive intermediate certificates from the root certificate. The intermediate certificate and key are then stored in the cacert Kubernetes secret and configure istiod as an intermediate CA.

Overriding the Default Behavior

The default behavior can be overridden by plugging in our CA, which the Istio CA picks up instead of generating a new one. To do so, we have to store the CA certificates as a secret named cacerts in the installation namespace istio-system, containing the same data as the default behavior. The Istio deployment in each workload cluster requires a certificate authority (CA) certificate in the cacerts Kubernetes secret in the istio-system namespace.

Challenges

It is important to note - Istio is very particular in how it wants the cacerts Secret to be formatted. By default, (at the time) it only accepted the format of the Secret containing:
  • ca-cert.pem
  • ca-key.pem
  • root-cert.pem
  • cert-chain.pem

This format is problematic due to the way Cert-Manager creates Secrets from approved and signed Certificates. Cert-Manager will create the Secret (with pem bundle) with ca.crt, tls.crt, and tls.key. I was able to work directly with Istio maintainers to address this issue and resolve it. This was a significant roadblock, as without a fix for this, I would have had to build a brittle work around for Istio to accept the Certificates. Luckily, in working with the Istio maintainers, we were able to put in a patch for this and release a new version of Istio fairly quickly.

With this new release of Istio, Istiod (Pilot) now had the ability to accept the format Cert-Manager creates the Secret and places the crt and keys in. With the fix, Istio would now accept Certificate data in the format that Cert-Manager creates it in, and places it in Secret resource(s) in the cluster:

  • ca.crt - The intermediate CA’s certificate.
  • tls.crt - The concatenation of the intermediate CA’s certificate and the root CA certificate that forms the trust chain.
  • tls.key - The intermediate CA’s private key.

Summary

Without getting into too much detail, as the technical details are proprietary, below is a diagram to visualize the finalized architectural design of the custom PKI Infrastructure. There are expanded views of each section provided for improved visibility if you wish to zoom in on a particular section.

Written: January 27, 2025
BACK HOME