Home Blog Automate k8s Secrets Creation Using Terraform and ExternalSecrets

Automate k8s Secrets Creation Using Terraform and ExternalSecrets

By Netanel Koli

Cloud and DevOps engineers often try to automate creation and management of application secrets. Many options for the best or most secure way to create and keep secrets in a k8s cluster have been discussed. Whether it’s static or dynamic secrets creation, I always had a sense that the capability to automate the entire process was missing.

At Argus Cyber Security we use Google Kubernetes Engine as our Kubernetes cluster and Terraform as our Infrastructure as Code tool. Using ExternalSecrets Custom Resource Definition (CRD), we found a way to automate secret creation in a secure manner.

In short, the concept is to auto-generate a secret and keep it in GCP’s Secret Manager store. Then, the ExternalSecrets controller can retrieve the secret’s value and create a k8s secret to be used by another resource.

Tools used in this methodology:

Terraform (with GCP backend)
GKE cluster (with Workload Identity)
ExternalSecret controller
Helm3

All code blocks and instructions can be found in this GitHub repository: https://github.com/netanelkoli/automate-gke-secrets

Let’s start with some Terraform code…

The Terraform Code

The Terraform module is responsible for creating the secret in GCP Secret Manager. Either pre-defined or auto-generated, this module is responsible for the secret store in the cloud

Prerequisites:

Terraform is configured with a GCP provider.
A “variable “progect_id” {default = <projectID>}” declared.
Optional: it is preferable to configure Terraform with an encrypted GCS backend.

Let’s start by looking at the secret manager module.

Source: https://github.com/netanelkoli/automate-gke-secrets

It will be much more convenient if we create a module from this to be used in the following code:

What we actually see here is two .tf code blocks:

The first block is a secret creation Terraform module used to store a secret value in GCP Secret Manager.
The second block calls the first one. However, it has the option to auto-generate a value upon predefined configurations (using special chars, length of the value, etc.)

You may use the above code as is. However, we can also take advantage of Terraform’s modular approach and create another module wrapping both of them:

NOTE: It is highly recommended to use sensitive output in your outputs.tf file. In addition, since the auto generated password will be saved in your Terraform state file, it is also recommended to store it in an encrypted backend. See example here

Why is that recommendation important?

Because now, we can create a secrets map to allow multiple secrets created by Terraform. Let’s take as an example the following locals.tf file:

Here we are creating a map which the Terraform module from the previous step iterates upon, generating passwords and storing them on GCP secret manager.
Apply the Terraform code and make sure secrets were created in the GCP Secrets Manager console.

Terraform ExternalSecrets IAM Resources

We haven’t finished with Terraform just yet. We still need to prepare some IAM resources for ExternalSecrets to be used once we deploy the controller on the GKE cluster (the ExternalSecrets helm chart installation is documented below)

Source: https://github.com/netanelkoli/automate-gke-secrets

We are using some more Terraform modules, but to simplify things I added the actual resources list here:

Here we are creating a Workload identity binded IAM serviceAccount resource, which we will use later.

NOTE: it is assumed we will deploy the ExternalSecrets in a k8s namespace named “shared”.

Installing ExternalSecrets Using Helm

We are using helm3 for all our deployments. Therefore, it is natural we used the ExternalSecrets helm chart to deploy the controller to our cluster.

Prerequisites:

GKE cluster with workload identity enabled. The process of enabling it is not covered in this post.
Helm3 and kubectl installed.

Follow the instructions in the official documentation. Here, we will use workload identity to authenticate to the GCP secrets manager.

Source: https://github.com/netanelkoli/automate-gke-secrets

Create the following helm-values.yaml file (make sure to change the <projectID>)

Installation

Make sure controller is up:

Create Secrets from Secret Manager

We have created the secret values in the GCP secret manager and deployed the ExternalSecrets controller on our GKE cluster.

So how do we create an actual secret and use it?

Source: https://github.com/netanelkoli/automate-gke-secrets

Create a secrets.yaml file with following code:

Apply it to the GKE cluster:

kubectl apply -n my-namespace -f secrets.yaml

Let’s check if the secret was created successfully:

It works! We can see that k8s secrets are retrieved from the GCP Secret Manager!

Wrap Up

This post describes how Argus handles secrets management using different cloud technologies. In a nutshell, we use IaC (Terraform) to create new secrets from a given value (or generate one) and upload them to the GCP Secret Manager. Then, we use ExternalSecret CRD to retrieve the value from the cloud and create a k8s secret resource in our namespace.

While there are many tools and strategies to handle secrets when working with k8s, my understanding is that most engineers are still struggling to find the optimal solution. Some methods are more complex than others, while others are less secure. We believe that the solution described in this post is a happy medium between the two.

Notes:

We also considered using kubernets-secret-generator, however we found that there are some major disadvantages to this alternative. Using Argus’ method as described above, the generated secret is kept in both the Terraform state file and the GCP Secret Manager.

On the contrary, when using kubernetes-secret-manager the secret is only kept in the k8s resource. So in case of a disaster or if the secret is unintentionally deleted, the secret is completely lost. This can be painful if you’re using the secret for DBs (postgresql or mongoDB) where it’s not easy to update the password.