Securing Google Cloud Part 1: Dealing with Secrets on App Engine

The Problem: Sharing and deploying plaintext secrets

I built a Node.js API for a client recently, and we decided to deploy it to Google’s App Engine product. We chose to use the Flex environment, which is basically a Docker container managed by Google, since it offers more options like support for Websockets. Overall it’s worked well and provides easy, one command deployments.

One problem we faced was how to secure secret keys on App Engine. Initially we chose to place the plaintext keys in a folder within the API project, and have git ignore the folder. This kept the keys off of GitHub, but introduced a few other problems:

  1. We had plaintext keys in the deployment pipeline and ultimately sitting out on App Engine Flex, which we didn’t like.
  2. We had to share plaintext keys with each person who wanted to deploy.

Another obvious option was to load the keys into environmental variables, using the app.yaml file that defined our app and deployment. This didn’t help solve the issues above, though, and also introduced the problem of adding secrets to GitHub, since our app.yaml was committed to master.

The Solution: Google KMS with runtime decryption

To solve this problem, we decided to encrypt the secret keys using Google’s Key Management Service (KMS). The first step was to create a KMS keyring, and use it to encrypt our plaintext secrets.

Step 1: Create a keyring and key

Creating a new keyring (which holds keys) and a key is easy and can be done in the Google Cloud console, in IAM & Admin > Cryptographic Keys. You’ll want to create a Symmetric Encrypt/Decrypt key. We did a Generated key with a global location, which made everything easy.

Step 2: Encrypt your secrets locally

Once you’ve created the keyring and key, install and initialize the gcloud CLI. Then, in the IAM & Admin section of the Google Cloud console, grant your account the role of Cloud KMS CryptoKey Encrypter/Decrypter. This should be the same account that you initialized the CLI with.

Finally, run the following command, encrypting each of your local plaintext secrets using the newly created key.

gcloud kms encrypt --location global --keyring [your keyring name] --key [your key] --plaintext-file [name of plaintext secret file] --ciphertext-file [name for new encrypted secret file]

Step 3: Add encrypted secrets to App Engine project

Once your secrets are all encrypted, you can add them to a folder in your App Engine project. This will ensure that they will be deployed, so they can be located and decrypted at runtime.

As an option, you can also add them to source control at this point. We chose to do this since the keys are now encrypted, and we also have our GitHub repo set to private.

Step 4: Give the App Engine default service permission to decrypt

In order to decrypt the secrets at run time via code, you have to give the App Engine default service the proper permissions. The App Engine default service is named [project name]@appspot.gserviceaccount.com and can be located in the IAM & Admin section of the Google Cloud console. Edit this account by clicking on the pencil icon, then assign the role of Cloud KMS CryptoKey Encrypter/Decrypter.

Step 5: Add code to decrypt secrets at runtime

To decrypt the secrets at runtime, you can use the @google-cloud/kms package. It’s as simple as reading the encrypted secret file from the disk on App Engine, and then decrypting it using @google-cloud/kms.

Below is a code sample that shows how to decrypt a secret using the CryptoService I wrote. This is in Typescript, but can easily be converted to plain Javascript.

Code sample:

//read the encrypted file from disk
let encryptedKey = fs.readFileSync(path.resolve(__dirname, keyPath));

//use the CryptoService to decrypt the secret
let cryptoService = new CryptoService();
cryptoService.kmsDecryptSecrets(encryptedKey)
    .then((plaintextKey) => {
        //use the plaintext key here...
    })
    .catch((err) => {
        console.error('Error decrypting secret with KMS: ', err);
    })

CryptoService:

import * as kms from '@google-cloud/kms';

export class CryptoService {

    private projectId = ''; // Your GCP projectId
    private keyRingId = ''; // Name of keyring
    private locationId = 'global'; //The location of the keyring
    private keyName = ''; // Name of the key used to encrypt api secrets
    private client: any;

    constructor(){
        this.client = new kms.KeyManagementServiceClient();
    }

    //public functions

    public async kmsEncryptSecrets(plaintext: Buffer): Promise<Buffer> {
        let name = this.getKmsName(this.client, this.keyName);
        
        // Encrypts the file using the specified crypto key
        const [result] = await this.client.encrypt({name, plaintext});
        return result.ciphertext;
    }

    public async kmsDecryptSecrets(ciphertext: Buffer): Promise<string> {
        let name = this.getKmsName(this.client, this.keyName);
        
        // Decrypts the file using the specified crypto key
        const [result] = await this.client.decrypt({name, ciphertext});
        return result.plaintext.toString();
    }

    //private functions

    private getKmsName(client: any, keyName: string) : any {
        const name = client.cryptoKeyPath(
            this.projectId,
            this.locationId,
            this.keyRingId,
            keyName
          );
        return name;
    }
}

If you need additional help, feel free to comment. Thanks for reading!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s