Enterprise scale deployment of secrets management tooling and development of custom functionality

Challenge

A major financial services company faced a familiar challenge: siloed secrets and inefficient Hashicorp Vault usage hampered security and operational agility. They sought to optimize their Vault deployment, and chose RKON to examine their environment, provide recommendations, and implement the solution to streamline access, enhance control, and ultimately, unlock the full potential of their secrets management infrastructure. This case study delves into their approach, highlighting the key decisions, tactical implementations, and measurable results achieved.

The existing solution had been developed as a proof-of-concept and had grown beyond its intended scale. It included multiple layers of cloud infrastructure and middleware to facilitate communications between a compartmentalized Vault cluster and target databases generally in unpeered AWS accounts. This approach had several drawbacks:

• It did not integrate with other infrastructure pipeline components, notably the security and compliance checkpoints
• There were several instances requiring burdensome ongoing maintenance and unnecessarily high costs

• Applications had overly broad access to secrets, as namespaces were shared haphazardly without policy enforcement

The platform team wanted to upgrade the solution to enforce enhanced security and compliance requirements while streamlining the developer experience.

Solution

RKON designed a solution to deploy, initialize, unseal, and configure Vault to the customer’s internal platforms with a custom plugin that enables integration with other internal services. This involved a combination of Helm, Golang automation modules, a Vault plugin, and an API layer provided by API Gateway with AWS Lambda.

The API provides a standard interface between a database and Vault to rotate credentials. The custom Vault plugin forwards the relevant pieces of its own configuration from the Vault server to the API. The API then calls a Lambda function to translate the request into DBMS administrative routines based on the request parameters and commands stored in text in an adjacent S3 bucket. Using private VPC endpoints, all communication between these components remains within AWS, without traversing the internet or other internal networks.

The customer’s internal deployment platforms leverage Helm to deploy infrastructure across a fleet of Kubernetes clusters. RKON developed a Helm chart to automate the deployment and management of Vault, with a custom plugin, to these customer’s internal platforms. The Vault plugin is managed in a single code repository and integrates with a pipeline to produce the plugin binary. A separate repository contains the code to deploy the containerized Vault server. This includes Terraform templates that deploy the cloud resources needed for Vault like KMS keys and DynamoDB tables. Additionally, it includes the input values that are provided to the official Vault Helm chart.

The Helm chart also references a container image containing a Golang application and the plugin binary. The Golang application initializes the newly deployed Vault instance, unsealing it and configuring the custom plugin. It is launched alongside Vault as an “initializer” container to copy the plugin binary to the Vault container and initialize Vault with the custom plugin.

Result

This solution enhanced the platform’s security by providing a robust service for protecting application secrets. Additionally, it reduced the management and deployment overhead of a security-critical system to a simple workflow. Developers can securely integrate secrets into their applications with a single change in source control.