Alternatives
I said at the beginning that Akeyless is the best tool for this job, but promised to mention some alternatives for the curious. Well, here you go! I’ll mainly be discussing to categories of options:
Hashicorp Vault
I’d be remiss if I didn’t mention Hashicorp Vault, as it’s certainly one of (if not the) largest player in the secret management space. I’ve used Vault and managed enterprise Vault clusters, so this is a tool that I’m well acquainted with. In principle, Vault can accomplish everything we discussed here:
- It has a variety of authentication methods (such as IAM Auth, OAuth2, SAML, etc…)
- It has dynamic producers for a variety of systems (Databases, Cloud Providers, SSH, etc…)
- Vault confgiuration can also be deployed by Terraform.
As a result, a quick glance suggests that Vault works just as well for our purposes as Akeyless. However, there are a few challenges with Vault that are worth pointing out which make it difficult to properly scale in many situations. All else being equal, I find these to be compelling reasons to prefer Akeyless over vault. Personally, I also prefer Akeyless - I find it much easier to use and appreciate that is backed by a globally available SaaS.
Rotation
One of our goals is to make temporary credentials the standard for every day access. However, there are a couple cases where “static” credentials are unavoidable (such as storing root passwords for databases). Akeyless makes automatic rotation for these credentials quite easy - it just takes a small amount of configuration. Vault, unfortunately, is lacking in this area - it exposes endpoints to rotate these credentials, but it doesn’t provide any options for automatically invoking them on a schedule. As a result, with Vault, we also have to manage infrastructure to schedule rotations.
Network Access
Vault database producers require direct network access to our databases, just like Akeyless database producers. However, Vault doesn’t have an equivalent of the Gateway that can be deployed directly into the network. As a result of this, you have to setup VPC peering/VPNs/Transit Gateways/etc to connect all of your networks to your secret manager’s network. In fact, if you sign up for a hosted Vault cluster, the first thing they will do is ask you what cloud provider you use so that they can setup VPC peering between their network and yours. When I used Vault I just ignored database producers, as using them would have rolled back ongoing initiatives around proper network segregation.
AWS Access
IAM Auth was a critical part of this process, and Vault has AWS IAM Auth as well. However, there is a subtle catch. Vault works a bit differently under the hood, with the end result that using IAM auth requires mutual cross-account permissions when your Vault cluster and your infrastructure live in different cloud accounts. This is fine if you only have a few cloud accounts, but can pose a scaling issue otherwise.
On Prem Access
In this example we didn’t need access for “on prem” resources, but these are still very common in most organizations. Therefore, I touched on Universal identity from Akeyless and how it perfectly fits the need for credential-less access from on-prem infrastructure. Unfortunately, Vault doesn’t have an equivalent (at least not that I ever found). When granting access to on-prem resources we inevitably just ended up creating vault tokens and copying and pasting them into our infrastructure. Unfortunately, there were no options for automatically rotating such tokens, which meant that are on prem systems were effectively forced to use static credentials.
Cloud Native
Since our example used cloud hosting it’s worth talking about cloud-native solutions that can help us with our goals. We do have one clear winner for network access, but only mixed results elsewhere.
SSM Connect
The major cloud providers have their own alternatives to SSH that you can use to connect to virtual machines. Most importantly, they grant access based on cloud permissions instead of static credentials, and often they allow you to connnect directly to machines even without network connectivity! This means that you don’t even need a bastion to connect to virtual machines in private networks. The AWS version is called SSM Connect. It’s important to clarify that it won’t allow you to connect to an RDS directly (just EC2 instances), so you do still need a bastion. However, you can deploy a VM into the same private subnet as the RDS and then use SSM to open a tunnel through the VM to the RDS. In essence, it allows you to build a “private” bastion that is completely innacessbile from the outside world.
This ability to directly connect to instances and use a bastion host that isn’t exposed to the internet is a huge plus, both in terms of security and ease-of-use (since you don’t have to worry about maintaining an IP address whitelist and can connect from anywhere). Since access to SSM connect is via AWS permissions (which are already handed out by Akeyless), this can fit into our current model quite well. As a result, SSM Connect is an even better option than certificate-based access with SSH. GCP also has an equivalent option (Cloud IAP), and Azure has a close relative (Azure Bastion). If you don’t have these options though (e.g. on-prem infrastructure, or virtual machines hosted outside of the major three cloud providers), then certificate-based SSH access is your best bet. This is why I used certificate-based authentication in these examples - SSM definitely has some good advantages, but not everyone uses AWS and certificate-based authentication can work anywhere.
Cloud-Native Secret Managers
Of course, all the major cloud providers have services intended for secret management. Unlike with Akeyless, they don’t have any concept of dynamic credentials. Rather, they are just a place to securely store secrets/access credentials. As a result, we end up back in the “static secret” boat. We can still follow the same principles of managing static secrets that we described previously: letting our application pull them directly out of the secret manager and automatically re-fetching them in the event of an authentication failure. This is still a big step up over pulling secrets out of the environment when the application launches. In addition, the AWS flavor (AWS Secret Manager) even has options for credential rotation - a handy way to ensure that your static secrets don’t stay static. This does require you to manage a Lambda in your account for each kind of secret you want to rotate. Compared to Akeyless this isn’t quite as effective (since everyone ends up using a shared credential) or as convenient (since you have to manage Lambdas for rotation). However, it’s still much better than the typical process of having the application pull secrets out of the environment when it launches and manually issuing/copying credentials around everywhere they are needed.
Pipeline/Action Authentication
The same way we used our job’s JWT to login to Akeyless can also be used to assume a cloud role directly from an Action. This Action login flow is by no means unique to Akeyless, and works with most major cloud providers. See for instance AWS, Azure, and GCP. This same process is also available to Gitlab pipelines via an equivalent JWT.
With Github and Gitlab you can also grant cloud access to your Pipelines/Actions by deploying a runner into your AWS account, using that to run your Workflows, and utilizing the permissions attached to your instance via it’s instance role. However, I would personally use JWT-based authentication instead of a role attached to my runner for two reasons:
- It’s more flexible and allows you to host your runners anywhere you want (or use public runners if your plan comes with free minutes).
- It’s more granular, allowing you to limit permissions and account access based on the brannch the job is running for.
IAM Database Authentication
AWS has an option for authenticating to databases using IAM permissions. This results in the same benefits of using database dynamic producers, so is a possible alternative when available. However, there are some limitations to IAM database authentication that result in it not always being recommended for production workloads.
Is That Really Everything?
Okay, so I certainly can’t claim to have tried all the secret management platforms out there, especially since there are more every year. Still, we have some very basic requirements that a secret manager must meet for it to have any hope of accomplishing our goals in this blog series:
- Credential-less access from anywhere (cloud, on-prem, pipelines, developer machines, people) to the secret manager
- Broad support for dynamic secrets to secure all access with credential-less authentication
- Tools for automatic configuration and deployment
I have yet to meet many vendors that can check those 3 boxes.
Summary
So, is Akeyless really the clear winner? Absolutely. On a very mature cloud platform, such as AWS, there are some native tools available that can help level up the secrets management game and have a good ROI. However, it misses out on dynamic credentials (and thus we have one set of credentials shared by everyone), and of course it only works as long as you stay locked down entirely in that single cloud platform. I have yet to meet many businesses that can honestly say that they are 100% commmitted to a single cloud provider.
For the rest of us, there’s Akeyless. Frankly, it’s better anyway :)
Summary