Local Access

So, we have built, deployed, and are running a fully functional application without using any static credentials. We are managing access centrally from our secret manager and can easily grant access to new infrastructure without having to copy and paste credentials around. This is great! Of course, none of this does us any good if we can’t also make it easy for developers/DBAs/whatever to access the systems when needed. So how do we make that happen?

Database Access

The most important part is database access. Fortunately that is already solved. In Step #3 we configured SAML auth and gave admin access to everything in our environment’s folder. When we deployed our database (Step #8) this added two database producers to our environment’s folder: one with full access and one with more limited CRUD access. As a result, our SAML admins can already fetch database credentials. They just need network access!

Network Access

Certificate-Based SSH Authentication

Fortunately, network access is also very solvable. As part of our database deployment, we also launched an SSH bastion host in our VPC, so we just need to grant access to this. Rather than using SSH keys to access the bastion (which would just be another form of static credential), it is already configured for certificate-based authentication. Certificate-based authentication is an alternate authentication scheme available in most SSH server implementations that relies on certificates for authentication, in much the same way that browsers use certificates for SSL validation. Like with SSL validation, our certificates can have a lifetime, which means that we grant temporary access. Also like with SSL validation, we need a certificate authority that will issue our SSH certificates. This latter requirement is likely one of the reasons why certificate-based authentication is not more common.

Akeyless as an SSH Certificate Authority

Fortunately, Akeyless can be the certificate authority that manages access to the bastion. This was already setup by Terraform for us, but it never hurts to see the pieces. So, if you wanted to manually set up a certificate authority and configure a machine to trust it for SSH authentication, it would look like this:

1. In Akeyless, create a new DFC key
2. In the UI, go to “Secrets & Keys” -> “+ New” -> “Encryption Key” -> “DFC”
3. Specify a name and location
4. Set the type to “RSA4096”
5. Next, create an SSH Cert issuer
6. In the UI, go to “Secrets & Keys” -> “+ New” -> “SSH Cert Issuer”
7. Specify a name and location
8. For “Sign the SSH certificate with the following key” use the DFC key you just created
9. In “Allowed Users” specify the username you will use when connecting to the machine (e.g. ubuntu or ec2-user)
10. Extract the public key of the DFC key you created above
11. Set the public key as a trusted certificate issuer on the machine you will connect to (see the bastion startup script for an example of how to do this for ubuntu)

Again, our Terraform repository already did all of this for us. Therefore, all you have to do is use the certificate authority to connect to the bastion and from there to the database. That process looks roughly like this:

1. Login to Akeyless using SAML
2. Make sure you have an SSH public key locally (any public key will do)
3. Send your SSH public key to Akeyless and ask it to sign it using the certificate issuer. You must also specify the opeating system user you will be logging in as (e.g. ubuntu, ec2-user).
4. Receive back a temporary certificate signed by the certificate issuer
5. Connect to the bastion using your temporary certificate and open a tunnel through the bastion to the database host
6. Fetch database credentials from the database dynamic producer
7. Connect to the database through the local end of the tunnel using the credentials from the database producer

Of course, we’re not going to manually walk through those steps each time. Instead, we’ll encapsulate this behavior into some tooling/code that we can call as needed.

Network Access From clearskies

clearskies makes this process easy because it allows us to encapsulate this behavior in a class and drop it in our application configuration. This allows us to completely change the way that the application connects to our database using a single line of configuration (and some appropriately set environment variables, of course). It also comes with three main ways of interacting with our database from a local machine:

1. Run the local version of the application but connect to the remote database
2. Launch a standard mysql terminal connected to the remote database
3. Open network connectivity to the remote database but then dump temporary user credentials so you can connect with a client of your choice

Option #1 makes it easy to connect a local application to a remote database. For instance, consider the python we used to launch our application in a Lambda:

import app
import clearskies
import clearskies_aws


# build our context outside of the lambda handler and AWS will cache it for us for better performance
api_in_lambda = clearskies_aws.contexts.lambda_elb(
    app.api,
    additional_configs=[
        clearskies.secrets.additional_configs.mysql_connection_dynamic_producer(),
        clearskies.secrets.akeyless_aws_iam_auth(),
    ],
)
def lambda_handler(event, context):
    return api_in_lambda(event, context)

Keep in mind that app.api is the application that defines our API endpoint. We can take that same application and deploy it in a context, and with configuration, that is more appropriate for local execution:

import app
import clearskies


api_in_wsgi = clearskies.contexts.wsgi(
    app.api,
    additional_configs=[
        clearskies.secrets.additional_configs.mysql_connection_dynamic_producer_via_ssh_cert_bastion(),
        clearskies.secrets.akeyless_saml_auth(),
    ],
)
def application(env, start_response):
    return api_in_wsgi(env, start_response)

This code could then be attached to a WSGI server and run locally. If you wanted to connect to the database with an SQL client instead of the application, clearskies also has built in tooling to launch an SQL client connected to the database. Of course, it needs configuration information to do this- the IP address of the bastion host, the path to the dynamic producer, etc… These could be set as environment variables or you could put them in a .env file to be read by clearskies:

akeyless_mysql_dynamic_producer = "/path/to/producer/in/Akeyless"
akeyless_mysql_bastion_host = "555.555.555.555" # Ip address for bastion
akeyless_mysql_bastion_username = "ubuntu"
akeyless_mysql_bastion_cert_issuer_name = "/path/to/ssh/cert/issuer/in/Akeyless"
akeyless_mysql_bastion_local_proxy_port = 8888
db_host = "database.server.endpoint.aws.com"
db_database = "database_name"

In fact, since none of this is actually sensitive data, you can just add this .env file to your repository (likely with a BIG comment on the top letting developers know that they must not place secrets in it). This makes it incredibly easy to grant access to developers. If a new teammate joins and needs access to the database from their local machine, we simply grant them SAML access to the appropriate role in Akeyless. They can then clone the repository and immediately connect to the database and run applications against it: no handing out SSH keys, asking a database admin for credentials, copying and pasting hard-coded credentials, etc…

Similarly, if a developer leaves, they lose network, database, and AWS access immediately when their SAML access is revoked. This pattern also provides a strong layer of protection against stolen or lost developer machines/backups - on the off chance that an attacker gains access to the contents of a developers harddrive, they won’t find any credentials to steal.

Time Saving ROI!

To be explicit, these are the factors that come together to save substantial development time:

1. Our application fetches credentials directly from the secret manager
2. The secret manager automatically issues new credentials as needed.

The combination of these facts and a little bit of tooling means that we can easily control both network and database access in a single place, we ensure that every user has their own credentials, that all credentials are short-lived and automatically revoked, and that no manual effort is required when onboarding new developers. We end up with both stronger security and an easier development process!