Report a problem Edit this page

Use case: database credential rotation

Now that we know how to create custom workflow step script plugin in the hello world tutorial, let’s walk through a real world use case: rotation database credentials. Credential rotation has traditionally been a complicated, thankless administrative chore. It’s the type of work that might be labeled toil and exactly the kind of work that Rundeck has been designed to streamline.

The end result of this tutorial as well as a Docker environment to run it in can be found on GitHub

The problem

What’s difficult about database credential rotation?

  • It involves multiple steps across multiple nodes in your production cluster.
  • It’s necessary to handle secrets in the form of the new credentials you’re creating, plus credentials of a super user that can create the new login.
  • Manually run steps are error prone and mistakes can potentially cause a site-wide outage if your app can’t authenticate with the database anymore.

Addressing the problem with Rundeck

By creating a custom script plugin to interact with the database, its downstream clients and the Rundeck Key Storage, we can automate the creation and deployment of the new credentials as well as safely handle the decomissioning of the old credentials.

Secure secrets handling

Since the Key Storage stores the credentials, the administrator triggering the credential rotation doesn’t ever need to see the actual credentials of the database super user or the newly generated database user.

Critical logic is tested ahead of time

The logic for creating the database login is encapsulated in a script that has been tested prior to actually needing to rotate the credentials, so we avoid putting a human in the stressful, error-prone situation of having to figure out correct syntax to run on a live system on the fly.

Similarly, the app restart logic is encapsulated as well, with health check logic as an extra safety measure to halt the process in case something has gone wrong before applying the change to the whole cluster.

Ease of use encourages proactive security

Lastly, by creating a single button solution to rotate database credentials, we’re much more likely to rotate our credentials on a regular basis to mitigate potential security risks, rather than only after a security incident has already occured.

Automating credential rotation with Rundeck

In order to automate the credential rotation, we need to automate the individual steps, then orchestrate the steps in a single process. The steps we want to automate are:

  • Creating a new database login
  • Updating the application configuration to use the new database login
  • Restarting the applications to apply the configuration change
  • Deleting the old database login

Restarting the apps

We can start with restarting the applications because that’s something we can test on its own before proceeding. We’ll create a new plugin metadata file rundeck-plugins/db-creds/plugin.yaml:

Our restart script takes 2 parameters: the name of the process on the node, and a url that we can poll to check that it’s healthy. The implementation of the restart script is specific to the Docker playground environment so we’ll leave out the details, but from a high level it:

  • Kills the process and lets the supervising parent process restart it
  • Polls the health check url until it returns healthy or times out
  • Exits with an error code if it times out to let the calling process know whether to continue or not with downstream steps

In order to be able to run the restart, we create a job that specifies the nodes to run it on and the input parameters. rundeck-project/jobs/RestartApp.yaml:

We specify a uuid here so that we know how to refer to it from other jobs, otherwise Rundeck will assign a random uuid.

We can test the restart using our playground Docker environment:

Modifying the app config

Now we can create a step to update the configured database login on the application nodes. In your production environment you might use a configuration management tool like Ansible or Chef to accomplish this but here we use a simple Python script. We can add this to the providers key of rundeck-plugins/db-creds/plugin.yaml:

Note the valueConversion: "STORAGE_PATH_AUTOMATIC_READ" setting to able to refer to a path in Key Storage for the database password.

Corresponding job config rundeck-project/jobs/UpdateAppConfig.yaml:

In the job, we expose an option to the job user to specify the database user to set the config to. We use that value as part of the Key Storage path to look up the password. By using a naming convention for the key path, we hide the details of the Key Storage setup from the job user.

We can test it with:

Creating the new db login

Creating a new database login requires a super user database login plus a user and password for the new login. Add this to the providers key of rundeck-plugins/db-creds/plugin.yaml:

Corresponding job config rundeck-project/jobs/CreateDbUser.yaml:

Deleting the old db login

Updating rundeck-plugins/db-creds-plugin/plugin.yaml:

And the corresponding job rundeck-project/jobs/DeleteDbUser.yaml:

Tying it all together

To tie all the steps together into a single job, we can use job reference steps. rundeck-project/jobs/RotateDbCredentials.yaml:

We’ll need a new user and password, which we can create in the Key Storage:

Then running the job should push the new key. We need to specify both the new version and the previous version that needs deleting.

And there you have it: a single command to rotate your database credentials!

Report a problem Edit this page