Skip to main content

S3 Log Storage Plugin

S3 Log Storage Plugin

This is a plugin for Rundeckopen in new window that uses Amazon S3open in new window to store execution log files, for backup or for cloud-friendly behavior.

The source code lives at https://github.com/rundeck-plugins/rundeck-s3-log-pluginopen in new window.

Build

./gradlew clean build

Install

Copy the rundeck-s3-log-plugin-x.y.jar file to the libext/ directory inside your Rundeck installation.

Enable the ExecutionFileStorage provider named org.rundeck.amazon-s3 in your rundeck-config file:

rundeck.execution.logs.fileStoragePlugin=org.rundeck.amazon-s3

AWS Credentials

The plugin will by default use the "credentials provider chain" for AWS access credentials, which allows you to externally configure the credentials in any of three ways:

  1. Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_KEY
  2. Java system properties aws.accessKeyId and aws.secretKey
  3. AWS credentials file
  4. Instance Profile credentials, if you are running on EC2. (See the IAM user guideopen in new window).

If you want to specify access key and secret key, you can do so in the configuration:

Configuration

To configure the AWS access credentials you can set these property values:

AWSAccessKeyId : access key, required if using AWSSecretKey

AWSSecretKey : secret key, required if using AWSAccessKeyId

AWSCredentialsFile : properties file which contains accessKey and secretKey entries. Alternative to specifying the AWSAccessKeyId andAWSSecretKey`

S3 configuration uses these plugin configuration property values:

bucket : name of the S3 bucket to use

path : a path-like string that defines where in the bucket to store the log for a particular execution. You can include variables to expand. Default value: rundeck/project/$PROJECT/logs/$ID

Variables in the path value include:

  • ${job.execid} - the execution ID
  • ${job.project} - the project name
  • ${job.id} - the Job UUID if it exists
  • ${job.group} - the Job Group if it exists
  • ${job.path} - the Job Name if it exists

region : AWS region name to use. Default: us-east-1

endpoint: Optional, a custom S3 compatible endpoint to use, such as https://my-host.com/s3

pathStyle: Optional, boolean, default=False, set to True if you need to define the bucket in your S3 like endpoint URL. e.g https://<s3_like_end_point_url>/<your_bucket_name> A custom way of defining buckets in your endpoint. Useful for non-AWS S3 like object storage technology e.g swift stack, Optums, etc. Background information http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html. May be useful if you have an https endpoint URL.

encodeUserMetadata : Encode user metadata to URL format. This allows you to add special characters not included on US-ASCII. Default: false

You can define the configuration values in framework.properties by prefixing the property name with the stem: framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.. Or in a project's project.properties file with the stem project.plugin.ExecutionFileStorage.org.rundeck.amazon-s3..

For example:

#AWSAccessKeyId and AWSSecretKey can be specified in the file
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSAccessKeyId=ABC123...
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSSecretKey=ABC321...

#alternately, AWSCredentialsFile can point to a file which contains `accessKey` and `secretKey`
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.AWSCredentialsFile=/path/to/awscredentials.properties

#name of the bucket
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.bucket=test-rundeck-logs

#path to store the logs
framework.plugin.ExecutionFileStorage.org.rundeck.amazon-s3.path=logs/${job.project}/${job.execid}.log

To use an AWS credentials file, you must not use the AWSAccessKeyId, AWSSecretKey or AWSCredentialsFile properties. Instead, create a credentials file in the Rundeck user's home directory under ~/.aws/credentials . This allows you to use full AWS AIM identities for authentication to AWS.

An example of a working configuration:

[userauthentication]
aws_access_key_id = AKIWWWWWWWWWWWWWW
aws_secret_access_key = wo9........pYq

[default]
role_arn = arn:aws:iam::012345678900:role/yourrolehere
source_profile = userauthentication
region = us-east-2

Using with Rundeck SSL Configuration

If you want to use this plugin when you have Rundeck configured with a custom SSL truststore, you will need to import the Amazon S3 SSL certificates to your truststore.

echo -n | openssl s_client -connect my-bucket.s3.amazonaws.com:443 > certs.out
keytool -importcert -trustcacerts -file certs.out -alias s3-amazonaws -keystore $RDECK_BASE/etc/truststore