Skip to main content

Overview of System Maintenance for On-Premise Servers

Overview of System Maintenance for On-Premise Servers

In managing a Rundeck server or Process Automation cluster, it is worth looking at log management and server tuning. The notes about execution logs are also relevant for Runbook Automation.

Managing and Pruning Logs

There are two different types of logs that need to be considered and managed. First, there are system logs that are written and stored on each Rundeck server. The second type of logs are the logs for each job execution, which are stored by default on the server but could be stored in an alternate location using the S3 plugin.

Server System Logs

Server logs are managed using log4j 2open in new window and log size can be managed using logrotateopen in new window on Linuxopen in new window and Windowsopen in new window. For more mature logging environments, it is also possible to streamopen in new window these logs. For debugging purposes, the main log to use when investigating is service.log.

Execution Logs

Each time a job is executed, a log is written to the server for later review. By default, these logs will continue to be written and over time this can cause an issue with disk-space so pruning logs on a regular basis is recommended. This can be accomplished on a per-project basis by using a feature called “Execution History Clean”open in new window which ought to be set up early for each project to avoid issues down the line. It is also beneficial to copy the execution logs to an off-server location, typically either an S3-compatibleopen in new window storage location (using a tool such as Minioopen in new window) or using Azure Log Storageopen in new window.

Tuning the Server

There can be a number of different elements worth looking at to improve performance but there are three worth addressing from the beginning to prevent later server bottlenecks.

Increase number of file descriptorsopen in new window

One element that may be a limiter on Linux servers is the number of files that can be open at a time. Updating the file descriptors can improve server performance.

Managing Java heap sizeopen in new window

Java heap size controls how much memory is allocated to the server’s Java application. For a server with a large amount of system memory, try setting the maximum heap size to something like two-thirds of the total system memory (or half on Windows servers).

Adjusting Quartz job threadCountopen in new window

This setting determines the maximum number of concurrent threads that can be run on the server. By default, it is set fairly low so increasing this value can make the server more productive, especially for large numbers of concurrent jobs or jobs with large numbers of target nodes.


Monitoring the server with JMXopen in new window