Skip to main content

Monitor a Runner with Prometheus and Grafana

Monitor a Runner with Prometheus and Grafana

Rundeck Runners expose operation-queue, report-delivery, and JVM metrics that are useful for monitoring Runner health and capacity planning. Unlike the Rundeck server, a Runner does not serve an HTTP metrics endpoint — its metrics are published as JMX MBeans. To get them into Prometheus and Grafana, run the Prometheus JMX Exporter alongside the Runner.

This guide uses the jmx_prometheus_javaagent, which runs in-process and exposes the Runner's metrics on an HTTP port that Prometheus can scrape.

Validate metric names in your environment

The JMX-to-Prometheus name mapping depends on the exporter rules you use, so the exact Prometheus series names can vary. Treat the names in the example queries below as a starting point and confirm them against your own exporter output (see Step 4).

For the full list of Runner metrics and what they mean, see the Runner Metrics Reference. For a server-side Prometheus + Grafana walkthrough, see Monitor the Rundeck Server with Prometheus and Grafana.

Architecture

  Runner JVM (pd-runner.jar)
    └─ JMX MBeans  ─→  jmx_prometheus_javaagent (:9404)  ─→  Prometheus (:9090)  ─→  Grafana (:3000)

The exporter reads the Runner's MBeans from inside the same JVM and serves them in Prometheus format. No JMX remote port needs to be opened.

Prerequisites

  • A running Rundeck Runner (Replica) version 6.0 or later. Metrics export over JMX is enabled by default.
  • Access to the Runner's startup command so you can add a JVM argument.
  • A Prometheus + Grafana stack. If you do not already have one, follow Monitor the Rundeck Server with Prometheus and Grafana to stand one up, then add the Runner as an extra scrape target.

Step 1: Download the JMX Exporter

Download the jmx_prometheus_javaagent JAR from Maven Central and place it next to the Runner JAR:

curl -L -o jmx_prometheus_javaagent.jar \
  https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.0/jmx_prometheus_javaagent-1.0.0.jar

Step 2: Create the exporter configuration

The Runner publishes its metrics through Micrometer's JMX registry (under the metrics domain), so the exporter rules map those MBeans to the exact Prometheus series names the Runner Grafana dashboard expects. The pattern, first-match-wins, is: operation gauges and counters to plain names, timers to SUMMARY metrics with {quantile} labels (milliseconds converted to seconds via valueFactor), JVM threads/GC to Micrometer-compatible aliases, noisy duplicate attributes suppressed, and a catch-all so nothing is silently dropped.

A representative excerpt — one gauge, one timer, and the catch-all — is shown below. It is abbreviated: the Runner dashboard and the panel queries in Step 6 rely on the full rule set, so use the complete file from docker-zoo (linked in the tip after the excerpt), not just this snippet.

startDelaySeconds: 0
ssl: false
lowercaseOutputName: true
lowercaseOutputLabelNames: true

rules:
  # Runner operation gauge → exact dashboard metric name
  - pattern: 'metrics<name=runnerOperationsPoolUtilization, type=gauges><>Value'
    name: runner_operations_pool_utilization
    type: GAUGE
    help: "Runner operation thread pool utilization ratio (0.0–1.0)"

  # Timer → SUMMARY with {quantile} labels; Micrometer emits ms, valueFactor converts to seconds
  - pattern: 'metrics<name=runnerOperationsInvocations\.handler\.([^,]+), type=timers><>(\d+)thPercentile'
    name: runner_operations_invocations_seconds
    type: SUMMARY
    valueFactor: 0.001
    labels:
      handler: "$1"
      quantile: "0.$2"

  # Catch-all: export every remaining JMX bean unchanged
  - pattern: ".*"

Use the complete, tested configuration

The full jmx-config.yml — mapping every Runner operation, reporter, HTTP-client, and JVM metric the dashboard uses, plus the suppression rules that drop noisy duplicate attributes — is published as a runnable example in docker-zoo. Copy runner-agent/jmx-config.yml from there rather than assembling the rules by hand.

Step 3: Start the Runner with the exporter attached

Add the -javaagent argument to the Runner's startup command. The 9404 below is the port the exporter will serve metrics on:

java \
  -javaagent:./jmx_prometheus_javaagent.jar=9404:./jmx-config.yml \
  -jar pd-runner.jar

Use your actual Runner JAR name (for example runner-<uuid>.jar).

For a Docker-based Runner, mount the exporter JAR and config into the container and append the agent to the Java command (or JAVA_TOOL_OPTIONS), and publish port 9404:

runner:
  ports:
    - "9404:9404"
  volumes:
    - ./jmx_prometheus_javaagent.jar:/app/jmx_prometheus_javaagent.jar:ro
    - ./jmx-config.yml:/app/jmx-config.yml:ro
  command: >
    java
    -javaagent:/app/jmx_prometheus_javaagent.jar=9404:/app/jmx-config.yml
    -Drunner.credentials.file=/app/.rdrunner-creds
    -jar pd-runner.jar

Step 4: Verify the exposed metrics

With the Runner running, scrape the exporter directly to confirm metrics are being published and to discover the exact series names produced by your rules:

curl http://localhost:9404/metrics | grep -i runner

You should see the Runner's operation and report-delivery metrics, for example series derived from runner.operations.pool.utilization, runner.operations.running, runner.reporter.queue.size.total, and runner.reporter.max_delivery_delay_seconds. Note the exact names returned here — you will use them in your Grafana queries. If a metric you expect is missing, widen the rules in jmx-config.yml (the catch-all should surface it under an auto-generated name).

Step 5: Add a Prometheus scrape target

Add the exporter to your prometheus.yml:

scrape_configs:
  - job_name: 'rundeck-runner'
    static_configs:
      - targets: ['runner:9404']
        labels:
          service: runner

Replace runner:9404 with the address Prometheus uses to reach the exporter. Reload or restart Prometheus, then confirm the rundeck-runner target is UP under Status → Targets.

Step 6: Build dashboard panels

Start from the ready-made dashboard

Rather than building every panel by hand, import the pre-built Runner dashboard from the docker-zoo monitoring example — grafana/dashboards/Runner-Dashboard.json. In Grafana use Dashboards → New → Import. It binds to a Prometheus data source with uid prometheus and expects the metric names produced by the runner-agent/jmx-config.yml mapping from Step 2. The panels below explain the individual queries if you prefer to build your own.

In Grafana, add panels using the Prometheus data source. Use the names you confirmed in Step 4 — the queries below assume the full rule set from Step 2 (the complete jmx-config.yml in docker-zoo), not only the abbreviated excerpt.

Operation pool utilization (0–100%):

runner_operations_pool_utilization * 100

Running vs. queued operations:

runner_operations_running
runner_operations_queued
runner_operations_inflight
runner_operations_pool_capacity

Report backlog and delivery delay (leading timeout indicator):

runner_reporter_queue_size_total
runner_reporter_max_delivery_delay_seconds

JVM heap usage:

jvm_memory_bytes{area="HeapMemoryUsage"}

Diagnosing report-delivery timeouts

runner.reporter.max_delivery_delay_seconds is the earliest warning sign: as it approaches 540 seconds, a server-side timeout (at 600 seconds) is imminent. Pair this Runner dashboard with the server-side runner_server_report_* metrics from Monitor the Rundeck Server with Prometheus and Grafana to see both ends of the pipeline. See the Runner Metrics Reference for the full diagnostic playbook and suggested alerts.

Alternative: scrape JMX remotely

If you prefer not to run an in-process agent, you can instead enable JMX remote access on the Runner (see Status & Monitoring → Monitoring Replicas) and run the standalone jmx_prometheus_httpserver as a separate process pointed at the Runner's JMX port. The in-process javaagent approach in this guide is simpler because it does not require opening a remote JMX port.

Ship container logs to Grafana with Loki

The JMX exporter above covers metrics. To get the Runner's logs into the same Grafana — when the Runner runs in Docker — use the Loki Docker logging driver: Docker ships the container's stdout and stderr straight to Loki, with no extra agent, collector, or Docker-socket access. You reuse the same Loki you stood up for the server in Monitor the Rundeck Server with Prometheus and Grafana; only Loki and Grafana are involved.

  Runner container ─(Docker Loki log driver)→ Loki ─→ Grafana (:3000)

Step 1: Install the Loki Docker driver plugin

If you have not already installed it for the server, install the plugin on the Docker host once:

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
docker plugin ls | grep loki   # ENABLED should be "true"

Step 2: Route the Runner container's logs to Loki

Add a logging block to the Runner service:

  runner:
    logging:
      driver: loki
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
        loki-retries: "5"
        loki-batch-size: "400"
        mode: "non-blocking"

The loki-url is resolved by the Docker daemon on the host, so it uses localhost:3100 (Loki's published port), not the compose service name. mode: "non-blocking" keeps the container from stalling if Loki is briefly unavailable.

Step 3: Explore Runner logs in Grafana

Restart the stack (docker compose up -d). In Grafana, open Explore, select the Loki data source, and query by the labels the driver attaches automatically:

  • {compose_service="runner"} — logs from the Runner container.
  • {compose_project="<your-project>"} — every container in the stack.

Browse without writing queries: Logs Drilldown

Grafana's Logs Drilldown app (Drilldown → Logs in the left menu) lists each service and lets you filter and drill into logs visually, no LogQL required. It relies on Loki's volume endpoint — make sure the server guide's loki-config.yml sets limits_config.volume_enabled: true. Services appear by their service_name label, which with the Docker driver defaults to the container name.

Not running in Docker?

The logging driver only applies to containers. If the Runner runs as a plain process, point a log shipper such as Grafana Alloy or Promtail at its log file instead.

Next steps

Privacy Policy |