Building Grafana Dashboards for Job Queues

Grafana is where the four signals of queue health become a picture an engineer can read at a glance, and this guide covers it as the visualisation layer of Observability & Monitoring for Job Queues. Prometheus stores the time series; Grafana turns them into the dashboard you stare at during an incident and the panels that drive your alerts.

A good queue dashboard is not a wall of every metric you collect — it is a deliberate, top-to-bottom narrative: backlog and trajectory at the top (is the system keeping up), throughput and saturation in the middle (why), and latency percentiles and failure rate below (what users feel). This guide builds that layout against Prometheus data, makes it reusable across every queue with template variables, and wires panel-level alerts so the dashboard and the paging come from one source of truth.

Problem Framing: From Raw Series to a Readable Story

The metrics from Prometheus Metrics for Workers are correct but unreadable in raw form — a hundred rate() queries across dozens of queues is noise. The dashboard's job is to compress that into the few panels that answer the operator's real questions in order: Are we falling behind? How fast? Is it a throughput problem or a downstream problem? Which queue? Get the panel choice and ordering wrong and the dashboard becomes decoration nobody opens during an incident.

Data Sources

Grafana reads from Prometheus over HTTP. Provision the data source as code so dashboards are reproducible and never depend on someone clicking through the UI. For BullMQ fleets where you scrape your own getJobCounts() poller, the same Prometheus data source serves those series too.

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: "15s"      # match your scrape_interval so rate() ranges line up
      httpMethod: POST          # POST handles long PromQL queries that overflow GET

Panels for Depth, Throughput, and Latency

Each row of the dashboard is one or two panels backed by a PromQL query. Lean on the recording rules from Prometheus Metrics for Workers so panels read cheap pre-computed series.

Backlog and trajectory (top row). A time-series panel of current depth, plus a stat panel projecting time-to-drain.

# Backlog depth for the selected queue (time-series panel)
queue_depth{queue="$queue"}

# Projected seconds to drain at the current net completion rate (stat panel)
queue_depth{queue="$queue"}
  / clamp_min(job:throughput:rate5m{queue="$queue"}, 0.001)

Throughput and saturation (middle row). Stack throughput by status so a retry storm is visually obvious — a growing red band of failures under a flat green band of successes.

# Jobs/sec by terminal status — stacked series panel
sum(rate(worker_jobs_total{queue="$queue"}[5m])) by (status)

# Worker saturation as a fraction of capacity (0–1 gauge panel)
sum(worker_inflight_jobs{queue="$queue"})
  / sum(worker_pool_size{queue="$queue"})

Latency percentiles (bottom row). Plot p50, p95, and p99 on one panel so the gap between typical and tail experience is visible at a glance.

# p50 / p95 / p99 execution latency from the histogram (one query per series)
histogram_quantile(0.50, sum(rate(worker_exec_seconds_bucket{queue="$queue"}[5m])) by (le))
histogram_quantile(0.95, sum(rate(worker_exec_seconds_bucket{queue="$queue"}[5m])) by (le))
histogram_quantile(0.99, sum(rate(worker_exec_seconds_bucket{queue="$queue"}[5m])) by (le))

Template Variables for Per-Queue Drill-Down

Hardcoding queue="default" into every panel means a new dashboard per queue — unmaintainable. A template variable turns one dashboard into a reusable view across every queue, with a dropdown that rewrites all panels at once. Define the variable as a label_values query so it auto-populates from the metrics themselves.

{
  "templating": {
    "list": [
      {
        "name": "queue",
        "type": "query",
        "datasource": "Prometheus",
        "query": "label_values(queue_depth, queue)",
        "refresh": 2,
        "includeAll": true,
        "multi": true,
        "sort": 1
      }
    ]
  }
}

With multi and includeAll enabled, panels should aggregate over the selection — use =~"$queue" (regex match) rather than ="$queue" in the PromQL so selecting multiple queues or "All" works correctly. This single change is what makes a queue dashboard scale to a fleet without per-queue duplication.

Dashboard-Managed Alerting

Grafana can own alert rules alongside the panels they visualise, so the threshold you see drawn on a graph is the threshold that pages. Define the rule against the same query and provision it as code.

# grafana/provisioning/alerting/queue.yml — backlog growth alert tied to the depth panel
apiVersion: 1
groups:
  - name: queue_health
    folder: Queues
    interval: 1m
    rules:
      - title: BacklogWillBreachDrainSLO
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: |
                queue_depth{queue=~".+"}
                  / clamp_min(job:throughput:rate5m, 0.001) > 600
          - refId: C
            type: threshold
            model: { conditions: [ { evaluator: { type: gt, params: [0] } } ] }
        for: 5m                 # require 5m sustained before paging — suppresses flapping
        labels: { severity: page }
        annotations:
          summary: "Queue {{ $labels.queue }} will not drain within the 10-minute SLO"

Whether Grafana or Alertmanager owns alerting is a team choice — Alertmanager centralises routing across many sources, while Grafana-managed alerts keep the rule visually next to its panel. The deeper backlog-alert design and Alertmanager routing live in Alerting on queue backlog with Prometheus.

Trade-off Analysis: Alerting Location and Panel Choices

Failure Modes & Recovery

Latency panel shows a flat line at the top bucket. The histogram buckets do not cover real latencies, so histogram_quantile interpolates from the +Inf bucket. Recovery: fix the buckets in the worker instrumentation (see Prometheus Metrics for Workers); Grafana can only render what the histogram captured.

Template variable dropdown is empty. The label_values query targets a metric or label that does not exist, or the data source UID is wrong. Recovery: run the label_values(...) query in Explore to confirm it returns values, and verify the variable's datasource matches the provisioned UID.

Multi-select breaks panels. Panels use ="$queue" (exact match) while the variable is multi-value. Recovery: switch every query to =~"$queue" so the regex match handles multiple selections and the "All" option.

Dashboard edits lost on redeploy. UI edits are not in the provisioned JSON. Recovery: treat provisioned dashboards as read-only, export changes back to the JSON in source control, and redeploy — never hand-edit production dashboards as the source of truth.

Performance Tuning

Heavy dashboards stress Prometheus, not Grafana. The fix is to push aggregation into recording rules so panels read a single pre-computed series instead of recomputing rate() and histogram_quantile() over millions of raw samples on every refresh. Set the dashboard's minimum interval to your scrape interval (15s) so panels never request finer resolution than the data supports. Cap auto-refresh — a 10s refresh on a 15s scrape just hammers Prometheus for data that has not changed; 30s–1m is plenty for a wall display. For very wide fleets, use one templated dashboard with multi-select rather than dozens of static ones, and put the busiest panels behind $queue so an operator loads only what they are looking at. A BullMQ-specific build of this dashboard, panel by panel, is in Building a BullMQ Grafana dashboard.

Building a Dashboard Someone Will Actually Use

A queue dashboard succeeds or fails on layout rather than on query sophistication. The organising principle that works is to arrange panels in the order an operator asks questions, so that scanning top to bottom reproduces a diagnosis.

The first row answers whether work is arriving: enqueue rate per queue, compared against the same window a week earlier. A collapse here is a producer-side incident that no consumer metric will reveal, and putting it first means it cannot be missed.

The second row answers whether work is being processed: throughput, failure rate, retry share and active workers. Zero throughput beside a non-empty queue is the loudest signal a job system produces, and it belongs where the eye lands second.

The third row answers how far behind the system is: oldest-message age, estimated drain time and queue-time percentiles. These are the numbers a responder communicates to everyone else, which is why they deserve their own row rather than being buried among rates.

The fourth is not a row of panels but an overlay: deploy and configuration-change annotations across every graph above. This is the single highest-value addition to any queue dashboard, because most incidents correlate with a change, and an annotation converts "throughput dropped around 14:20" into "throughput dropped at the 14:19 release".

Two anti-patterns are worth naming. Per-pod breakdowns churn constantly on an autoscaling fleet and rarely answer a question — aggregate by queue and keep pod detail for ad-hoc queries. And single-stat panels showing raw depth invite the wrong conclusion, because depth without duration is meaningless; if you display depth, display estimated drain time beside it.

Finally, keep the alerting rules that matter in Prometheus rather than in the dashboard, so they continue to fire when Grafana is unavailable. Dashboard-managed alerts are convenient for exploratory thresholds and a poor place for anything that pages.

FAQ

Should alerts live in Grafana or in Prometheus Alertmanager? Both work; pick by how your team routes pages. Alertmanager centralises routing, silencing, and deduplication across every alert source, which is the right home once you have more than a couple of services. Grafana-managed alerts keep the rule visually attached to the panel it watches, which some teams prefer for queue dashboards. The threshold logic is the same either way — what differs is where routing and on-call configuration live.

How do I make one dashboard work for every queue? Use a template variable defined as label_values(queue_depth, queue) so the dropdown auto-populates from your metrics, then reference it as =~"$queue" (regex match, not exact match) in every panel query. With multi-select and "Include All" enabled, one dashboard then covers the whole fleet, and a new queue appears in the dropdown automatically the moment it emits a metric.

Why are my p99 latency panels flat or obviously wrong? Almost always the histogram buckets in the worker instrumentation do not span the real latency range, so every slow observation lands in the +Inf bucket and histogram_quantile cannot interpolate a meaningful value. Grafana renders faithfully what the histogram captured — the fix is in the instrumentation, by setting bucket boundaries that bracket your actual p50 through p99.

Can Grafana read BullMQ or Sidekiq metrics, not just Celery? Yes — Grafana is framework-agnostic because it only reads Prometheus. As long as something exposes the series (the celery-exporter, a BullMQ getJobCounts() poller, or yabeda-sidekiq), the same panels and template variables work. You typically just adjust the metric names in the panel queries to match each exporter's naming.

Templating and Reuse Across Queues

A dashboard written for one queue and copied five times drifts within a quarter. Template variables solve this properly: a single dashboard with a queue selector, backed by a label query, covers every queue and stays correct as queues are added or removed.

Two details make templated dashboards work in practice. The variable should be populated from a label query rather than a hard-coded list, so a new queue appears without an edit. And panels should use the variable in every query rather than in some, because a half-templated dashboard silently shows one queue's depth beside another's throughput, which is worse than no dashboard at all.

Where a fleet has genuinely different queue classes — interactive, standard and bulk with different objectives — a single template can still serve all three, provided thresholds are expressed relative to the objective rather than as absolute numbers. A panel that colours red above thirty seconds is correct for the interactive queue and meaningless for the nightly batch; a panel that colours red above the queue's own objective is correct for both.

Keep the dashboard definition in version control and provision it rather than editing in the browser. The reason is not process purity: it is that an incident frequently ends with someone changing a threshold, and without a diff nobody can later tell whether the alert stopped firing because the system improved or because the threshold moved.

Alerts That Belong on the Dashboard, and Ones That Do Not

Grafana can define alert rules, and the convenience is real: the rule sits beside the panel that shows the same data, and creating it is a few clicks rather than a pull request.

That convenience is appropriate for exploratory thresholds — a value someone is watching this week to understand a workload — and inappropriate for anything that pages. A paging rule should be evaluated by the system that stores the data, defined in version control, and unaffected by the availability of a visualisation layer. Grafana being down during an incident is not far-fetched, and an alerting path that depends on it fails at exactly the wrong moment.

The practical split most teams settle on: paging rules in Prometheus, defined alongside the recording rules they depend on; informational and exploratory rules in Grafana, routed to a channel rather than a pager; and a periodic review that promotes anything in the second category that has proven itself into the first.

Keeping a Dashboard Honest

Dashboards decay in two directions and both are worth guarding against. They accumulate panels, because adding one during an incident is easy and removing one afterwards is nobody's job — and a dashboard with forty panels is scanned by nobody. And they drift from reality as queue names change, metrics are renamed and thresholds are adjusted in place without a record.

Two habits keep them useful. Review the dashboard after each significant incident and ask which panel actually answered the question; anything that answered nothing over several incidents can go. And provision the definition from version control so that threshold changes appear in a diff, which is what lets you distinguish "the alert stopped firing because we fixed it" from "the alert stopped firing because someone moved the line".

A final practical point: build the dashboard for the person who will open it at three in the morning, not for the person who built the system. That means fewer panels, explicit units on every axis, thresholds that encode the objective rather than a round number, and a text panel at the top naming the owning team and linking the runbook. Those four additions cost an afternoon and are what turn a wall of graphs into something a responder can act on without prior familiarity.

Finally, treat the dashboard as documentation of what the team believes matters. If a panel has not informed a decision in six months it is probably describing a concern the system has outgrown, and removing it makes the remaining panels easier to read.

One more habit worth adopting: annotate deploys onto every queue panel. Most incidents correlate with a change, and the annotation is what turns a coincidence in time into a hypothesis worth testing.

Related

• Building a BullMQ Grafana dashboard — a concrete, panel-by-panel build for a Node/BullMQ fleet.
• Observability & Monitoring for Job Queues — the four signals these panels visualise.
• Prometheus Metrics for Workers — the series and recording rules that back every panel.
• Alerting on queue backlog with Prometheus — designing the backlog alert these panels mirror.
• Horizontal Worker Scaling — the scaling actions a saturation panel triggers.