Prometheus Metrics for Workers

This guide goes deep on the metrics layer of Observability & Monitoring for Job Queues: how to design, expose, scrape, and alert on worker metrics with Prometheus so that queue depth, saturation, latency, and failure rate become first-class time series you can graph and page on.

Prometheus is a pull-based system. Your workers do not push anywhere; instead each worker (or a sidecar exporter beside it) exposes a plain-text /metrics HTTP endpoint, and the Prometheus server scrapes that endpoint on a fixed interval, storing every sample in its local time-series database. This model is robust — a worker that crashes simply stops being scraped, which is itself a signal — but it imposes two design constraints that shape everything else: metrics must be cheap to expose on every scrape, and label cardinality must stay bounded because every unique label set is a separate stored series.

Problem Framing

The naive instinct is to log everything and grep later. That fails for queue health because the questions you need answered in an incident — "is the backlog draining", "what is the p99 right now", "has the failure rate doubled" — are aggregate, time-windowed, and need to be answered in milliseconds while you are paging. Logs cannot do that at scale; pre-aggregated metrics can. The job here is to pick the right instrument type for each of the four queue signals, expose them efficiently, and write recording and alerting rules that turn raw counters into operational answers.

Architecture: Pull, Scrape, Store

The data path is: worker process → exporter /metrics endpoint → Prometheus scrape → TSDB → recording/alert rule evaluation → Grafana queries and Alertmanager pages. For a Celery fleet the topology looks like this:

 ┌──────────────┐   listens to    ┌────────────────┐  scrape /metrics  ┌────────────┐
 │ Celery worker│── task events ─▶│ celery-exporter │◀──── 15s ────────│ Prometheus │
 │ worker N ... │   (broker bus)  │  :9808/metrics  │                   │   TSDB     │
 └──────────────┘                 └────────────────┘                   └─────┬──────┘
                                                                              │ rules
                                                                  ┌───────────┴───────────┐
                                                                  ▼                       ▼
                                                            Grafana panels         Alertmanager

The exporter exists because Celery has no native /metrics endpoint — it instead emits a stream of lifecycle events on the broker. The exporter subscribes to that stream and translates events into metrics. BullMQ and Sidekiq differ: their state lives in Redis and is read directly, so the "exporter" is a thin polling service you write or a gem you add. The scrape config below targets all three.

# prometheus.yml — scrape config for a mixed worker fleet
global:
  scrape_interval: 15s          # default cadence; do not poll brokers faster than this
  evaluation_interval: 15s      # how often recording/alert rules run

rule_files:
  - "rules/queue_recording.yml"
  - "rules/queue_alerts.yml"

scrape_configs:
  - job_name: "celery"
    static_configs:
      - targets: ["celery-exporter:9808"]   # one exporter per broker/cluster
  - job_name: "bullmq"
    static_configs:
      - targets: ["bull-metrics:3000"]       # your getJobCounts() polling service
  - job_name: "sidekiq"
    static_configs:
      - targets: ["sidekiq:9394"]            # yabeda-sidekiq default port

Implementation 1: prometheus_client Instrument Design

When you own the worker code, instrument it directly with the official prometheus_client. The art is choosing the right instrument per signal. The rule: counters for things that only go up (totals you take a rate() of), gauges for things that go up and down (current backlog, in-flight count), and histograms for distributions (the only way to get percentiles).

from prometheus_client import Counter, Gauge, Histogram, start_http_server

# --- Failure / retry rate: a counter, sliced by terminal status ---
JOBS_TOTAL = Counter(
    "worker_jobs_total",
    "Jobs that reached a terminal state",
    ["queue", "task", "status"],     # status ∈ {success, failure, retry}
)

# --- Queue depth & saturation: gauges (instantaneous values) ---
QUEUE_DEPTH = Gauge("worker_queue_depth", "Jobs waiting, not yet picked up", ["queue"])
INFLIGHT = Gauge("worker_inflight_jobs", "Jobs currently executing", ["queue"])

# --- Latency: histograms, split into wait and execution ---
QUEUE_WAIT = Histogram(
    "worker_queue_wait_seconds",
    "Time from enqueue to dequeue (scheduling latency)",
    ["queue"],
    buckets=(0.01, 0.05, 0.1, 0.5, 1, 5, 15, 60, 300),
)
EXEC_TIME = Histogram(
    "worker_exec_seconds",
    "Time from dequeue to ack (handler latency)",
    ["queue", "task"],
    buckets=(0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300),
)

def process(job):
    QUEUE_WAIT.labels(job.queue).observe(job.dequeued_at - job.enqueued_at)
    INFLIGHT.labels(job.queue).inc()
    try:
        with EXEC_TIME.labels(job.queue, job.task).time():
            job.run()
        JOBS_TOTAL.labels(job.queue, job.task, "success").inc()
    except Retryable:
        JOBS_TOTAL.labels(job.queue, job.task, "retry").inc()
        raise
    except Exception:
        JOBS_TOTAL.labels(job.queue, job.task, "failure").inc()
        raise
    finally:
        INFLIGHT.labels(job.queue).dec()

if __name__ == "__main__":
    start_http_server(8000)   # exposes /metrics on :8000 for Prometheus to scrape

Two subtleties bite people in production. First, histogram buckets must match your real latency range — the library defaults top out around 10 seconds, so a 5-minute ETL job lands entirely in the +Inf bucket and histogram_quantile returns garbage. Set buckets that bracket your actual p50 through p99. Second, in a forked worker model (Celery prefork, Gunicorn) each child process has its own metrics registry; use prometheus_client's multiprocess mode with a shared PROMETHEUS_MULTIPROC_DIR, or scrape an exporter that aggregates across processes instead.

Implementation 2: celery-exporter (No Code Changes)

When you cannot or do not want to edit task code, celery-exporter gives you the four signals from Celery's event stream alone. Enable events on the workers, run the exporter as a sidecar, and point Prometheus at it.

# celery_app.py — turn on the event stream the exporter consumes
app.conf.update(
    worker_send_task_events=True,   # publish task-started / -succeeded / -failed / -retried
    task_send_sent_event=True,      # publish task-sent at enqueue (enables queue-wait metrics)
)

# docker-compose.yml — exporter as a sidecar against the same broker
services:
  celery-exporter:
    image: danihodovic/celery-exporter:latest
    command: ["--broker-url=redis://redis:6379/0"]
    ports: ["9808:9808"]

The exporter publishes celery_task_sent_total, celery_task_succeeded_total, celery_task_failed_total, celery_task_retried_total (counters, sliced by name and queue) and celery_task_runtime_seconds (a histogram). It also exposes celery_queue_length by reading the broker directly, which is your backlog gauge. This is the lowest-friction path to a fully instrumented Celery fleet; the end-to-end walkthrough is in Instrumenting Celery with a Prometheus exporter.

Recording Rules: Pre-Compute the Expensive Queries

Dashboards and alerts repeatedly evaluate the same rate() and histogram_quantile() expressions. Recording rules compute them once per evaluation interval and store the result as a new, cheap series, so every panel and alert reads a single value instead of re-aggregating thousands of raw samples.

# rules/queue_recording.yml
groups:
  - name: queue_derived
    interval: 15s
    rules:
      # Throughput: successful jobs per second, per queue
      - record: job:throughput:rate5m
        expr: sum(rate(celery_task_succeeded_total[5m])) by (queue)

      # Failure rate: failures as a fraction of completed jobs
      - record: job:failure_rate:5m
        expr: |
          sum(rate(celery_task_failed_total[5m])) by (queue)
          /
          (sum(rate(celery_task_succeeded_total[5m])) by (queue)
           + sum(rate(celery_task_failed_total[5m])) by (queue))

      # p99 execution latency, pre-computed from the runtime histogram
      - record: job:exec_p99:5m
        expr: |
          histogram_quantile(0.99,
            sum(rate(celery_task_runtime_seconds_bucket[5m])) by (le, queue))

Trade-off Analysis: Instrument Strategies

Failure Modes & Recovery

Cardinality explosion. Someone adds job_id (or user_id, or a URL) as a label. Each unique value becomes a permanent series; memory climbs until Prometheus OOMs. Recovery: find the offending metric with topk(10, count by (__name__)({__name__=~".+"})), drop the label at the exporter or with metric_relabel_configs, and never reintroduce unbounded labels.

Histogram percentiles look wrong (flat at the top bucket). Buckets do not cover the real latency range, so observations pile into +Inf and histogram_quantile interpolates nonsense. Recovery: widen the bucket boundaries to bracket your observed p50–p99 and redeploy; historical data keeps the old buckets, so the fix is forward-looking.

Counter resets on deploy read as negative spikes. A worker restart resets counters to zero. Recovery: this is expected — always wrap counters in rate() or increase(), which are reset-aware. Never alert on a raw counter value.

Stale gauge after a worker dies. A gauge like worker_inflight_jobs keeps its last value if the worker vanishes mid-job, hiding the loss. Recovery: prefer reading backlog from the broker (where the truth lives) over in-process gauges, and add an up == 0 alert so a missing scrape target itself pages.

Performance Tuning

Keep the scrape cheap: a /metrics response should render in single-digit milliseconds. The dominant cost is total series count (cardinality), not request volume, so the highest-leverage tuning is pruning labels. Use a 15s scrape_interval as the default and align any in-app polling (the BullMQ getJobCounts() loop) to it — polling Redis every second when you scrape every 15s just burns broker round-trips. For long retention, push to remote-write storage (Thanos, Mimir, Cortex) rather than growing local TSDB unbounded. Put expensive aggregations in recording rules so the read path stays fast under dashboard load. These derived series feed directly into the panels described in Grafana Dashboards for Queues and the backlog alerts in Alerting on queue backlog with Prometheus.

The Metric Set Worth Standardising On

Most queue instrumentation converges on the same dozen series, and standardising them across every service in an organisation is worth more than any individual metric, because it makes dashboards and alerts portable.

Counters. jobs_enqueued_total, jobs_completed_total, jobs_failed_total and job_retries_total, each labelled by queue and — where cardinality permits — by task. Every rate, ratio and error-budget query derives from these four, and their cumulative nature means a worker restart is handled correctly by rate() without special cases.

Gauges. queue_depth and queue_oldest_message_age_seconds per queue, plus worker_slots_busy and worker_slots_total per fleet. Depth alone is famously misleading; age is what distinguishes a burst from a stall, and the slot pair gives saturation without inferring it from CPU.

Histograms. job_queue_time_seconds and job_duration_seconds, with boundaries placed on every threshold you intend to alert on. These are the expensive series and the ones that answer latency questions, which is why their labels should stay coarse — queue, and at most task.

Two conventions make the set considerably more usable. Label values should be bounded and stable: queue names and task names, never job identifiers, tenant identifiers or exception messages. And every service should emit the same names, so a query written for one fleet works unchanged against another — which is what makes a shared dashboard possible at all.

What to leave out is equally important. Per-job gauges, per-tenant histograms and anything keyed on a user-supplied value will eventually take the monitoring system down, and the failure is gradual enough that nobody connects it to the metric that caused it. Put that detail in logs or exemplars, where unbounded cardinality is expected.

From Metrics to Decisions

Instrumentation is only useful insofar as it changes what someone does, and three derived quantities do most of that work.

Estimated drain time — backlog divided by current throughput — converts a queue depth into a sentence a responder can say out loud. "Forty minutes behind" prompts a decision; "482,000 messages" prompts a follow-up question.

Retry share — retries as a fraction of total attempts — distinguishes a dependency that is slow from one that is failing, and those have different mitigations. A rising share with stable throughput usually means transient errors are being absorbed; a rising share with falling throughput means the retries are the problem.

Saturation — busy slots over total slots — answers whether adding capacity will help. High saturation with a growing backlog is a scaling problem; low saturation with a growing backlog is a downstream problem, and adding workers will make it worse.

Precompute all three as recording rules rather than assembling them ad hoc during an incident. The cost is trivial and the benefit is that the number is already on the dashboard when someone needs it, rather than being derived under pressure from two panels and a calculator.

FAQ

Counter, gauge, or histogram — how do I choose for a job metric? Match the instrument to the question. "How many jobs failed since start" is a total that only increases, so it is a counter and you query it with rate(). "How many jobs are waiting right now" goes up and down, so it is a gauge. "What is the p99 execution time" is a distribution, which only a histogram can answer — no combination of counters and gauges reconstructs percentiles after the fact.

Why use the celery-exporter instead of instrumenting tasks directly? The exporter requires zero changes to task code — you flip on Celery's event stream and run a sidecar, and you immediately get throughput, failure, retry, and runtime metrics across every task. Direct instrumentation gives you finer control and custom labels but means editing and redeploying every worker. Most teams start with the exporter and add direct instrumentation only for the handful of tasks that need bespoke business metrics.

How do I expose metrics from a prefork Celery worker without losing per-child data? Each forked child has its own in-process registry, so a plain start_http_server only ever shows one child's numbers. Use prometheus_client's multiprocess mode with a shared PROMETHEUS_MULTIPROC_DIR so children write to a common directory that a single collector aggregates, or sidestep the problem entirely by using the celery-exporter, which reads the broker-level event stream rather than per-process state.

What scrape interval and retention should I run? Start at a 15s scrape interval and tune only with evidence — 10s for latency-critical alerting, slower for cheap background fleets. For retention, local TSDB is fine for weeks; beyond that, ship to remote-write storage like Thanos or Mimir rather than letting the local database grow without bound, and lean on recording rules so long-range queries stay affordable.

Scrape Configuration for a Churning Fleet

An autoscaling worker fleet creates and destroys targets continuously, which interacts with a pull-based collector in ways worth planning for. Service discovery should find workers by label rather than by a static list, and the scrape interval should be short enough that a worker living for two minutes is still observed at least a few times. Fifteen seconds is a reasonable default and gives a five-minute rate window forty samples to work with.

Short-lived processes are the harder case. A worker that exits between scrapes is never observed at all, and its counters vanish with it — which is fine for rates, since rate() handles disappearing series correctly, and wrong for anything that needs the final value. Where jobs run in genuinely ephemeral processes, a push gateway is the supported answer, with the caveat that it becomes a stateful component of its own and needs its own retention policy.

Two smaller details save time later. Give every worker a stable instance label derived from something meaningful — pod name is usually right — so a series can be correlated with logs. And avoid per-pod panels on dashboards: on a fleet that scales several times an hour, they churn constantly and answer questions nobody asked, while an aggregation by queue stays readable through the same churn.

Retention and Recording Rules

Raw series at fifteen-second resolution are expensive to keep for a year and rarely consulted at that granularity. Recording rules that pre-aggregate the queries dashboards actually use — per-queue rates, ratios and quantiles — let you retain the aggregates for long periods while the raw series expire sooner.

Keep the rule definitions beside the alerts that consume them, so a change to one is reviewed against the other.

Related

• Instrumenting Celery with a Prometheus exporter — the full no-code-change exporter setup.
• Alerting on queue backlog with Prometheus — turning the metrics here into pages that fire at the right time.
• Observability & Monitoring for Job Queues — the four signals and how metrics fit alongside logs and traces.
• Grafana Dashboards for Queues — visualising these series in panels.
• Horizontal Worker Scaling — the scaling decisions these saturation metrics drive.