Celery Beat Periodic Task Scheduling

This guide builds on the worker and broker setup from Celery Architecture & Configuration and the wider Backend Frameworks & Worker Scaling material, focusing on the scheduler that turns Celery from a fire-and-forget job runner into a cron replacement.

Celery Beat is the component that periodically dispatches tasks onto your queues — nightly reports, hourly cache warmers, a sweep every five minutes. The two failure modes engineers hit most are silent: a schedule that simply never fires because Beat isn't running, and the opposite — every task firing two or three times because two Beat processes are alive at once (a leftover from a rolling deploy, or one per replica in a scaled deployment). Both produce no error, just wrong behavior. This page walks through defining schedules with crontab, solar, and timedelta, persisting them in a database for runtime edits, pinning the timezone so "midnight" means what you expect, and guaranteeing exactly one Beat process drives the schedule.

Prerequisites

  • A running Celery app with broker and result backend (see Setting up Celery with Redis broker and RabbitMQ backend).
  • celery>=5.3; for the database-backed scheduler, django-celery-beat on a Django project, or redbeat for a non-Django stack.
  • At least one worker consuming the queues your scheduled tasks route to — Beat only dispatches, it does not execute.
  • A decision on which single host or pod will own the Beat process.
Two places a schedule can live A static schedule in configuration is version-controlled and reviewed, and changing it needs a deploy. A database-backed schedule can be edited at runtime by an operator or an admin interface, which is convenient and means the running schedule is no longer what the repository says. Reviewed in a pull request, or editable at 3am static, in code version-controlled and reviewable a change needs a deploy database-backed editable without a release the repository stops being the truth

Step 1 — Define a static schedule with beat_schedule

The simplest schedule lives in your Celery config as the beat_schedule dict. Each entry names the task, its arguments, and a schedule object. Use a plain float for fixed intervals.

# celeryconfig.py
from celery.schedules import crontab

beat_schedule = {
    "warm-cache-every-30s": {
        "task": "myapp.tasks.warm_cache",
        "schedule": 30.0,                 # seconds; fires every 30s
    },
    "nightly-report": {
        "task": "myapp.tasks.build_report",
        "schedule": crontab(hour=2, minute=0),   # 02:00 every day
        "args": ("daily",),
        "options": {"queue": "reports"},  # route to a dedicated queue
    },
}

The options key accepts the same routing and execution options as apply_async, so you can pin scheduled work to a specific queue and keep it off the path of latency-sensitive jobs.

Step 2 — Use crontab, timedelta, and solar schedule types

Celery ships three schedule constructors that cover almost every recurrence pattern. crontab mirrors Unix cron fields, timedelta expresses fixed intervals readably, and solar fires relative to sunrise/sunset at a geographic point.

# celeryconfig.py
from datetime import timedelta
from celery.schedules import crontab, solar

beat_schedule = {
    "weekday-mornings": {
        "task": "myapp.tasks.send_digest",
        # 07:30 Monday-Friday
        "schedule": crontab(hour=7, minute=30, day_of_week="mon-fri"),
    },
    "every-15-minutes": {
        "task": "myapp.tasks.poll_inbox",
        "schedule": timedelta(minutes=15),   # clearer than 900.0
    },
    "quarter-past-each-hour": {
        "task": "myapp.tasks.reconcile",
        "schedule": crontab(minute=15),      # :15 of every hour
    },
    "at-dusk": {
        "task": "myapp.tasks.dim_lights",
        # sunset at Amsterdam; requires latitude/longitude
        "schedule": solar("sunset", 52.3676, 4.9041),
    },
}

crontab supports ranges and steps just like cron (minute="*/5", day_of_week="mon-fri"). For cron-style scheduling patterns shared across stacks, see cron-style scheduling with Celery Beat.

A schedule that fires is not work that runs Beat publishes on time, but if the target queue has no free workers the job waits like any other message. Schedule adherence therefore has two components: whether the tick fired, and whether capacity existed when it did. Two failure points, one symptom beat fires at 02:00 queue already 40,000 deep runs at 02:47 Give scheduled work its own queue when the clock time matters to anyone.

Step 3 — Pin the timezone

A crontab(hour=2) entry means 02:00 in Celery's configured timezone, which defaults to UTC. If your team thinks in local time, set timezone explicitly so the schedule fires when you intend and survives daylight-saving transitions correctly.

# celeryconfig.py
timezone = "Europe/Amsterdam"   # crontab times are interpreted in this zone
enable_utc = True               # store/transport timestamps in UTC internally

Keep enable_utc = True: timestamps on the wire stay UTC (avoiding ambiguity), while crontab interpretation uses the timezone value. Setting a local timezone without this combination is the usual cause of "the report ran an hour early after the clocks changed."

Step 4 — Start the Beat process

Beat runs as its own process, separate from workers. For development you can embed it in a worker with -B, but never do that in production with more than one worker — every worker would run its own scheduler.

# development only: embed beat in a single worker
celery -A myapp worker -B --loglevel=INFO

# production: run beat as a dedicated, single process
celery -A myapp beat --loglevel=INFO \
    --schedule=/var/run/celery/beat-schedule   # path to the persistent schedule DB

The --schedule file is a small shelve database where the default PersistentScheduler records the last run time of each task, so a Beat restart doesn't re-fire everything or skip a window.

Step 5 — Move schedules into a database with DatabaseScheduler

Editing beat_schedule requires a redeploy. For schedules that operators need to change at runtime — pausing a job, shifting a report time — use a database-backed scheduler. On Django, django-celery-beat stores schedules in tables editable from the admin.

pip install django-celery-beat
# settings.py
INSTALLED_APPS = [
    # ...
    "django_celery_beat",
]
# then: python manage.py migrate

Start Beat with the database scheduler so it reads schedules from the database instead of the config file:

celery -A myapp beat --loglevel=INFO \
    --scheduler=django_celery_beat.schedulers:DatabaseScheduler

Create and toggle schedules programmatically; changes take effect without restarting Beat:

# create a periodic task at runtime
from django_celery_beat.models import PeriodicTask, CrontabSchedule

schedule, _ = CrontabSchedule.objects.get_or_create(hour=3, minute=0)
PeriodicTask.objects.create(
    crontab=schedule,
    name="cleanup-tmp",
    task="myapp.tasks.cleanup",
    enabled=True,
)

For a non-Django stack, redbeat (RedBeatScheduler) stores the schedule in Redis and additionally provides a distributed lock, which directly helps with the duplicate-process problem below.

Recreate for a singleton, never RollingUpdate A rolling update starts the new scheduler before the old one exits, so both are live for a few seconds and any tick in that window fires twice. Recreate terminates the old pod first, accepting a short gap with no scheduler in exchange for never having two. A gap is safe; an overlap is not RollingUpdate overlap — every tick here fires twice Recreate a few seconds with no scheduler — one delayed tick at worst

Step 6 — Guarantee exactly one Beat process

Beat has no built-in leader election with the default scheduler. Two Beat processes mean every task fires twice. Run exactly one. In Kubernetes, that means a Deployment with replicas: 1 and the Recreate strategy so a rolling update never briefly runs two pods.

# k8s/beat-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-beat
spec:
  replicas: 1                 # never more than one beat scheduler
  strategy:
    type: Recreate            # terminate old pod before starting new one (no overlap)
  template:
    spec:
      containers:
        - name: beat
          image: myapp:latest
          command: ["celery", "-A", "myapp", "beat", "--loglevel=INFO"]

If you cannot guarantee a single replica, switch to redbeat, whose RedBeatScheduler acquires a Redis lock so only the lock holder dispatches, making accidental duplicate Beat processes harmless.

Verification

Confirm the schedule is loaded and firing as expected.

On startup, Beat logs each registered entry and its next run time:

# look for "Scheduler: Sending due task ..." lines at the expected wall-clock time
celery -A myapp beat --loglevel=DEBUG

Check that a worker actually received and ran the dispatched task by watching the event stream:

celery -A myapp events --dump   # expect task-received / task-succeeded for the scheduled task

Assert the schedule registers correctly in a test:

# tests/test_schedule.py
from myapp.celery import app

def test_nightly_report_scheduled():
    entry = app.conf.beat_schedule["nightly-report"]
    assert entry["task"] == "myapp.tasks.build_report"
    assert entry["schedule"].hour == {2}       # crontab stores fields as sets
    assert entry["schedule"].minute == {0}

Gotchas & edge cases

  • Two Beat processes = double execution. This is the single most common Beat bug, usually from replicas: 2 or a worker started with -B alongside a standalone Beat. Enforce one scheduler or use redbeat's lock.
  • Timezone defaults to UTC. A crontab(hour=0) you assumed was local midnight fires at UTC midnight. Set timezone explicitly and verify after a daylight-saving change.
  • Beat dispatches, it does not execute. If no worker consumes the target queue, tasks pile up on the broker and "nothing runs." Confirm a worker is subscribed to the queue named in options.
  • Long-running tasks can overlap. Beat fires on schedule regardless of whether the previous run finished. For tasks that must not overlap, guard the body with a distributed lock rather than assuming serialization.
  • A missing or unwritable --schedule file. With the default PersistentScheduler, an unwritable path makes Beat unable to record last-run times, causing re-fires on restart. Ensure the directory exists and is writable.

Operating Beat Alongside the Workers

Beat is deployed with the workers and behaves nothing like them, which is worth making explicit in the manifests rather than leaving to convention.

It is a singleton, so its replica count is one and its deployment strategy recreates rather than rolls. It is a producer rather than a consumer, so its resource requirements are trivial and its capacity is irrelevant to throughput. And its failure mode is silence: when Beat is down nothing errors, nothing queues, and nothing alerts unless you have explicitly checked that expected ticks occurred.

That last property is the one that catches teams. Add a liveness signal driven by tick emission rather than by process existence — a heartbeat metric incremented on each schedule evaluation, alerted on when it stops — so a Beat that is running but stuck is distinguishable from one that is working.

Give scheduled work its own queue. A tick that fires punctually into a saturated queue is late anyway, and separating it makes schedule adherence measurable rather than confounded with whatever burst the general queues are absorbing.

Related