Celery Worker and Beat

Celery is a distributed task queue system for processing background jobs. django-prodserver supports both Celery Worker (for processing tasks) and Celery Beat (for scheduling periodic tasks).

Use when: Background tasks, scheduled jobs, distributed processing Don’t use when: Simple tasks without external deps (use Django Tasks)

Installation

pip install django-prodserver[celery]

# Plus a broker
pip install redis  # or rabbitmq

Setup

Celery App

# myproject/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

app = Celery('myproject')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

# myproject/__init__.py
from .celery import app as celery_app
__all__ = ('celery_app',)

Django Settings

CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'

Worker

PRODUCTION_PROCESSES = {
    "worker": {
        "BACKEND": "django_prodserver.backends.workers.celery.CeleryWorker",
        "APP": "myproject.celery.app",
        "ARGS": {
            "concurrency": "4",
            "loglevel": "info",
        }
    }
}

Worker ARGS

Argument	Default	Description
concurrency	CPU count	Worker processes/threads
loglevel	info	Log level
pool	prefork	Pool type (prefork, gevent, solo)
queues	All	Comma-separated queue names
max-tasks-per-child	None	Restart after N tasks
max-memory-per-child	None	Restart after N KB
time-limit	None	Hard task timeout
soft-time-limit	None	Soft task timeout

Beat (Scheduler)

PRODUCTION_PROCESSES = {
    "beat": {
        "BACKEND": "django_prodserver.backends.workers.celery.CeleryBeat",
        "APP": "myproject.celery.app",
        "ARGS": {"loglevel": "info"},
    }
}

Important: Run only ONE beat instance per application.

Beat ARGS

Argument	Default	Description
loglevel	info	Log level
scheduler	Default	Scheduler class

Flower (Monitoring)

Flower is a web-based tool for monitoring and administering Celery clusters. The CeleryFlower backend starts Flower via the flower subcommand registered on your Celery app.

Installation

pip install django-prodserver[flower]

Configuration

PRODUCTION_PROCESSES = {
    "flower": {
        "BACKEND": "django_prodserver.backends.servers.flower.CeleryFlower",
        "APP": "myproject.celery.app",
        "ARGS": {"port": "5555", "address": "0.0.0.0"},
    }
}

python manage.py server flower

Flower ARGS

Argument	Default	Description
port	5555	Port the Flower web server listens on
address	0.0.0.0	Address to bind to
url_prefix	None	Run Flower under a URL prefix
basic_auth	None	user:password pairs for HTTP basic auth
persistent	False	Persist monitoring data across restarts
db	flower	Database file when persistent is enabled
max_tasks	10000	Maximum number of tasks to keep in memory

See the Flower documentation for the full list of options. Options can also be supplied via FLOWER_* environment variables.

Examples

Full Stack

PRODUCTION_PROCESSES = {
    "web": {
        "BACKEND": "django_prodserver.backends.servers.gunicorn.GunicornServer",
        "ARGS": {"bind": "0.0.0.0:8000", "workers": "4"},
    },
    "worker": {
        "BACKEND": "django_prodserver.backends.workers.celery.CeleryWorker",
        "APP": "myproject.celery.app",
        "ARGS": {"concurrency": "4", "max-tasks-per-child": "1000"},
    },
    "beat": {
        "BACKEND": "django_prodserver.backends.workers.celery.CeleryBeat",
        "APP": "myproject.celery.app",
        "ARGS": {"loglevel": "info"},
    },
}

Multiple Queues

PRODUCTION_PROCESSES = {
    "default_worker": {
        "BACKEND": "django_prodserver.backends.workers.celery.CeleryWorker",
        "APP": "myproject.celery.app",
        "ARGS": {"queues": "default", "concurrency": "4"},
    },
    "priority_worker": {
        "BACKEND": "django_prodserver.backends.workers.celery.CeleryWorker",
        "APP": "myproject.celery.app",
        "ARGS": {"queues": "priority", "concurrency": "2"},
    },
}

Gevent Pool

pip install celery[gevent]

"ARGS": {"pool": "gevent", "concurrency": "100"}

Troubleshooting

Broker connection failed: Check Redis/RabbitMQ is running and CELERY_BROKER_URL

Tasks not discovered: Ensure autodiscover_tasks() and tasks.py in apps

Memory leaks: Add max-tasks-per-child

Duplicate scheduled tasks: Run only one beat instance

Links

• Celery Docs

• Django Integration