Daemonization — Celery 5.5.2 documentation (original) (raw)

This document describes the current stable version of Celery (5.5). For development docs,go here.

Most Linux distributions these days use systemd for managing the lifecycle of system and user services.

You can check if your Linux distribution uses systemd by typing:

$ systemctl --version systemd 249 (v249.9-1.fc35) +PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

If you have output similar to the above, please refer toour systemd documentation for guidance.

However, the init.d script should still work in those Linux distributions as well since systemd provides the systemd-sysv compatibility layer which generates services automatically from the init.d scripts we provide.

If you package Celery for multiple Linux distributions and some do not support systemd or to other Unix systems as well, you may want to refer to our init.d documentation.

Generic init-scripts

See the extra/generic-init.d/ directory Celery distribution.

This directory contains generic bash init-scripts for thecelery worker program, these should run on Linux, FreeBSD, OpenBSD, and other Unix-like platforms.

Init-script: celeryd

Usage:

/etc/init.d/celeryd {start|stop|restart|status}

Configuration file:

/etc/default/celeryd

To configure this script to run the worker properly you probably need to at least tell it where to change directory to when it starts (to find the module containing your app, or your configuration module).

The daemonization script is configured by the file /etc/default/celeryd. This is a shell (sh) script where you can add environment variables like the configuration options below. To add real environment variables affecting the worker you must also export them (e.g., export DISPLAY=":0")

Superuser privileges required

The init-scripts can only be used by root, and the shell configuration file must also be owned by root.

Unprivileged users don’t need to use the init-script, instead they can use the celery multi utility (orcelery worker --detach):

$ celery -A proj multi start worker1
--pidfile="$HOME/run/celery/%n.pid"
--logfile="$HOME/log/celery/%n%I.log"

$ celery -A proj multi restart worker1
--logfile="$HOME/log/celery/%n%I.log"
--pidfile="$HOME/run/celery/%n.pid

$ celery multi stopwait worker1 --pidfile="$HOME/run/celery/%n.pid"

Example configuration

This is an example configuration for a Python project.

/etc/default/celeryd:

Names of nodes to start

most people will only start one node:

CELERYD_NODES="worker1"

but you can also start multiple and configure settings

for each in CELERYD_OPTS

#CELERYD_NODES="worker1 worker2 worker3"

alternatively, you can specify the number of nodes to start:

#CELERYD_NODES=10

Absolute or relative path to the 'celery' command:

CELERY_BIN="/usr/local/bin/celery" #CELERY_BIN="/virtualenvs/def/bin/celery"

App instance to use

comment out this line if you don't use an app

CELERY_APP="proj"

or fully qualified:

#CELERY_APP="proj.tasks:app"

Where to chdir at start.

CELERYD_CHDIR="/opt/Myproject/"

Extra command-line arguments to the worker

CELERYD_OPTS="--time-limit=300 --concurrency=8"

Configure node-specific settings by appending node name to arguments:

#CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"

Set logging level to DEBUG

#CELERYD_LOG_LEVEL="DEBUG"

%n will be replaced with the first part of the nodename.

CELERYD_LOG_FILE="/var/log/celery/%n%I.log" CELERYD_PID_FILE="/var/run/celery/%n.pid"

Workers should run as an unprivileged user.

You need to create this user manually (or you can choose

a user/group combination that already exists (e.g., nobody).

CELERYD_USER="celery" CELERYD_GROUP="celery"

If enabled pid and log directories will be created if missing,

and owned by the userid/group configured.

CELERY_CREATE_DIRS=1

Using a login shell

You can inherit the environment of the CELERYD_USER by using a login shell:

Note that this isn’t recommended, and that you should only use this option when absolutely necessary.

Example Django configuration

Django users now uses the exact same template as above, but make sure that the module that defines your Celery app instance also sets a default value for DJANGO_SETTINGS_MODULEas shown in the example Django project in First steps with Django.

Available options

Init-script: celerybeat

Usage:

/etc/init.d/celerybeat {start|stop|restart}

Configuration file:

/etc/default/celerybeat or/etc/default/celeryd.

Example configuration

This is an example configuration for a Python project:

/etc/default/celerybeat:

Absolute or relative path to the 'celery' command:

CELERY_BIN="/usr/local/bin/celery" #CELERY_BIN="/virtualenvs/def/bin/celery"

App instance to use

comment out this line if you don't use an app

CELERY_APP="proj"

or fully qualified:

#CELERY_APP="proj.tasks:app"

Where to chdir at start.

CELERYBEAT_CHDIR="/opt/Myproject/"

Extra arguments to celerybeat

CELERYBEAT_OPTS="--schedule=/var/run/celery/celerybeat-schedule"

Example Django configuration

You should use the same template as above, but make sure theDJANGO_SETTINGS_MODULE variable is set (and exported), and thatCELERYD_CHDIR is set to the projects directory:

export DJANGO_SETTINGS_MODULE="settings"

CELERYD_CHDIR="/opt/MyProject"

Available options

Troubleshooting

If you can’t get the init-scripts to work, you should try running them in verbose mode:

sh -x /etc/init.d/celeryd start

This can reveal hints as to why the service won’t start.

If the worker starts with “OK” but exits almost immediately afterwards and there’s no evidence in the log file, then there’s probably an error but as the daemons standard outputs are already closed you’ll not be able to see them anywhere. For this situation you can use the C_FAKEFORK environment variable to skip the daemonization step:

C_FAKEFORK=1 sh -x /etc/init.d/celeryd start

and now you should be able to see the errors.

Commonly such errors are caused by insufficient permissions to read from, or write to a file, and also by syntax errors in configuration modules, user modules, third-party libraries, or even from Celery itself (if you’ve found a bug you should report it).

Usage systemd

Usage:

systemctl {start|stop|restart|status} celery.service

Configuration file:

/etc/conf.d/celery

Service file: celery.service

This is an example systemd file:

/etc/systemd/system/celery.service:

[Unit] Description=Celery Service After=network.target

[Service] Type=forking User=celery Group=celery EnvironmentFile=/etc/conf.d/celery WorkingDirectory=/opt/celery ExecStart=/bin/sh -c '${CELERY_BIN} -A CELERYAPPmultistartCELERY_APP multi start CELERYA​PPmultistartCELERYD_NODES
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE}
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE}
--loglevel="${CELERYD_LOG_LEVEL}"' ExecReload=/bin/sh -c '${CELERY_BIN} -A CELERYAPPmultirestartCELERY_APP multi restart CELERYA​PPmultirestartCELERYD_NODES
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE}
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS' Restart=always

[Install] WantedBy=multi-user.target

Once you’ve put that file in /etc/systemd/system, you should runsystemctl daemon-reload in order that Systemd acknowledges that file. You should also run that command each time you modify it. Use systemctl enable celery.service if you want the celery service to automatically start when (re)booting the system.

Optionally you can specify extra dependencies for the celery service: e.g. if you use RabbitMQ as a broker, you could specify rabbitmq-server.service in both After= and Requires=in the [Unit] systemd section.

To configure user, group, chdir change settings:User, Group, and WorkingDirectory defined in/etc/systemd/system/celery.service.

You can also use systemd-tmpfiles in order to create working directories (for logs and pid).

file:

/etc/tmpfiles.d/celery.conf

d /run/celery 0755 celery celery - d /var/log/celery 0755 celery celery -

Example configuration

This is an example configuration for a Python project:

/etc/conf.d/celery:

Name of nodes to start

here we have a single node

CELERYD_NODES="w1"

or we could have three nodes:

#CELERYD_NODES="w1 w2 w3"

Absolute or relative path to the 'celery' command:

CELERY_BIN="/usr/local/bin/celery" #CELERY_BIN="/virtualenvs/def/bin/celery"

App instance to use

comment out this line if you don't use an app

CELERY_APP="proj"

or fully qualified:

#CELERY_APP="proj.tasks:app"

How to call manage.py

CELERYD_MULTI="multi"

Extra command-line arguments to the worker

CELERYD_OPTS="--time-limit=300 --concurrency=8"

- %n will be replaced with the first part of the nodename.

- %I will be replaced with the current child process index

and is important when using the prefork pool to avoid race conditions.

CELERYD_PID_FILE="/var/run/celery/%n.pid" CELERYD_LOG_FILE="/var/log/celery/%n%I.log" CELERYD_LOG_LEVEL="INFO"

you may wish to add these options for Celery Beat

CELERYBEAT_PID_FILE="/var/run/celery/beat.pid" CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"

Service file: celerybeat.service

This is an example systemd file for Celery Beat:

/etc/systemd/system/celerybeat.service:

[Unit] Description=Celery Beat Service After=network.target

[Service] Type=simple User=celery Group=celery EnvironmentFile=/etc/conf.d/celery WorkingDirectory=/opt/celery ExecStart=/bin/sh -c '${CELERY_BIN} -A ${CELERY_APP} beat
--pidfile=${CELERYBEAT_PID_FILE}
--logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL}' Restart=always

[Install] WantedBy=multi-user.target

Once you’ve put that file in /etc/systemd/system, you should runsystemctl daemon-reload in order that Systemd acknowledges that file. You should also run that command each time you modify it. Use systemctl enable celerybeat.service if you want the celery beat service to automatically start when (re)booting the system.

Running the worker with superuser privileges (root)

Running the worker with superuser privileges is a very dangerous practice. There should always be a workaround to avoid running as root. Celery may run arbitrary code in messages serialized with pickle - this is dangerous, especially when run as root.

By default Celery won’t run workers as root. The associated error message may not be visible in the logs but may be seen if C_FAKEFORKis used.

To force Celery to run workers as root use C_FORCE_ROOT.

When running as root without C_FORCE_ROOT the worker will appear to start with “OK” but exit immediately after with no apparent errors. This problem may appear when running the project in a new development or production environment (inadvertently) as root.

https://pypi.org/project/supervisor/

launchd (macOS)