Concurrency with gevent — Celery 5.5.2 documentation (original) (raw)

This document describes the current stable version of Celery (5.5). For development docs,go here.

Introduction¶

The gevent homepage describes it a coroutine -based Python networking library that usesgreenlet to provide a high-level synchronous API on top of the libevor libuv event loop.

Features include:

• Fast event loop based on libev or libuv.
• Lightweight execution units based on greenlets.
• API that reuses concepts from the Python standard library (for examples there are events andqueues).
• Cooperative sockets with SSL support
• Cooperative DNS queries performed through a threadpool, dnspython, or c-ares.
• Monkey patching utility to get 3rd party modules to become cooperative
• TCP/UDP/HTTP servers
• Subprocess support (through gevent.subprocess)
• Thread pools

gevent is inspired by eventlet but features a more consistent API, simpler implementation and better performance. Read why others use gevent and check out the list of the open source projects based on gevent.

Enabling gevent¶

You can enable the gevent pool by using thecelery worker -P gevent or celery worker --pool=geventworker option.

$ celery -A proj worker -P gevent -c 1000

Examples¶

See the gevent examples directory in the Celery distribution for some examples taking use of Eventlet support.

Known issues¶

There is a known issue using python 3.11 and gevent. The issue is documented here and addressed in a gevent issue. Upgrading to greenlet 3.0 solves it.