Task Priority In Celery With Redis

I would like to implement a distributed job execution system with celery. Given that rabbitMQ doesn't support priorities and I'm painfully needing this feature, I turned to celery+

Solution 1:

The Celery Redis transport does honor the priority field, but Redis itself has no notion of priorities.

The priority support is implemented by creating n lists for each queue and using that order in the BRPOP command. I say n here because even though there are 10 (0-9) priority levels, these are consolidated into 4 levels by default to save resources. This means that a queue named celery will really be split into 4 queues:

['celery0', 'celery3`, `celery6`, `celery9`]

If you want more priority levels you can set the priority_steps transport option:

BROKER_TRANSPORT_OPTIONS = {
    'priority_steps': list(range(10)),
}

That said, note that this will never be as good as priorities implemented at the server level, and may be approximate at best. But it may still be good enough for your application.

Solution 2:

Celery docs about redis message priorities is here redis-message-priorities,you can customize priority levels.Take 10 for example:

1. set the priority_steps transport option

app.conf.broker_transport_options = {
    'priority_steps': list(range(10)),
    'queue_order_strategy': 'priority',
}

2. start celery worker in normal way

celery -A tasks worker --loglevel=info

3. calling tasks, 0 being highest priority and 9 being lowest priority

custom_priority=5 
task.apply_async(args=[arg1, arg2], kwargs={'kwarg1': 'x', 'kwarg2': 'y'},priority=custom_priority)