From 30 Minutes to 1: Isolating Heavy Celery Tasks

A clinic admin clicks “Export calls”. The export should take about a minute. It was taking 25 to 30. Worse, while it ran, other things that had nothing to do with exports (webhook processing, notifications, small API-triggered jobs) also slowed to a crawl.

This is a write-up of that investigation at Simbie AI. The fix wasn’t clever. The diagnosis is the useful part, because the same two problems show up in almost every Celery deployment that grew without anyone looking at its queues.

The symptom that gave it away

Slow exports on their own would suggest a slow query. But the fact that unrelated short tasks were also delayed pointed somewhere else: they were waiting for a worker slot, not for the database.

Checking the worker confirmed it. Every API-triggered task, long or short, went to the same default queue, consumed by the same worker deployment.

Problem 1: one queue for everything

With one queue, a worker process that picks up a 25-minute export is unavailable for 25 minutes. Pick up a few exports and you’ve lost most of your capacity. Every short task behind them waits.

This is head-of-line blocking, and it’s the default behaviour if you never set task_routes.

Problem 2: prefetch made it worse

Celery workers prefetch. By default each process reserves worker_prefetch_multiplier = 4 messages ahead of what it’s executing. For short tasks that’s a nice throughput boost. For a mix of short and very long tasks it’s a trap: a process busy with an export is also holding several short tasks it has reserved and can’t run. Other idle processes can’t take them, because they’re already reserved.

So exports blocked their own slots, and quietly held on to work that other processes could have done.

The fix

Route by cost

Tasks are routed to queues by how expensive they are, not by which module they live in:

task_routes = {
    "exports.*":    {"queue": "heavy"},
    "backfills.*":  {"queue": "heavy"},
    # everything latency-sensitive stays on the API queue
    "*":            {"queue": "api"},
}

Separate workers with settings that match the work

# API worker: many short tasks, fail fast, don't hoard
celery -A app worker -Q api   --concurrency=8 --prefetch-multiplier=1 -O fair
 
# Heavy worker: few long tasks, never prefetch, hard time limit
celery -A app worker -Q heavy --concurrency=2 --prefetch-multiplier=1 \
       --time-limit=3600 --soft-time-limit=3300

And in config:

task_acks_late = True              # ack after the task finishes, not when it starts
task_reject_on_worker_lost = True  # requeue if the worker dies mid-task
worker_prefetch_multiplier = 1

acks_late plus a prefetch of 1 means a worker only holds what it’s actually running, and a crashed worker puts its task back on the queue instead of losing it. That only works if tasks are idempotent, which exports are: they write to a new file keyed by the export request id.

Right-size the API worker

The second half of the fix was boring and important. The API worker’s resources and concurrency had been set once and never revisited. We sized concurrency to the real number of tasks it handled and what each one needed, so it wasn’t starved for CPU or memory under normal load.

Make the export itself cheaper

With isolation in place we also looked at the export task, since it now had its own capacity budget:

• Stream rows from the database in batches instead of loading the full result set into memory.
• Write incrementally to the output file.
• Avoid N+1 lookups for per-call details by fetching them in bulk per batch.

Raising the export limit to 100,000 rows was only reasonable once the task streamed instead of buffering.

Result

Call exports went from 25–30 minutes to about one minute, and exports no longer block other API-triggered work, because they physically can’t: they run on a different queue, on a different worker.

A checklist for any Celery deployment

If you’ve inherited a Celery setup, these questions take ten minutes and catch most of it:

1. Is there more than one queue? If not, what’s the longest task, and what’s waiting behind it?
2. What’s the prefetch multiplier on workers that run long tasks? It should almost certainly be 1.
3. Are tasks acks_late, and if so, are they idempotent?
4. Do long tasks have time limits? A stuck task without one holds a slot forever.
5. Can you see queue depth and wait time per queue? If you can’t, you’ll find out about the next one of these from a customer.

The last one is the one I’d fix first. We added queue-level metrics afterwards, and they would have shown this problem long before anyone noticed it in the UI.