Stuck Calls: Fixing a Webhook Race With a State Machine and a Backfill

A small percentage of calls on our platform would get stuck. The call had clearly finished (there was a transcript, the patient had hung up) but in our system it sat at ended forever and never moved to analyzed. Because analysis drives the call summary, the review pipeline and analytics, a stuck call was effectively invisible to the clinic.

It wasn’t consistent, which is the usual sign of a race.

What the system assumed

After a call, the voice runtime sends a series of webhooks. In the order you’d expect, call_ended arrives first and call_analyzed arrives some seconds later, once post-call analysis is done.

The handler was simple: take the event type, map it to a status, write it.

UPDATE calls SET status = :status_from_event, updated_at = now() WHERE id = :id;

Last write wins. That’s correct only if events always arrive in order, exactly once.

What actually happened

They didn’t always arrive in order. Short calls in particular could be analysed almost immediately, and network timing, retries or our own handler latency meant call_analyzed was sometimes processed before call_ended. The sequence then went: status becomes analyzed, then call_ended is processed and overwrites it back to ended. Nothing else was coming, so the call stayed there.

Fix 1: transitions only move forward

The call lifecycle is a sequence: registered → ongoing → ended → analyzed → exported. A late event for an earlier state should never move a call backwards. We encoded that directly in the update, so the database enforces it atomically:

UPDATE calls
SET status = :new_status, updated_at = now()
WHERE id = :id
  AND status_rank(status) < status_rank(:new_status);

If zero rows are updated, the event is stale. We log it and return 200 so the sender doesn’t retry, because retrying a stale event will never help.

Two things worth noting:

• The rank lives in one place (a small SQL function or a constant shared by the code), not as string comparisons scattered around handlers.
• Data from a late event can still be merged. A late call_ended might carry the final duration. “Don’t move the status backwards” is not the same as “ignore the payload”; fields are merged where they’re newer or missing.

Fix 2: store every event

We started writing each incoming webhook to a webhook_events table before processing it, with a unique key on (call_id, event_type).

That gives us three things for very little cost:

• Idempotency. A duplicate delivery hits the unique constraint and is dropped.
• An audit trail. When a call looks wrong, we can see exactly which events arrived and when.
• Replay. If a handler had a bug, we can fix it and reprocess stored events instead of asking the provider to resend them.

Fix 3: a reconciler that doesn’t trust webhooks at all

Ordering bugs are one way webhooks fail. The other is that they don’t arrive: our service was deploying, a node restarted, the provider had an incident. So the last piece doesn’t depend on webhooks.

Every few minutes a reconciler looks for calls that should have reached a terminal state by now and haven’t:

export async function reconcileStuckCalls() {
  const stuck = await calls.find({
    statusNotIn: ["analyzed", "exported"],
    endedBefore: minutesAgo(15),
    limit: 200,
  });
 
  for (const call of stuck) {
    const remote = await voiceProvider.getCall(call.providerCallId); // source of truth
    await applyRemoteState(call, remote);   // same monotonic update path as webhooks
  }
}

The important detail is that it uses the same update path as the webhook handler. There’s one way state changes, whether it was pushed or pulled.

Backfilling downtime

The same mechanism handles outages. If we know our webhook endpoint was unavailable between two times, a backfill job lists calls from the provider for that window and runs them through applyRemoteState. Calls that happened while we were down get processed as if nothing went wrong, just later.

Watching it

We added a metric for calls in a non-terminal state older than the threshold, plus heartbeat checks on the reconciler itself, both on the system health dashboard. The metric should sit at zero. If it doesn’t, something upstream changed, and we know before a clinic does.

The general lesson

For any system fed by webhooks, I now start from three assumptions:

1. Events will arrive out of order. Make state transitions monotonic.
2. Events will arrive more than once. Store them with a unique key.
3. Events will not arrive. Have a reconciler that pulls from the source of truth.

If the design is correct with all three, webhooks become what they should be: a fast path, not the only path.