JB

Discord's March 2026 Voice Outage: When Recovery Became the Failure

July 11, 2026

Overview

On March 25, 2026, Discord’s voice and video systems were degraded for more than three hours. The official incident report contains a lot of Discord-specific terminology, but the core failure pattern is easier to understand if we step back from the names of individual services.

A stateful system was scaled down before its live sessions had safely moved elsewhere. That disconnected a large number of users at the same time. Those users then tried to reconnect together, which turned recovery into a new source of load. The reconnect wave created more internal events, more downstream work, more queues, and more retries than the system could process in time.

TLDR: The outage was not just “some sessions died.” The deeper issue was that a sudden loss of state created a thundering herd of recovery work. That recovery work fanned out across downstream systems, overloaded shared bottlenecks, and caused queues to grow faster than they could drain. Discord recovered by reducing incoming work, adding capacity, and bringing systems back gradually instead of letting all recovery traffic back in at once.

Official Discord engineering write-up

The Simplified Failure Chain

The whole incident can be reduced to this chain:

Stateful instances were scaled down
    ↓
Live sessions terminated before handoff completed
    ↓
Clients reconnected together
    ↓
Reconnect work overloaded the existing systems
    ↓
Each reconnect generated more downstream work
    ↓
Downstream queues grew faster than they drained
    ↓
Recovery attempts replayed the same overload

That is the shape of the incident. The individual Discord components matter less than the pattern: a system lost state, then the act of rebuilding that state became more expensive than the system could handle.

This is why the outage is interesting. The initial failure was serious, but the long recovery came from the system’s inability to absorb the recovery workload.

Why Scaling Down a Stateful System Is Risky

Scaling down a stateless service is usually straightforward. If one web server disappears, the next request can go to another web server. The server did not own anything unique.

Scaling down a stateful service is different. A stateful instance owns live things: sessions, connections, leases, in-flight jobs, local coordination state, or timers. If the instance is terminated before those things move, the system does not merely lose capacity. It loses live state.

Stateless instance
┌───────────────────────┐
│ Handles requests      │
│ Owns no unique state  │
└───────────────────────┘

Terminate it:
Next request goes elsewhere


Stateful instance
┌───────────────────────┐
│ Owns live sessions    │
│ Tracks active users   │
│ Coordinates recovery  │
└───────────────────────┘

Terminate it too early:
Sessions disappear

The safe operation is not “stop this instance.” The safe operation is:

Stop accepting new ownership
    ↓
Move existing owned state
    ↓
Prove owned state is zero
    ↓
Terminate the instance

The important word is prove. Waiting a fixed amount of time is not the same thing as proving the instance drained.

Note: This is simplified pseudocode to illustrate the concept, not Discord’s implementation.

type StatefulInstance = {
    id: string;
    activeSessions: number;
    handoffsInProgress: number;
};

async function safelyRemoveInstance(instance: StatefulInstance): Promise<void> {
    await stopAcceptingNewSessions(instance.id);
    await handoffSessionsToOtherInstances(instance.id);

    const latest = await getInstanceState(instance.id);

    if (latest.activeSessions > 0 || latest.handoffsInProgress > 0) {
        throw new Error("Refusing to terminate instance that still owns state");
    }

    await terminateInstance(instance.id);
}

The bug class is not limited to WebSocket sessions. The same idea appears when terminating workers that own queue leases, database coordinators that own partitions, or stream processors that own shards.

The First Cascade: The Reconnect Herd

Once sessions disappeared, clients did what clients are supposed to do: they tried to reconnect.

That behaviour is normal and usually helpful. If your Wi-Fi drops for a few seconds, the app reconnects and you continue. The problem appears when many clients do this at the same time.

One user disconnects:
1 reconnect attempt

Many users disconnect together:
Thousands or millions of reconnect attempts

This is a thundering herd. The users are not malicious. The clients are not broken. Each individual client is behaving reasonably, but their combined behaviour creates a sudden traffic spike.

The system now has two jobs:

  1. Continue serving users who are still connected.
  2. Rebuild sessions for users who were disconnected.

Those two jobs compete for the same capacity. If too much reconnect work is admitted at once, the system can spend so much effort recovering lost sessions that it harms healthy sessions too.

That is the first feedback loop:

Sessions die
    ↓
Clients reconnect
    ↓
Reconnect work consumes capacity
    ↓
More connections become unhealthy
    ↓
More clients reconnect

This is why rate limits and backpressure matter. They are not only about protecting against abusive traffic. They also protect a system from its own well-behaved clients during recovery.

Rate Limits Are Brakes, Not Magic

The public Discord write-up says a start-session rate limit existed, but it was not tuned correctly for the system at the time of the incident. We do not need to know the exact internal placement of that limit to understand the lesson.

The key point is simple: a rate limit only protects the system if it limits the work that actually exhausts the system.

For example, imagine a reconnect path like this:

Client reconnect
    ↓
Accept reconnect request
    ↓
Create or resume session
    ↓
Notify related systems
    ↓
Restore voice state

If the system can safely create 10,000 sessions per second, allowing 50,000 session starts per second is not backpressure. It is permission to overload.

The same problem happens when the limit is attached to the wrong unit. A limit on parent work can still allow too much child work.

Allow 1 recovery job
    ↓
That job creates 500 downstream operations

That is not really “1 operation.” It is one doorway into a much larger amount of work.

Note: This is simplified pseudocode to illustrate the concept, not Discord’s implementation.

// Problem: the parent operation is limited, but the expensive fan-out is not.
async function recoverUserSession(userId: string): Promise<void> {
    await rateLimit("session-recovery");

    await recreateSession(userId);

    const downstreamTasks = await getRecoveryTasks(userId);

    await Promise.all(
        downstreamTasks.map(task => processRecoveryTask(task))
    );
}

A safer design limits the expensive unit of work, not only the outer request.

// Better: each downstream operation also has to pass through backpressure.
async function recoverUserSession(userId: string): Promise<void> {
    await rateLimit("session-recovery");

    await recreateSession(userId);

    const downstreamTasks = await getRecoveryTasks(userId);

    for (const task of downstreamTasks) {
        await rateLimit("downstream-recovery-work");
        await processRecoveryTask(task);
    }
}

The goal is not to make recovery instant. The goal is to make recovery stable. A slower recovery that completes is better than an aggressive recovery that keeps knocking the system back over.

The Second Cascade: Fan-Out

A lost session is not just one lost connection. In a real-time product, one session can be attached to many pieces of state:

  • User presence.
  • Message delivery.
  • Server membership.
  • Voice participation.
  • Call ownership.
  • Cleanup timers.
  • Internal monitors.

When the session disappears, other systems need to react. When the user reconnects, those systems need to rebuild state.

One lost session
    ├── Mark user disconnected
    ├── Stop sending events to that session
    ├── Update voice membership
    ├── Possibly stop an empty call
    ├── Recreate state when the user returns
    └── Notify other internal processes

This is fan-out: one external event becomes many internal events.

Fan-out is not inherently bad. It is how distributed systems keep related state up to date. The danger is that fan-out multiplies load during incidents.

If one lost session creates five internal operations, losing one million sessions does not create one million units of work. It creates five million units of work, plus retries, plus cleanup, plus recovery.

That is where the outage becomes more than a reconnect problem. The reconnect herd pushed work into downstream systems. Those downstream systems then had to process a much larger event stream than they normally saw.

Backlogs: The Point Where Recovery Stops Recovering

The most important queueing idea in this incident is simple:

If work arrives faster than it completes, the backlog grows.

At first, that sounds obvious. The hard part is what happens next. A growing backlog often makes each unit of work slower. Queues get longer, memory pressure rises, searches through pending work take longer, timeouts fire, clients retry, and operators restart services that immediately receive the same flood again.

Arrival rate:    10,000 operations per second
Completion rate:  6,000 operations per second

Backlog growth:   4,000 operations per second

That system is not “a bit behind.” It is unstable. Every second makes recovery harder.

The only ways out are:

  • Reduce the arrival rate.
  • Increase the completion rate.
  • Drop or shed non-essential work.
  • Restart gradually only after demand is under control.

Restarting alone does not fix the math.

Restart clears local queue
    ↓
Instance looks healthy
    ↓
Backlogged work rushes back in
    ↓
Queue grows again
    ↓
Instance falls behind again

That is why “turn it off and on again” can make an overloaded distributed system worse. A restart clears symptoms on one node, but it does not remove the demand that created the symptoms.

Shared Bottlenecks Make Cascades Worse

Highly concurrent systems still have serial bottlenecks. There is often a shared place where many parallel tasks must pass through a smaller number of workers, queues, locks, connection pools, or supervisors.

Many recovery tasks
      │ │ │ │ │
      ▼ ▼ ▼ ▼ ▼
┌───────────────────────┐
│ Shared bottleneck     │
│ queue / pool / lock   │
└──────────┬────────────┘
           ▼
    Downstream system

The bottleneck may be perfectly fine during normal traffic. It may even be invisible. It only becomes obvious when recovery traffic arrives in a burst.

This is one reason incidents are so frustrating. Average CPU can look acceptable while the system is functionally unhealthy. The real problem may be queue age, mailbox length, connection-pool wait time, or the number of operations already past their deadline.

For recovery, these are often better questions than “is the process running?”

  • Is the oldest queued item getting older?
  • Is completion rate higher than arrival rate?
  • Are retries increasing or decreasing?
  • Are health checks competing with normal work?
  • Can the system still shed load?

If health checks, service registration, cancellation, or rollback use the same saturated path as normal application work, the system can lose its ability to recover even while the process is technically still alive.

Why Recovery Required Reducing Demand

Discord eventually recovered by doing three things together:

  1. Reducing incoming recovery work.
  2. Adding more processing capacity.
  3. Restarting systems gradually.

That combination matters. Adding capacity helps only if the new capacity is not immediately overwhelmed. Restarting helps only if the restarted instance is not immediately handed the same unbounded backlog. Rate limits help only if they are placed close enough to the expensive work.

The recovery shape was:

Before mitigation
Work arrives faster than it completes
Backlog grows

After mitigation
Work completes faster than it arrives
Backlog drains

Once the system crossed that line, recovery became possible.

The Core Lessons

1. Scale Down by State, Not by Instance Count

Reducing instance count is not the real operation in a stateful system. The real operation is moving ownership.

If an instance owns 100,000 live sessions, then terminating that instance is a 100,000-session migration. Treat it that way.

The safe invariant is not:

The shutdown timer expired.

The safe invariant is:

This instance owns zero live sessions.

2. Recovery Traffic Is Production Traffic

Reconnects, retries, replays, and rebuilds are not separate from production load. They are production load, often at the worst possible time.

Design recovery paths with the same seriousness as normal request paths.

3. Rate Limit the Expensive Work

A limit at the edge may not protect a downstream bottleneck. A limit on parent work may not control child fan-out.

Useful limits are attached to the scarce resource:

  • Session creation.
  • Downstream API calls.
  • Voice or media placement.
  • Database writes.
  • Queue consumers.
  • Connection creation.

4. Watch Drain Rate, Not Just Error Rate

During overload, the system can be failing before errors spike. If queue age is growing and completion rate is below arrival rate, the system is already in trouble.

Good overload dashboards show:

  • Arrival rate.
  • Completion rate.
  • Oldest queued item age.
  • Retry volume.
  • Connection-pool wait time.
  • Dropped or shed work.
  • Estimated time to drain.

5. Preserve a Recovery Lane

Health checks, lease renewal, service discovery, cancellation, and admin controls should not be starved by normal application work.

Sometimes the fix is not glamorous. A separate connection pool, reserved worker capacity, or priority queue can be the difference between “overloaded but recoverable” and “alive but unable to heal.”

Final Mental Model

The incident is easiest to understand as a capacity failure during recovery:

The system lost state quickly.
The system tried to rebuild state quickly.
Rebuilding state created more work than the system could process.
That work fanned out into downstream systems.
Queues grew faster than they drained.
Recovery only worked after demand was reduced and capacity was increased.

That is the general lesson. Discord’s scale made the numbers enormous, but the pattern applies to much smaller systems too: WebSocket services, queue workers, stream processors, payment orchestrators, cache rebuilds, and deployment systems can all fail this way.

Learn More