Webhook Retries and Idempotency

Here's a fun fact that will save you a production incident someday: webhook delivery is at-least-once, not exactly-once. Every provider retries failed deliveries. Some retry even when the delivery succeeded but your server was a bit slow responding. Your handler will see duplicates, and it needs to handle them without doing something twice.

At-Least-Once Delivery

Providers guarantee they'll try to deliver every event. They don't guarantee your handler receives it exactly once. This happens more often than you'd think:

• Your server returned 200, but the provider's connection timed out before seeing the response
• A network blip made the provider assume delivery failed
• The provider's retry system fired before the first attempt was confirmed
• You have multiple webhook endpoints subscribed to the same event types

This isn't a bug in the provider — it's a deliberate design choice. Exactly-once delivery across network boundaries is basically impossible without the receiver cooperating. That cooperation is called idempotency.

Event Identity

Every major webhook provider includes a unique identifier with each event:

This identifier is your idempotency key. If you've already processed an event with the same ID, skip it.

Building Idempotent Handlers

The simplest approach that actually works: a processed-events table with a unique constraint.

class ProcessedWebhookEvent < ApplicationRecord
  validates :provider_event_id, uniqueness: { scope: :provider }
end

def handle_event(provider:, event_id:, payload:)
  return if ProcessedWebhookEvent.exists?(provider: provider, provider_event_id: event_id)

  ActiveRecord::Base.transaction do
    ProcessedWebhookEvent.create!(provider: provider, provider_event_id: event_id)
    process_payload(payload)
  end
end

The unique constraint on [provider, provider_event_id] means that even if two duplicate deliveries hit your server at the exact same millisecond, only one will get through. The database does the hard concurrency work for you.

Acknowledge Fast, Process Later

Providers have delivery timeouts — typically 5 to 30 seconds. If your handler takes longer than that, the provider assumes it failed and queues a retry. Now you have the original still processing and a retry coming in. Wonderful.

The fix: acknowledge immediately, process in the background.

class WebhooksController < ApplicationController
  def stripe
    event = verify_and_parse(request)
    StripeWebhookJob.perform_later(event.to_json)
    head :ok
  end
end

Your background job handles the actual processing, including the idempotency check. The webhook endpoint always responds within milliseconds. Provider happy, no retries triggered.

Natural Idempotency

Some operations are naturally idempotent — running them twice produces the same result. UPDATE users SET email = 'new@example.com' WHERE id = 123 is the same whether you run it once or five times. Look for these patterns:

• Setting a status to a final state (marking an order as paid)
• Upserting records by external ID
• Writing to a log or append-only store

Operations that are not naturally idempotent and absolutely need protection:

• Sending emails or notifications (nobody wants three "Payment received!" emails)
• Incrementing counters or balances
• Creating records without deduplication
• Triggering external API calls

Replay Safety

When you're developing with CatchHook's tunnel and your handler processes a webhook live, then you later trigger the same event type again, the idempotency check should handle it. This is actually a great test — if a re-triggered event processes twice, your idempotency logic has a gap.

For development, you might want to clear the processed-events table between test iterations, or use a separate dev database that doesn't share state with your tunnel session.

Provider Retry Behavior

Each provider retries differently. Knowing the schedule helps when you're debugging why your handler got the same event five times.

Stripe

Stripe retries up to 12 times over roughly 3 days with exponential backoff. The first retry comes about an hour after the initial failure. Any non-2xx response counts as a failure.

CatchHook shows each delivery attempt as a separate request with the same event ID, making it easy to see the retry pattern.

GitHub

GitHub retries 3 times within 1 hour, with a 10-second timeout per attempt. 4xx responses are treated as permanent failures (no more retries), while 5xx responses get retried. See GitHub's webhook best practices.

Shopify

Shopify retries up to 19 times over 48 hours. After enough failures, Shopify may just disable the webhook entirely. Keep an eye on webhook/disabled events in your Shopify admin if you're seeing gaps.

Failure Recovery Patterns

Dead Letter Tracking

When your handler rejects an event it can't process — validation error, missing related data, unexpected schema — log it and move on:

def handle_event(event)
  process(event)
rescue => e
  DeadLetterEvent.create!(
    provider: event.provider,
    event_id: event.id,
    payload: event.to_json,
    error: e.message
  )
  head :ok
end

Returning 200 even on failure prevents the provider from retrying an event your code can't handle right now. Fix the bug, then reprocess from the dead letter queue. Fighting the provider's retry logic while also trying to fix a bug is not a good time.

Out-of-Order Delivery

Providers don't guarantee ordering. A customer.subscription.updated event might arrive before customer.subscription.created. Three strategies, in order of practicality:

1. Accept and reconcile — process events based on the provider's event timestamp, not arrival order
2. Fetch current state — when an event arrives, call the provider's API to get the current object state instead of relying on the payload
3. Queue and order — buffer events and process them in timestamp order (complex, rarely worth it)

Option 2 is the most robust in practice. The event tells you what changed; the API tells you the current truth.

Debugging Retries with CatchHook

CatchHook captures every delivery attempt as its own request, so when a provider retries you'll see each one individually. Filter by event type and look for repeated event IDs to spot retry sequences. The request timeline shows exact delivery timing, making it easy to compare what happened on the first attempt versus the retries.