Webhook Handling Best Practices for Payment Systems

Webhooks Are Your Source of Truth

In payment systems, webhooks carry the authoritative state. A customer might close their browser after payment, your redirect might fail, or the confirmation page might error out. None of that matters if your webhook handler correctly processes the payment notification from your PSP.

Getting webhook handling wrong leads to unfulfilled orders, duplicate charges, or missed cancellations. These are the bugs that cost real money and destroy customer trust.

Webhook Architecture

Receive Fast, Process Later

Your webhook endpoint has one job: acknowledge receipt and hand off processing. PSPs expect a timely HTTP 200 response. If your endpoint takes too long, the PSP will time out and retry, potentially creating duplicate deliveries.

// Webhook controller: receive and dispatch
public function handle(Request $request): Response
{
    // Store raw payload for debugging
    WebhookLog::create([
        'source' => 'mollie',
        'payload' => $request->getContent(),
        'headers' => $request->headers->all(),
        'received_at' => now(),
    ]);

    // Dispatch async job
    ProcessPaymentNotification::dispatch(
        $request->input('id'),
    );

    return response('', 200);
}

Always store the raw payload. Before any processing, write the incoming request to a log table. This is your forensic record when things go wrong.

Verify Before Processing

Never trust a webhook payload without verification. Attackers can send fake webhooks to your endpoint to trigger unauthorized actions.

Signature verification (preferred): Most PSPs sign webhook payloads with a shared secret or API key. Verify the signature before processing:

public function verifySignature(Request $request): void
{
    $signature = $request->header('X-Webhook-Signature');
    $payload = $request->getContent();
    $expected = hash_hmac('sha256', $payload, config('services.psp.webhook_secret'));

    if (! hash_equals($expected, $signature)) {
        abort(401, 'Invalid webhook signature');
    }
}

API callback verification (Mollie pattern): Mollie sends only the payment ID in the webhook. You then call the Mollie API to fetch the full payment details. This design inherently verifies authenticity because you are fetching data from Mollie's API, not trusting the webhook payload.

Idempotent Processing

Webhooks will be delivered more than once. PSPs retry on timeouts, network errors, or non-200 responses. Your handler must produce the same result regardless of how many times it runs.

State-Check Pattern

Before applying a state change, check whether it has already been applied:

public function processPayment(string $paymentId): void
{
    $payment = $this->psp->getPayment($paymentId);
    $order = Order::where('external_payment_id', $paymentId)->firstOrFail();

    // Idempotency check: is this state already applied?
    if ($order->payment_status === $payment->status) {
        return; // Already processed
    }

    // Only allow valid state transitions
    if (! $order->canTransitionTo($payment->status)) {
        Log::warning("Invalid transition", [
            'order' => $order->id,
            'from' => $order->payment_status,
            'to' => $payment->status,
        ]);
        return;
    }

    DB::transaction(function () use ($order, $payment) {
        $order->update(['payment_status' => $payment->status]);
        // Trigger side effects (fulfillment, notifications)
    });
}

Idempotency Key Pattern

For operations that create records (like issuing a refund or generating a credit), use an idempotency key to prevent duplicates:

$key = "refund:{$paymentId}:{$amount}";

$existingRefund = Refund::where('idempotency_key', $key)->first();

if ($existingRefund) {
    return $existingRefund;
}

$refund = Refund::create([
    'idempotency_key' => $key,
    'payment_id' => $paymentId,
    'amount' => $amount,
]);

Handling Out-of-Order Delivery

Webhooks may arrive in a different order than the events occurred. A "paid" webhook might arrive before a "created" webhook, or a "refunded" webhook might arrive before "paid."

Timestamp-Based Ordering

Include the event timestamp in your processing logic:

public function processEvent(WebhookEvent $event): void
{
    $order = Order::find($event->orderId);

    // Skip if we have already processed a newer event
    if ($order->last_event_at && $event->occurredAt->isBefore($order->last_event_at)) {
        Log::info("Skipping stale event", [
            'order' => $order->id,
            'event_time' => $event->occurredAt,
            'last_processed' => $order->last_event_at,
        ]);
        return;
    }

    // Process and update timestamp
    $order->update([
        'payment_status' => $event->status,
        'last_event_at' => $event->occurredAt,
    ]);
}

Fetch-and-Compare Pattern

The simplest approach for most payment webhooks: ignore the webhook payload entirely and fetch the current state from the PSP's API. The API always returns the latest state, so ordering does not matter:

// The webhook says "something happened to payment X"
// We don't care what it says happened - we ask the PSP directly
$currentState = $this->psp->getPayment($paymentId);
$this->applyState($order, $currentState);

This is the approach Mollie's webhook design encourages, and it is the most robust.

Failure Recovery

Retry With Backoff

If your processing fails (database error, downstream service unavailable), the job should retry with exponential backoff:

class ProcessPaymentNotification implements ShouldQueue
{
    public int $tries = 8;
    public int $maxExceptions = 3;

    public function backoff(): array
    {
        return [30, 60, 300, 600, 1800, 3600, 7200, 14400];
    }
}

Dead Letter Handling

After all retries are exhausted, the webhook lands in a dead letter queue. You need a process (automated or manual) to review and reprocess failed webhooks:

public function failed(Throwable $exception): void
{
    FailedWebhook::create([
        'payment_id' => $this->paymentId,
        'exception' => $exception->getMessage(),
        'failed_at' => now(),
    ]);

    // Alert the team
    Notification::route('slack', config('services.slack.payments'))
        ->notify(new WebhookProcessingFailed($this->paymentId, $exception));
}

Reconciliation as Safety Net

Even with perfect webhook handling, run daily reconciliation. Fetch recent payments from your PSP's API and compare against your local records. Flag discrepancies for manual review:

// Daily reconciliation job
$pspPayments = $this->psp->listPayments(['from' => now()->subDay()]);

foreach ($pspPayments as $pspPayment) {
    $local = Order::where('external_payment_id', $pspPayment->id)->first();

    if (! $local || $local->payment_status !== $pspPayment->status) {
        ReconciliationMismatch::create([
            'payment_id' => $pspPayment->id,
            'psp_status' => $pspPayment->status,
            'local_status' => $local?->payment_status,
        ]);
    }
}

Monitoring

Track these webhook metrics:

• Delivery latency (time between event and webhook receipt)
• Processing success rate (percentage processed without errors)
• Retry rate (percentage requiring retries)
• Dead letter rate (percentage failing all retries)
• Reconciliation mismatch rate (percentage of discrepancies found in daily check)

Alert on anomalies. A sudden spike in failed webhooks usually indicates either a code deployment issue or a PSP-side change.

Webhooks are infrastructure, not features. Treat them with the same rigor you apply to database transactions: verify, process idempotently, handle failures gracefully, and reconcile regularly. Your payment system's reliability depends on it.