Webhooks Shouldn't Be an Afterthought

Every API platform eventually ships webhooks. The pattern is always the same: an engineer spends a sprint adding a basic webhook system — store a URL, POST a JSON payload when events happen, maybe retry once or twice on failure. Ship it. Move on.

Six months later, customers are asking why they missed events. The webhook table has 50 million rows and no retention policy. A single endpoint that returns 500s is consuming your retry queue. And nobody can answer the question "did this event get delivered?"

Webhooks are deceptively simple to build and deceptively hard to operate. We decided to treat them as first-class infrastructure in Zyphr, not a checkbox feature.

What "first-class" means

When we say Zyphr's webhook system is first-class, we mean it has the same operational rigor as the rest of the platform. Here's what that includes.

Cryptographic signing

Every webhook payload is signed with HMAC-SHA256 using a per-webhook secret. The signature is sent in the X-Zyphr-Signature header, along with a timestamp to prevent replay attacks.

// Verify a webhook in your handler (Standard Webhooks spec)
import express from 'express';
import { WebhooksResource, WebhookVerificationError } from '@zyphr-dev/node-sdk';

const WEBHOOK_SECRET = process.env.ZYPHR_WEBHOOK_SECRET!; // whsec_...

// IMPORTANT: Use express.raw() to get the raw body for verification
app.post('/webhooks/zyphr', express.raw({ type: 'application/json' }), (req, res) => {
  try {
    WebhooksResource.verifySignature(
      req.body,
      {
        'webhook-id': req.headers['webhook-id'] as string,
        'webhook-timestamp': req.headers['webhook-timestamp'] as string,
        'webhook-signature': req.headers['webhook-signature'] as string,
      },
      WEBHOOK_SECRET
    );
  } catch (err) {
    if (err instanceof WebhookVerificationError) {
      return res.status(401).json({ error: err.message });
    }
    throw err;
  }

  const event = JSON.parse(req.body.toString());
  // Process the event
  res.status(200).send('OK');
});

Secrets can be rotated without downtime — Zyphr signs with both the old and new secret during the rotation window.

Automatic retries with exponential backoff

When a webhook delivery fails (non-2xx response or timeout), Zyphr retries with exponential backoff:

After all retry attempts are exhausted, the delivery is marked as exhausted and routed to a dead-letter queue where you can inspect and manually replay it.

Circuit breaker

If an endpoint fails repeatedly, Zyphr's circuit breaker stops sending to it — preventing a broken endpoint from consuming your retry budget and delaying deliveries to healthy endpoints.

The circuit breaker tracks failure rates over a sliding window:

• Closed (normal): Deliveries proceed as usual
• Open (tripped): Deliveries are paused. Zyphr sends you an alert.
• Half-open (testing): Zyphr sends a test delivery. If it succeeds, the circuit closes. If it fails, it stays open.

You can see circuit state in the dashboard and manually reset it.

Delivery logs with full observability

Every delivery attempt is logged with:

• Request timestamp
• Response status code
• Response time
• Response body (first 1KB)
• Retry count
• Delivery status (pending, delivered, failed, exhausted)

You can filter by event type, status, endpoint, and time range. When a customer says "I didn't get the webhook," you can look up exactly what happened — including every retry attempt and the response your server returned.

Event filtering

Not every endpoint needs every event. Zyphr lets you subscribe to specific event types per webhook:

await zyphr.webhooks.create({
  url: 'https://yourapp.com/hooks/auth',
  events: ['auth.register', 'auth.login', 'auth.password_reset'],
  secret: generatedSecret,
});

Wildcard patterns are supported too — auth.* subscribes to all auth events. This reduces noise on your endpoints and keeps payloads relevant.

Why this matters

Bad webhook infrastructure creates three problems:

1. Silent data loss. If deliveries fail and aren't retried properly, your customers miss critical events. A payment webhook that doesn't arrive means revenue doesn't get recorded.

2. Operational overhead. Without circuit breakers and rate limiting, a single misconfigured endpoint can overwhelm your delivery pipeline and affect all customers.

3. Support burden. Without delivery logs, "I didn't get the webhook" becomes an unsolvable mystery. With logs, it's a 30-second lookup.

The dashboard

The Zyphr webhook dashboard gives you full operational visibility:

• Webhook list: All configured webhooks with their status, event subscriptions, and circuit state
• Delivery log: Filterable list of every delivery attempt with status and timing
• Event replay: Select failed deliveries and replay them with one click
• Secret rotation: Rotate signing secrets with a configurable overlap window
• Health metrics: Success rate, average response time, and failure trends per endpoint

It's the webhook dashboard we wished existed when we were building webhook integrations for other platforms.

Start with webhooks that work

If you're building a platform that needs to notify external systems about events, don't bolt on webhooks as an afterthought. Build on infrastructure that handles signing, retries, circuit breaking, and observability from day one.

Explore webhook docs →