# Webhook Architecture This document explains the webhook system architecture, including how events are emitted, queued, delivered, and displayed. ## Architecture Diagram ``` ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ EVENT SOURCES │ ├─────────────────────┬─────────────────────┬─────────────────────────────────────────┤ │ Email Service │ Contact Service │ Domain Service │ │ (SES callbacks) │ (CRUD operations) │ (verification, etc.) │ └─────────┬───────────┴──────────┬──────────┴──────────────────┬──────────────────────┘ │ │ │ │ ▼ │ │ ┌───────────────────────┐ │ └────────►│ WebhookService.emit │◄─────────────────┘ │ (teamId, type, │ │ payload) │ └───────────┬───────────┘ │ ┌───────────▼───────────┐ │ Find Active Webhooks │ │ matching event type │ └───────────┬───────────┘ │ ┌─────────────────┼─────────────────┐ ▼ ▼ ▼ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Webhook A │ │ Webhook B │ │ Webhook C │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ PostgreSQL Database │ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │ │ WebhookCall (one per matching webhook) │ │ │ │ ├── status: PENDING │ │ │ │ ├── payload: { event data only } │ │ │ │ └── attempt: 0 │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ Redis + BullMQ │ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │ │ WEBHOOK_DISPATCH_QUEUE │ │ │ │ ├── Job: { callId: "call_abc", teamId: 123 } │ │ │ │ ├── Job: { callId: "call_def", teamId: 123 } │ │ │ │ └── Job: { callId: "call_ghi", teamId: 456 } │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ │ BullMQ Worker (concurrency: 25) ▼ ┌───────────────────────┐ │ processWebhookCall │ └───────────┬───────────┘ │ ┌───────────▼───────────┐ │ Acquire Redis Lock │──────┐ │ (per webhook ID) │ │ Lock failed └───────────┬───────────┘ │ │ Lock acquired ▼ │ ┌─────────────┐ ┌───────────▼──────┐ │ Retry later │ │ Check webhook │ └─────────────┘ │ status = ACTIVE? │ └───────────┬──────┘ Yes │ No ┌───────────┘ └──────────────┐ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ buildPayload │ │ Mark call as │ │ (wrap event │ │ DISCARDED │ │ data) │ └─────────────────┘ └────────┬────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │ HTTP POST Request │ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │ │ Headers: │ │ │ │ ├── X-UseSend-Signature: v1= │ │ │ │ ├── X-UseSend-Timestamp: 1705312200000 │ │ │ │ ├── X-UseSend-Event: email.delivered │ │ │ │ └── X-UseSend-Call: call_abc123 │ │ │ │ │ │ │ │ Body: { │ │ │ │ "id": "call_abc123", │ │ │ │ "type": "email.delivered", │ │ │ │ "version": "2026-01-18", │ │ │ │ "createdAt": "...", │ │ │ │ "teamId": 123, │ │ │ │ "data": { ... event payload ... }, │ │ │ │ "attempt": 1 │ │ │ │ } │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────────────┘ │ ┌───────────┴───────────┐ ▼ ▼ ┌─────────────┐ ┌─────────────┐ │ 2xx OK │ │ Non-2xx / │ │ │ │ Timeout │ └──────┬──────┘ └──────┬──────┘ │ │ ▼ ▼ ┌─────────────────┐ ┌─────────────────────┐ │ Mark DELIVERED │ │ Increment failures │ │ Reset failures │ │ attempt < 6? │ │ to 0 │ └──────────┬──────────┘ └─────────────────┘ Yes │ No ┌──────────┘ └──────────┐ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Mark PENDING │ │ Mark FAILED │ │ Schedule retry │ │ │ │ (exp. backoff) │ │ failures >= 30? │ └─────────────────┘ └────────┬────────┘ Yes │ No ┌───────────┘ └────┐ ▼ ▼ ┌─────────────────┐ ┌──────────┐ │ AUTO_DISABLE │ │ Done │ │ webhook │ └──────────┘ └─────────────────┘ ``` ## Call Status State Machine ``` ┌──────────────────────────────────────┐ │ │ ▼ │ ┌─────────┐ enqueue ┌───────────┐ worker picks up ┌─────────────────┐ │ (start) │ ──────────►│ PENDING │ ─────────────────►│ IN_PROGRESS │ └─────────┘ └───────────┘ └────────┬────────┘ ▲ │ │ ┌────────────┼────────────┐ │ │ │ │ │ retry (attempt<6) │ │ │ │ ▼ ▼ ▼ │ ┌───────────┐ ┌───────────┐ ┌───────────┐ └────────────│ (fail) │ │ SUCCESS │ │ WEBHOOK │ └─────┬─────┘ └─────┬─────┘ │ INACTIVE │ │ │ └─────┬─────┘ │ │ │ attempt >= 6 │ │ │ ▼ ▼ │ ┌───────────┐ ┌───────────┐ └─────►│ FAILED │ │ DISCARDED │ └───────────┘ └───────────┘ ``` ## Overview The webhook system allows users to receive real-time HTTP notifications when events occur (emails sent, contacts created, domains verified, etc.). The system is built with reliability in mind, featuring: - Asynchronous delivery via BullMQ - Exponential backoff with jitter for retries - Automatic webhook disabling after consecutive failures - Per-webhook locking to ensure ordered delivery - HMAC signature verification for security ## Core Components ### 1. Database Models Located in `apps/web/prisma/schema.prisma`: ``` Webhook ├── id (cuid) ├── teamId (FK → Team) ├── url (endpoint URL) ├── secret (signing key, prefixed with "whsec_") ├── status (ACTIVE | PAUSED | AUTO_DISABLED) ├── eventTypes (string[] - empty means all events) ├── apiVersion (optional version string) ├── consecutiveFailures (counter for auto-disable) ├── lastFailureAt / lastSuccessAt (timestamps) └── createdByUserId (FK → User) WebhookCall ├── id (cuid) ├── webhookId (FK → Webhook) ├── teamId (FK → Team) ├── type (event type, e.g., "email.delivered") ├── payload (JSON string - event data only) ├── status (PENDING | IN_PROGRESS | DELIVERED | FAILED | DISCARDED) ├── attempt (current attempt number) ├── nextAttemptAt (scheduled retry time) ├── lastError (error message if failed) ├── responseStatus / responseTimeMs / responseText └── createdAt / updatedAt ``` ### 2. Service Layer Located in `apps/web/src/server/service/webhook-service.ts`: - **WebhookService**: CRUD operations for webhooks and webhook calls - **WebhookQueueService**: BullMQ queue management for async delivery ### 3. Event Types Defined in `packages/lib/src/webhook/webhook-events.ts`: ```typescript // Contact events "contact.created" | "contact.updated" | "contact.deleted"; // Domain events "domain.created" | "domain.verified" | "domain.updated" | "domain.deleted"; // Email events "email.queued" | "email.sent" | "email.delivery_delayed" | "email.delivered" | "email.bounced" | "email.rejected" | "email.rendering_failure" | "email.complained" | "email.failed" | "email.cancelled" | "email.suppressed" | "email.opened" | "email.clicked"; // Test events ("webhook.test"); ``` ## Webhook Flow ### Step 1: Event Emission When an event occurs in the system, `WebhookService.emit()` is called: ```typescript // Example: emitting an email.delivered event await WebhookService.emit(teamId, "email.delivered", { id: email.id, status: "DELIVERED", from: email.from, to: email.to, occurredAt: new Date().toISOString(), // ... other fields }); ``` ### Step 2: Webhook Matching & Call Creation `emit()` performs the following: 1. Finds all ACTIVE webhooks for the team that subscribe to the event type 2. Creates a `WebhookCall` record for each matching webhook (stores event data as `payload`) 3. Enqueues the call ID to BullMQ for async processing ```typescript // Webhook matching logic const activeWebhooks = await db.webhook.findMany({ where: { teamId, status: WebhookStatus.ACTIVE, OR: [ { eventTypes: { has: type } }, // Subscribed to this event { eventTypes: { isEmpty: true } }, // Subscribed to ALL events ], }, }); ``` ### Step 3: Queue Processing The BullMQ worker (`processWebhookCall`) handles delivery: 1. **Lock Acquisition**: Acquires a Redis lock per webhook to ensure ordered delivery 2. **Status Check**: Skips if webhook is no longer ACTIVE (marks call as DISCARDED) 3. **Payload Building**: Wraps the stored event data in the full payload structure 4. **HTTP POST**: Sends signed request to the webhook URL 5. **Result Handling**: Updates call status and webhook metrics ### Step 4: Payload Structure **Important**: The stored `WebhookCall.payload` contains only the event data. The actual HTTP request body is built at delivery time by `buildPayload()`: ```typescript // Stored in WebhookCall.payload (event data only): { "id": "email_123", "status": "DELIVERED", "from": "sender@example.com", "to": ["recipient@example.com"], "occurredAt": "2024-01-15T10:30:00Z" } // Actual payload sent to webhook endpoint: { "id": "call_abc123", // WebhookCall ID "type": "email.delivered", // Event type "version": "2026-01-18", // API version "createdAt": "2024-01-15T10:30:00Z", "teamId": 123, "data": { // Original event data nested here "id": "email_123", "status": "DELIVERED", "from": "sender@example.com", "to": ["recipient@example.com"], "occurredAt": "2024-01-15T10:30:00Z" }, "attempt": 1 } ``` ### Step 5: Request Signing Each request includes security headers for verification: ``` Content-Type: application/json User-Agent: UseSend-Webhook/1.0 X-UseSend-Event: email.delivered X-UseSend-Call: call_abc123 X-UseSend-Timestamp: 1705312200000 X-UseSend-Signature: v1= X-UseSend-Retry: false ``` Signature computation: ```typescript const signature = HMAC - SHA256(secret, `${timestamp}.${JSON.stringify(body)}`); // Format: "v1=" + hex(signature) ``` ## Retry & Failure Handling ### Retry Configuration ```typescript const WEBHOOK_MAX_ATTEMPTS = 6; const WEBHOOK_BASE_BACKOFF_MS = 5_000; // 5 seconds const WEBHOOK_AUTO_DISABLE_THRESHOLD = 30; ``` ### Backoff Schedule (approximate) | Attempt | Delay (base) | With Jitter | | ------- | ------------ | ----------- | | 1 | 5s | 5-6.5s | | 2 | 10s | 10-13s | | 3 | 20s | 20-26s | | 4 | 40s | 40-52s | | 5 | 80s | 80-104s | | 6 | 160s | 160-208s | ### Auto-Disable After 30 consecutive failures across any calls, the webhook is automatically set to `AUTO_DISABLED` status. This prevents continued delivery attempts to consistently failing endpoints. ### Call Status Flow ``` PENDING → IN_PROGRESS → DELIVERED (success) → PENDING (retry on failure, attempts < 6) → FAILED (max attempts reached) → DISCARDED (webhook disabled/paused) ``` ## SDK Webhook Verification Located in `packages/sdk/src/webhooks.ts`: ```typescript import { UseSend } from "usesend"; const usesend = new UseSend("us_api_key"); const webhooks = usesend.webhooks("whsec_your_secret"); // Option 1: Verify only (returns boolean) const isValid = webhooks.verify(rawBody, { headers: request.headers }); // Option 2: Verify and parse (throws on invalid) const event = webhooks.constructEvent(rawBody, { headers: request.headers }); if (event.type === "email.delivered") { console.log(event.data.to); // Type-safe access } ``` ## UI Payload Display The webhook call details UI (`apps/web/src/app/(dashboard)/webhooks/[webhookId]/webhook-call-details.tsx`) reconstructs the full payload for display, matching what was actually sent to the endpoint. This uses the same structure as `buildPayload()` in the service layer. ## Important Files | File | Purpose | | ------------------------------------------------ | --------------------------- | | `apps/web/prisma/schema.prisma` | Database models | | `apps/web/src/server/service/webhook-service.ts` | Core service & queue worker | | `apps/web/src/server/api/routers/webhook.ts` | TRPC API routes | | `apps/web/src/lib/constants/plans.ts` | Webhook limits per plan | | `packages/lib/src/webhook/webhook-events.ts` | Event type definitions | | `packages/sdk/src/webhooks.ts` | SDK verification utilities | | `apps/web/src/app/(dashboard)/webhooks/` | UI components | ## Configuration Constants ```typescript // apps/web/src/server/service/webhook-service.ts const WEBHOOK_DISPATCH_CONCURRENCY = 25; // Parallel workers const WEBHOOK_MAX_ATTEMPTS = 6; // Max delivery attempts const WEBHOOK_BASE_BACKOFF_MS = 5_000; // Initial retry delay const WEBHOOK_LOCK_TTL_MS = 15_000; // Redis lock TTL const WEBHOOK_LOCK_RETRY_DELAY_MS = 2_000; // Lock retry delay const WEBHOOK_AUTO_DISABLE_THRESHOLD = 30; // Failures before disable const WEBHOOK_REQUEST_TIMEOUT_MS = 10_000; // HTTP timeout const WEBHOOK_RESPONSE_TEXT_LIMIT = 4_096; // Max response body stored const WEBHOOK_EVENT_VERSION = "2026-01-18"; // Default API version ``` ## Plan Limits ```typescript // apps/web/src/lib/constants/plans.ts FREE: { webhooks: 1; } BASIC: { webhooks: -1; } // unlimited ```