Skip to main content
Techdome
All Insights/System Design

Webhook retries and idempotency keys, together

Webhook retries need stable idempotency keys so repeated deliveries are safe, deduplicated, observable, and recoverable through DLQs.

Techdome Engineering
Techdome EngineeringEngineering Team
Oct 26, 20263 min read
Verified Architecture
Illustration of webhook retry flow using a stable idempotency key, de-duplication store, and dead-letter queue.

Webhook retries and idempotency keys solve different parts of the same reliability problem. Retries make delivery likely after timeouts or non-2xx responses; idempotency keys make repeated delivery safe by letting the receiver identify the same logical event.

Webhook retries and idempotency keys flow

The sender creates an event, assigns a stable delivery identity, and sends it as a header or body field, often something like X-Idempotency-Key. Provider delivery IDs such as GitHub's X-GitHub-Delivery or Shopify's webhook ID can serve this role when they are preserved across redeliveries.

If the receiver returns non-2xx or times out, the sender puts the delivery into a retry queue. A retry worker later replays the same payload with the same key. That preservation is the contract: a new key turns a retry into a new event.

Flow showing a webhook sender retrying the same idempotency key through a retry worker until the receiver de-duplicates or sends exhausted attempts to a DLQ.
Flow showing a webhook sender retrying the same idempotency key through a retry worker until the receiver de-duplicates or sends exhausted attempts to a DLQ.

Receiver-side de-duplication

The receiver should verify the signature and timestamp first, then check a de-duplication store keyed by the idempotency key. Common implementations are a database row with a unique constraint, a Redis set with TTL, or a dedicated idempotency table that records status and result.

For a new key, insert or lock the record, enqueue work, and process side effects. For an existing key, return 200 or the recorded result without repeating the side effect. Downstream calls should also use idempotency keys or upsert semantics, because duplicate prevention at the webhook edge does not automatically protect payment, email, CRM, or provisioning APIs. Tools like Sparrow are useful for exercising duplicate deliveries while testing API behavior.

Trade-offs and when not to use it

Receiver-side de-duplication is usually the safest default because the receiver knows which side effects matter. Content-based de-duplication is weaker: two payloads can look similar but represent different intent, and hashes do not replace a stable delivery ID.

Store keys for longer than the sender's maximum retry window, plus operational margin. A short TTL can re-admit an old retry as new work. A very long TTL costs more storage but gives safer replay handling.

Do not add complex idempotency machinery for disposable analytics events where occasional duplicates are acceptable. Do use it for payments, provisioning, patient workflows, account changes, affiliate attribution, or any automation where repeating an action causes real operational cleanup.

How this looks in practice

In A Multi-Vendor Telehealth Care Ecosystem, the architecture connected payments, intake, scheduling, community, and consultation software through a unified identity and lifecycle model. The build included server-to-server affiliate postbacks and automated community provisioning after checkout completion, both patterns where event identity and replay safety matter.

The anonymised case study reports: Deal to Kickoff: ~2.5 Wks; Follow-on SOW: 13 Days; Vendor Logins: 5 → 1. A related automation pattern appears in Five Rapid Automation Deployments, where webhook and API orchestration supported rapid workflow deployments across operational bottlenecks.

Takeaways

  • Retries improve delivery; idempotency keys prevent duplicate effects.
  • The retry worker must replay the original payload with the original key.
  • The receiver should verify, de-dupe, enqueue, and acknowledge quickly.
  • Exhausted retries need a DLQ or manual-resolution queue that preserves key, payload, and metadata.

Further reading

References

FAQ

Questions readers ask

Should the sender or receiver generate the idempotency key?

Prefer a sender-preserved delivery ID from the provider. If none exists, a sender-generated UUID is fine, but it must stay the same across every retry.

What should happen if processing fails after the receiver returns 200?

The receiver should have already placed the work on a durable internal queue. Internal worker failures should then use their own retry and DLQ path.

How long should we keep idempotency keys?

Keep them at least as long as the sender's retry window, plus margin. Align TTLs with the provider or queue system that performs retries.

Can we skip the DLQ?

Only if loss is acceptable. For business-critical events, a DLQ preserves the original key, payload, and failure context for safe inspection or replay.

Continue reading

More foundry teardowns

View all
Pipeline illustration of an automation workflow with a queue feeding two workers, a retry loop, an alert bell and a review tray
n8n AutomationAugust 2026·7 min read

Production n8n: the patterns that keep automation from failing quietly

Idempotent triggers, retries with limits, failure alerts, human review and workflows kept in Git: the patterns we use to run n8n automation in production, with the builds they come from.

Rahul Joshi
Rahul JoshiChief Executive Officer
Computer screen with data
n8n Automation7 min read

"Automating mortgage back-office work with n8n"

Mortgage back offices drown in documents. A staged n8n workflow can handle intake, classification, extraction and validation, and route exceptions to people with a full audit trail.

Techdome Engineering
Techdome EngineeringEngineering Team
Computer screen filled with code
Engineering Practice8 min read

API testing explained: types, what to automate, and tools

API testing checks an interface directly, below the UI. Learn the five main types, how mocking and test data fit in, and which tests to automate first.

Techdome Engineering
Techdome EngineeringEngineering Team
+
+
+
+
Healthcare AIOct 28, 2026·2 min read

Asynchronous vs synchronous telehealth: which to build first

Asynchronous telehealth means a clinician reviews a prepared case; synchronous means a live visit. How they differ, what each changes in the build, and why most platforms end up with both.

Techdome Engineering
Techdome EngineeringEngineering Team
+
+
+
+
Healthcare AIOct 28, 2026·4 min read

Telehealth compliance checklist for the US and UK

An engineering checklist for a telehealth platform serving US and UK patients: contracts, access and audit, clinic isolation, licensing by jurisdiction, prescribing, signatures, payments and incidents.

Techdome Engineering
Techdome EngineeringEngineering Team
+
+
+
+
Healthcare AIOct 28, 2026·3 min read

How to build a GLP-1 telehealth platform

A builder's guide to a GLP-1 telehealth platform: compliance, multi-tenant clinics, eligibility checks, licensing by jurisdiction, chart review, pharmacy routing, cold chain and billing.

Techdome Engineering
Techdome EngineeringEngineering Team
+
+
+
+
Healthcare AIOct 28, 2026·2 min read

Telehealth metrics and KPIs: the numbers that matter

The 15 telehealth metrics an operator should track, from intake completion to time to physician decision and cold-chain excursions, with real figures from a US and UK GLP-1 platform.

Techdome Engineering
Techdome EngineeringEngineering Team
A technical illustration comparing queued task delivery with retained event streaming.
System DesignOct 26, 2026·3 min read

Message queues vs event streams: SQS, RabbitMQ, Kafka

Use SQS or RabbitMQ for work distribution; use Kafka when you need retained event logs, replay, fan-out, and per-key ordering.

Techdome Engineering
Techdome EngineeringEngineering Team
Blueprint-style illustration of a server rack behind a layered firewall shield, with a PHP version badge being upgraded
Engineering PracticeOct 22, 2026·6 min read

PHP2Shell: how to check and protect your PHP site

PHP2Shell (CVE-2026-109618 and CVE-2026-109619): how to check your PHP version, patch it or shield it with a WAF, with two real cases.

Techdome Engineering
Techdome EngineeringEngineering Team
Interface illustration of a browser window with three gauge dials, a heavy block swapped for a light placeholder, and images compressed
Engineering PracticeSeptember 2026·6 min read

Making a Next.js site fast: the fixes that moved Core Web Vitals on techdome.io

Hero images that wait for JavaScript, an embedded scheduler that blocked the page, oversized images and bloated page data: the real fixes behind our site's performance work, and a checklist for yours.

Rahul Joshi
Rahul JoshiChief Executive Officer
3D illustration of an AI agent connected to calendar, database and phone tools, with a sound wave and a small knowledge graph
Studio ArchitectureSeptember 2026·7 min read

Building AI agents that hold up in production: what we learned from voice, interviews and GraphRAG

Narrow jobs, grounded answers, deterministic scoring, a latency budget, full logs and a human handoff: the patterns behind our voice agents, AI interviewer and GraphRAG platform.

Rahul Joshi
Rahul JoshiChief Executive Officer
Isometric illustration of one platform split into separate clinic compartments, with a shield, an access gate and an audit log
Healthcare AIAugust 2026·8 min read

Engineering HIPAA-ready telehealth platforms: the decisions we make first

How tenant isolation, access control, audit logging, identity and vendor agreements shape a HIPAA-ready telehealth system, with what each looked like on three clinical platforms we built.

Rahul Joshi
Rahul JoshiChief Executive Officer
Blueprint-style illustration of five phases from a drawing board to a launched product, rising along a path
Studio ArchitectureAugust 2026·7 min read

The Techdome Blueprint: how we take a product from a blank repo to production in eight weeks

The five steps of our Blueprint, what each one delivers, how fixed-scope sprints and engineering pods are priced, and how recent builds ran from kickoff to production.

Rahul Joshi
Rahul JoshiChief Executive Officer