workflow automation reliability · field note

Billing webhook redelivery is ignored after a failed first attempt, so the subscription is never applied

A working diagnosis of this problem and the sequence that actually resolves it, written from the operational side rather than the marketing side.

Reported context: **Severity:** high ## Description `handleWebhook` saves the BillingEvent row before applying the event. If applying throws, the row is marked `failed` and the controller returns 500 so the provider retries. The retry has the same `provider_event_id`. `recordEvent` hits the unique constraint and returns null, and `handleWebhook` answers 200 `{duplicate:true}`. The event is never applied, and the provider stops retryin

What is actually going on

Automation failures cluster around unhandled rate limits and missing idempotency keys, which produce duplicate side effects under retry.

Silent failures are the expensive ones: a workflow that stops running without alerting is worse than one that errors loudly.

The sequence that works

  1. Add an idempotency key to every write operation so retries cannot duplicate effects.
  2. Wrap every external call in a retry with exponential backoff and a hard attempt ceiling, and log the final state either way.
  3. Respond to webhooks within the provider timeout by acknowledging immediately and processing asynchronously.
  4. Instrument a dead-letter queue and a heartbeat. Alert on absence of expected events, not only on errors.
  5. Load test the workflow at the provider's documented rate limit to find the real breaking point before production does.

Where people lose time

Verify before you trust it

Run the checks above against your own data instead of adopting them at face value. The failure modes are specific enough that a single afternoon of measurement will either confirm this diagnosis or rule it out — both outcomes are useful, and guessing is not.

Automation Workflow Pack
A ready-to-use package for exactly this workflow — $27.55, delivered immediately.
Get Automation Workflow Pack · $27.55
Sold via Gumroad, which acts as merchant of record and handles payment and applicable taxes. All sales are final; see the return policy before purchase.