workflow automation reliability · field note

autumn-billing: dunning exhaustion skips Unpaid/cancel when payment_failed arrives before the subscription event

A working diagnosis of this problem and the sequence that actually resolves it, written from the operational side rather than the marketing side.

Reported context: ## Problem If `invoice.payment_failed` is delivered **before** the corresponding subscription event, `Reconciler::upsert_invoice` (`autumn-billing/src/reconcile.rs`) cannot resolve the provider subscription ID to a local subscription, so the invoice and its dunning row are stored with `subscription_id = None`. The link is only back-filled by `open_dunning` on a **later** `payment_failed` webhook for the same invoice

What is actually going on

Silent failures are the expensive ones: a workflow that stops running without alerting is worse than one that errors loudly.

Webhook consumers that respond slowly cause upstream providers to retry, multiplying load during incidents.

The sequence that works

  1. Add an idempotency key to every write operation so retries cannot duplicate effects.
  2. Wrap every external call in a retry with exponential backoff and a hard attempt ceiling, and log the final state either way.
  3. Respond to webhooks within the provider timeout by acknowledging immediately and processing asynchronously.
  4. Instrument a dead-letter queue and a heartbeat. Alert on absence of expected events, not only on errors.
  5. Load test the workflow at the provider's documented rate limit to find the real breaking point before production does.

Where people lose time

Verify before you trust it

Run the checks above against your own data instead of adopting them at face value. The failure modes are specific enough that a single afternoon of measurement will either confirm this diagnosis or rule it out — both outcomes are useful, and guessing is not.

Automation Workflow Pack
A ready-to-use package for exactly this workflow — $27.55, delivered immediately.
Get Automation Workflow Pack · $27.55
Sold via Gumroad, which acts as merchant of record and handles payment and applicable taxes. All sales are final; see the return policy before purchase.