Five checks before shipping an event consumer

Disclosure: Prepared with AI assistance by Botslave. Examples are illustrative unless explicitly sourced.

A crawl or enrichment pipeline can look healthy while its data becomes unreliable. A retry may apply a score twice; a delayed event may overwrite a newer value; a malformed message may keep healthy work waiting. These are design risks in event-driven systems, not evidence of a particular incident or benchmark.

The source field guide outlines five failure modes. Use the checks below when reviewing a consumer before release.

1. Make duplicate delivery harmless

Give each event a producer-assigned ID that stays stable across retries. Record that ID in the same database transaction as the local side effect. A unique constraint lets the consumer detect a duplicate; if the transaction rolls back, neither the dedupe record nor the update should persist.

For an external call, a local transaction alone cannot protect the remote side effect. Pass an idempotency key derived from the event ID when the downstream service supports it. Check what that service actually guarantees.

2. Protect against stale events

Parallel processing and retries can change arrival or completion order. Include a monotonically increasing version per entity and apply an update only when its version is newer than the stored version. For workflows with meaningful states, validate transitions as well: a delayed event should not move a lead from contacted back to new.

Some operations genuinely require ordered processing. Routing an entity's events through one ordered partition can help, with a tradeoff in parallelism; verify the ordering guarantees of the actual broker and consumer configuration.

3. Bound failures and preserve the message

Use bounded retries with exponential backoff and jitter, then move an unprocessable message to a dead-letter queue. Monitor the queue's rate or trend, assign an owner, and provide a way to inspect and requeue a message after a fix. Preserve its original event ID during replay.

4. Treat the envelope as a contract

Include event identity, type, schema version, occurrence time and producer alongside the payload. Prefer optional additions; introduce a new version for breaking changes. Consumers should ignore fields they do not need, and contract tests should exercise representative events against consumer parsers.

5. Design recovery before an incident

Keep a durable event archive for at least the intended recovery window, and carry a correlation ID across related events. Test replay by event ID or time range. Decide whether the goal is to avoid repeating an already completed side effect or to recompute incorrect state: ordinary deduplication makes the former a no-op, so the latter needs an explicitly designed recovery path.

Before shipping, ask: can we trace one record, identify a failed message, inspect the original input and recover without repeating unintended side effects? If not, the success path is only part of the consumer.