Guaranteed Delivery Pattern

Zero message loss with event store and cursor-based resumption

This pattern provides zero message loss by combining persistent event storage with acknowledgment-based delivery. Events are stored durably in an event store database, enabling historical backfill and cursor-based resumption.

When to adopt this pattern

  • Zero message loss required: Messaging applications, notifications, email, financial transactions, audit logs

  • Historical data access: Clients need to query and receive events from before they subscribed

  • Cursor-based resumption: Clients reconnect and resume from their last received event without gaps

  • Event replay capabilities: Need to replay historical events for debugging or recovery

  • Persistent event storage: Can operate and maintain an event store database

  • Complex infrastructure acceptable: Willing to implement backfill logic, acknowledgment mechanics, and event persistence

Architecture diagram

Component responsibilities

Subgraph

  • Receive subscription request with callback URL and cursor (if resuming a subscription) from Router

  • Forward subscription registration to the broker via HTTP API

  • Return success or failure to Router

  • No further involvement in subscription lifecycle after registration

Subscription broker

  • Store subscription metadata in Redis (callback URL, cursor position, query, variables)

  • Query the event store for historical events when a cursor is provided

  • Deliver backfill events to Router before live events

  • Subscribe to PubSub topics for live event delivery

  • Send periodic heartbeats to Router to verify connection is alive

  • Update Redis TTL on successful heartbeat

  • Mark subscription as closed after consecutive heartbeat failures

  • Deliver live events to Router via callback URL

  • Only acknowledge PubSub messages after successful Router delivery

  • Negatively acknowledge and requeue messages on delivery failure

Event store database

  • Store all events durably with sequential IDs or timestamps

  • Support queries for events after a specific cursor

  • Set retention policy based on business requirements (hours, days, or indefinite)

  • Examples: PostgreSQL with an event table, EventStoreDB, SQLite

Redis

  • Store subscription metadata with TTL (subscription ID, callback URL, cursor position)

  • Set TTL to control subscription liveness

  • Coordinate distributed broker instances

  • Provide pub/sub for event distribution

PubSub system

  • Configure for manual acknowledgment (no auto-acknowledgement)

  • Keep messages in queue until explicitly acknowledged

  • Support negative acknowledgment and requeue for failed deliveries

  • Configure consumer groups for distributed processing

Heartbeat interval

The recommended heartbeat interval to set in Router configuration is 5000 milliseconds (5 seconds). This value balances keeping connections alive while minimizing overhead. Adjust the interval based on your needs:

  • Higher connection counts: Increase the interval to reduce heartbeat traffic (for example, 10000ms)

  • Network latency concerns: Decrease the interval to detect failures faster (for example, 3000ms)

  • Load balancer timeout: Ensure the interval is shorter than your load balancer's idle timeout

  • Note: This value is set in your Router configuration file and must be enforced by the broker. If the broker takes longer to send a heartbeat the Router will terminate the subscription.

Backfill and resumption

Initial subscription

Optionally include a cursor to resume from a specific point. If you don't provide a cursor, your subscription starts with live events only:

GraphQL
1subscription {
2  messages(after: "cursor_123") {  # optional - omit to start from now
3    id
4    content
5  }
6}

Backfill phase

  1. Broker queries event store: SELECT * FROM events WHERE id > 'cursor_123' ORDER BY cursor ASC

  2. Broker delivers historical events to Router one by one

  3. Router streams events to client via multipart HTTP

  4. Client updates their cursor as they receive events

Delivering live events

  1. After backfill completes, broker subscribes to PubSub topics

  2. Broker starts receiving new events as they are published

  3. Seamless transition from historical to live events

Resumping after disconnection

  1. Client detects connection loss

  2. Client reconnects with cursor from the last received event

  3. Broker queries event store for events after the cursor

  4. Broker delivers any missed events during disconnection

  5. Resume live event stream

Designing the event store

Schema considerations

Store your events with sequential identifiers for cursor-based queries:

sql
1CREATE TABLE IF NOT EXISTS events (
2  cursor INTEGER PRIMARY KEY AUTOINCREMENT,
3  id TEXT NOT NULL,
4  content TEXT NOT NULL,
5  timestamp TEXT NOT NULL,
6  created_at TEXT DEFAULT CURRENT_TIMESTAMP
7);

Retention policies

  • Time-based: Delete events older than N days

  • Size-based: Delete oldest when storage exceeds limit

  • Indefinite: Keep all events (audit trails, compliance)

Scaling considerations

Event store performance

  • Index cursor columns for fast range queries

  • Partition large tables by time period (monthly, yearly)

  • Read replicas for backfill queries to avoid impacting live writes

  • Cache recent events in Redis for faster backfill of short-duration disconnections

Subscription broker

  • Ensure your broker is able to leverage horizontal scaling

    • Configure auto-scaling based on CPU exceeding 70%, memory exceeding 80%, or response time exceeding 100ms

PubSub consumer configuration

  • Consumer prefetch limits prevent overwhelming the broker (for example, prefetch=10)

  • Manual acknowledgment prevents message loss during processing

  • Dead letter queues for messages that fail repeatedly

  • Monitor consumer lag (messages waiting in queue)

Adopting this pattern

When to consider moving to this pattern

  • Users report missing events during disconnections

  • Business requirements change to require guaranteed delivery

  • Need to build features requiring event history (message search, audit logs)

  • SLAs require zero message loss guarantees

Requirements for adopting this pattern

  1. Deploy your event store and start persisting events

  2. Add a cursor parameter to your GraphQL subgraph and broker service

  3. Build your subscription broker with backfill logic

  4. Configure your PubSub consumers for manual acknowledgment mode

  5. Test resumption with various cursor positions

Next steps

See the Guaranteed Delivery code example.
Feedback