Exactly-Once vs At-Least-Once Delivery

Explore the fundamental delivery guarantees in async job processing. True exactly-once delivery remains theoretically impossible in distributed networks. Modern backend architectures achieve practical exactly-once semantics through application-level idempotency, transactional outboxes, and precise broker configurations.

Key architectural considerations include:

The CAP theorem's impact on message routing and network partitions
Why at-least-once serves as the default for high-throughput distributed systems
Designing consumers that safely handle duplicate payloads
Operational trade-offs between broker-side guarantees and application-level deduplication

Theoretical Foundations & Distributed Systems Reality

Network partitions, packet loss, and clock drift make absolute exactly-once delivery mathematically unattainable. The CAP theorem dictates that distributed systems must sacrifice consistency or availability during partitions. Message brokers prioritize availability and partition tolerance.

Understanding baseline routing and acknowledgment flows requires a solid grasp of Queue Fundamentals & Architecture. Brokers rely on persistent storage and consumer heartbeat mechanisms to track message state. When a consumer crashes mid-processing, the broker cannot distinguish between a slow worker and a failed node.

Consequently, the broker requeues the message to ensure delivery. This safety mechanism inherently produces at-least-once semantics. Engineers must design systems assuming duplicate delivery is a baseline operational reality, not an edge case.

At-Least-Once Delivery: The Industry Standard

At-least-once delivery relies on explicit acknowledgment workflows. Producers publish messages to durable storage. Consumers pull payloads, process them, and send explicit ACK or NACK signals. Unacknowledged messages trigger automatic retries.

Properly configuring the Visibility Timeout Deep Dive prevents duplicate processing spikes during consumer scaling events. Visibility timeouts must exceed the P99 processing latency of your workload. Setting timeouts too low causes premature redelivery. Setting them too high delays failure recovery.

Design stateless consumers that tolerate duplicate payloads. Implement exponential backoff with jitter for retry policies. Configure Dead Letter Queue (DLQ) routing to isolate poison messages after max retry thresholds. This approach maximizes throughput while maintaining fault tolerance.

Achieving Practical Exactly-Once Semantics

Practical exactly-once behavior is engineered at the application layer. It combines at-least-once broker delivery with deterministic idempotency controls. Consumers generate or extract unique keys from incoming payloads.

Store these keys in a distributed, persistent datastore. Use database-level unique constraints or Redis SETNX operations to block duplicate execution. Upsert operations ensure subsequent payloads return cached results without side effects. A comprehensive implementation guide is detailed in Preventing duplicate job execution with idempotency.

Deploy the transactional outbox pattern for cross-system consistency. Write business state and outbox records in a single local database transaction. A background poller publishes outbox entries to the queue. This guarantees message publication matches committed state.

Broker-Level Exactly-Once: Kafka Streams & Pulsar

Modern streaming brokers offer native exactly-once semantics (EOS). Kafka EOS v2 utilizes transactional producers and consumer offset commits. Producers batch writes into atomic transactions. Consumers read only committed data via isolation.level=read_committed.

Apache Pulsar implements native transactional messaging with coordinated cursor management. Both systems guarantee exactly-once processing within their internal ecosystems. However, broker-side deduplication introduces measurable latency and throughput penalties.

Evaluating native versus application-level guarantees is essential when reviewing the Message Broker Comparison. Streaming brokers require careful partition tuning and transaction timeout configuration. Misconfigured transactional IDs cause producer fencing and throughput degradation.

Operational Workflows & Scaling Strategies

Platform teams must monitor duplicate detection rates and idempotency cache hit ratios. High duplicate rates indicate visibility timeout misalignment or consumer scaling thrash. Track DLQ ingestion velocity to identify systemic payload corruption.

Horizontal scaling requires careful partition affinity management. Consumer group rebalancing triggers temporary message stalls. Deploy sticky partition assignments to minimize state migration during scaling events. Share idempotency state across pods using Redis Cluster or a distributed relational database.

Balance cost against reliability using a service-tier matrix. Fire-and-forget analytics pipelines tolerate at-least-once delivery. Financial reconciliation and inventory updates require strict idempotency controls. Align broker selection and consumer architecture with your SLA requirements.

Code Examples

Idempotency Middleware (Node.js/Express)

const redis = require('redis');
const client = redis.createClient({ url: process.env.REDIS_URL });

// Production middleware: intercepts payloads, checks idempotency store
async function idempotencyMiddleware(req, res, next) {
 const idempotencyKey = req.headers['x-idempotency-key'] || req.body.idempotency_key;
 if (!idempotencyKey) return next(new Error('Missing idempotency key'));

 try {
 // TTL prevents unbounded cache growth; adjust based on SLA
 const cached = await client.get(`idemp:${idempotencyKey}`);
 if (cached) {
 // Short-circuit duplicate execution, return cached response
 return res.status(200).json(JSON.parse(cached));
 }
 
 // Attach key to request context for downstream handlers
 req.idempotencyKey = idempotencyKey;
 next();
 } catch (err) {
 // Fail-open strategy: allow processing if Redis is unavailable
 console.error('Idempotency check failed:', err);
 next();
 }
}

Kafka EOS Producer Configuration

import org.apache.kafka.clients.producer.ProducerConfig;
import java.util.Properties;

Properties props = new Properties();
// Enable idempotence to prevent duplicate sends during retries
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Unique transactional ID per producer instance; prevents fencing collisions
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "tx-payment-processor-1");
// Ensure consumers only read committed transactions
props.put(ProducerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
// Tune max.in.flight.requests.per.connection to 5 for EOS v2 performance
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5");
props.put(ProducerConfig.ACKS_CONFIG, "all");

Transactional Outbox Poller

import psycopg2
import time
from queue_publisher import publish_batch

def poll_outbox(db_conn, limit=100, backoff_sec=0.5):
 """Background worker reads unsent outbox records, publishes, and marks sent."""
 while True:
 try:
 with db_conn.cursor() as cur:
 # Lock rows to prevent concurrent poller collisions
 cur.execute("""
 SELECT id, payload FROM outbox 
 WHERE status = 'PENDING' 
 ORDER BY created_at ASC 
 LIMIT %s FOR UPDATE SKIP LOCKED
 """, (limit,))
 records = cur.fetchall()

 if not records:
 time.sleep(backoff_sec)
 continue

 # Publish to broker outside DB transaction to avoid holding locks
 publish_batch(records)

 # Mark as sent in a separate atomic update
 ids = [r[0] for r in records]
 cur.execute("UPDATE outbox SET status = 'SENT' WHERE id = ANY(%s)", (ids,))
 db_conn.commit()
 except Exception as e:
 db_conn.rollback()
 time.sleep(backoff_sec * 2)
 backoff_sec = min(backoff_sec * 2, 30)

Common Pitfalls

Assuming broker-level exactly-once guarantees eliminate the need for application-level idempotency
Over-engineering deduplication for low-risk, fire-and-forget async notifications
Failing to tune visibility timeouts, causing duplicate processing spikes during consumer scaling events
Ignoring idempotency key collisions during high-concurrency retries without exponential backoff
Storing idempotency state in ephemeral memory instead of a persistent, distributed datastore

FAQ

Is true exactly-once delivery possible in distributed systems? Theoretically, no. Network partitions, node failures, and clock drift make absolute exactly-once delivery impossible under the CAP theorem. Practically, engineers achieve effectively exactly-once semantics by combining at-least-once broker delivery with application-level idempotency and transactional outboxes.

How do I implement idempotency for database writes in async jobs? Generate a deterministic idempotency key from the job payload or request context. Before executing the write, check a persistent store (Redis or relational DB) for the key. If absent, execute the write and store the key with the result. If present, return the cached result without re-executing.

When should I use at-least-once over exactly-once? Use at-least-once when throughput, low latency, and cost efficiency are prioritized over strict data consistency. Examples include logging, analytics ingestion, or non-critical notifications. Use exactly-once (or practical idempotency) for financial transactions, inventory updates, or operations where duplicate execution causes data corruption.

Does Kafka's exactly-once guarantee eliminate the need for application-level deduplication? Kafka EOS guarantees exactly-once processing within the Kafka ecosystem (producer to broker to consumer offsets). However, if your consumer writes to external systems (PostgreSQL, S3, third-party APIs), you still need application-level idempotency or transactional outboxes to guarantee exactly-once end-to-end.