Kafka Message Ordering: Guarantee Sequence Across Partitions

The Problem: Your Messages Arrived Out of Order

You send events M1, M2, M3 to Kafka. Your consumer receives M1, M3, M2.

In a banking app, this means debiting before crediting. In an e-commerce system, shipping before payment. Order matters.

Kafka guarantees order within a partition, but not across partitions. Once you scale to multiple partitions for throughput, ordering breaks.

Quick Answer (TL;DR)

Strategy	Throughput	Ordering	Use When
Single partition	Low	Perfect	Small volume, strict order required
Key-based routing	Medium	Per-key	Order matters per entity (user, order)
Idempotent producer	High	Within partition	Prevent duplicates + order
External sequencing	High	Global	Need global order across partitions

// Key-based routing - orders for same user go to same partition
producer.send(new ProducerRecord<>(topic, order.getUserId(), order));

// Idempotent producer - prevents duplicate/reordering on retry
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");

---

Understanding Kafka’s Ordering Model

Why This Happens

1. Producer sends M1, M2, M3 in sequence
2. Partitioner routes each message to different partitions
3. Consumer polls from all partitions - order depends on poll timing
4. Result: M2 might arrive before M3 even though sent after

---

Strategy 1: Single Partition (Simplest)

Force all messages to one partition:

// Create topic with 1 partition
kafka-topics.sh --create --topic orders --partitions 1

// Producer - no key needed
producer.send(new ProducerRecord<>("orders", orderEvent));

Pros: Perfect ordering Cons: Single consumer, limited throughput (~10K msg/sec typical)

Use for: Audit logs, compliance events, small-volume critical sequences

---

Strategy 2: Key-Based Routing (Most Common)

Route related messages to the same partition using a key:

// All events for user-123 go to the same partition
ProducerRecord<String, OrderEvent> record = new ProducerRecord<>(
    "orders",
    order.getUserId(),  // Key determines partition
    orderEvent
);
producer.send(record);

How It Works

Kafka uses consistent hashing on the message key to route to partitions:

Same key always goes to the same partition = guaranteed order. Different keys go to different partitions = parallel processing.

Complete Example

@Service
public class OrderEventProducer {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public void sendOrderEvent(OrderEvent event) {
        // Key = orderId ensures all events for an order are ordered
        kafkaTemplate.send("order-events", event.getOrderId(), event);
    }
}

// Events sent in order:
// 1. ORDER_CREATED (order-123)
// 2. PAYMENT_RECEIVED (order-123)
// 3. ORDER_SHIPPED (order-123)
//
// Consumer receives in same order because same key → same partition

---

Strategy 3: Idempotent Producer (Prevent Duplicates)

Network failures can cause retries that duplicate or reorder messages. Idempotent producers fix this:

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");  // Key setting
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 5);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);

How Idempotence Works

Producer                    Broker
   │                           │
   │── Send M1 (seq=1) ──────▶│  ✓ Stored
   │                           │
   │── Send M2 (seq=2) ──────▶│  ✓ Stored
   │                           │
   │── Timeout (network)       │
   │                           │
   │── Retry M2 (seq=2) ─────▶│  ✗ Duplicate detected, ignored
   │                           │
   │◀── ACK ──────────────────│

Key settings:

Config	Value	Purpose
enable.idempotence	true	Enable dedup
acks	all	Wait for all replicas
max.in.flight.requests	≤5	Required for idempotence

---

Strategy 4: External Sequencing (Global Order)

When you need order across ALL partitions (rare but sometimes necessary):

Producer: Add Sequence Number

public class SequencedProducer {

    private final AtomicLong sequenceGenerator = new AtomicLong(0);
    private final KafkaProducer<Long, Event> producer;

    public void send(Event event) {
        long seq = sequenceGenerator.incrementAndGet();
        event.setGlobalSequence(seq);
        event.setTimestamp(System.nanoTime());

        // Use sequence as key for consistent routing
        producer.send(new ProducerRecord<>("events", seq, event));
    }
}

Consumer: Buffer and Reorder

public class OrderedConsumer {

    private final List<Event> buffer = new ArrayList<>();
    private final Duration bufferWindow = Duration.ofMillis(500);
    private Instant lastProcessTime = Instant.now();

    public void poll() {
        ConsumerRecords<Long, Event> records = consumer.poll(Duration.ofMillis(100));

        for (ConsumerRecord<Long, Event> record : records) {
            buffer.add(record.value());
        }

        // Process buffer when window expires
        if (Duration.between(lastProcessTime, Instant.now()).compareTo(bufferWindow) > 0) {
            processBuffer();
            lastProcessTime = Instant.now();
        }
    }

    private void processBuffer() {
        // Sort by global sequence number
        buffer.sort(Comparator.comparingLong(Event::getGlobalSequence));

        for (Event event : buffer) {
            process(event);
        }
        buffer.clear();
    }
}

Trade-offs

Aspect	Impact
Latency	Adds buffer window delay
Memory	Buffer grows with throughput
Complexity	State management required
Late arrivals	Messages after window missed

---

Configuration Deep Dive

Producer Settings

Properties props = new Properties();

// Ordering guarantee: process one request at a time
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, 1);

// Batching: balance latency vs throughput
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384);  // 16KB batches
props.put(ProducerConfig.LINGER_MS_CONFIG, 5);       // Wait 5ms for batch

// Reliability
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

Consumer Settings

Properties props = new Properties();

// Batch size: affects ordering complexity
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);

// Fetch settings
props.put(ConsumerConfig.FETCH_MIN_BYTES_CONFIG, 1);
props.put(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG, 500);

// Offset management
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);  // Manual commit
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

---

Decision Framework

Use this flowchart to choose your ordering strategy:

Most applications need per-entity ordering (same user, same order) which is solved with key-based partitioning. Global ordering requires a single partition, sacrificing parallelism.

---

Spring Kafka Example

@Configuration
public class KafkaConfig {

    @Bean
    public ProducerFactory<String, OrderEvent> producerFactory() {
        Map<String, Object> config = new HashMap<>();
        config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        config.put(ProducerConfig.ACKS_CONFIG, "all");
        config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
        return new DefaultKafkaProducerFactory<>(config);
    }

    @Bean
    public KafkaTemplate<String, OrderEvent> kafkaTemplate() {
        return new KafkaTemplate<>(producerFactory());
    }
}

@Service
public class OrderService {

    @Autowired
    private KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public void processOrder(Order order) {
        // All events for same order go to same partition
        kafkaTemplate.send("orders", order.getId(),
            new OrderEvent(order.getId(), "CREATED", Instant.now()));
    }
}

@KafkaListener(topics = "orders", groupId = "order-processor")
public void handleOrderEvent(OrderEvent event) {
    // Events for same orderId arrive in order
    log.info("Processing: {} for order {}", event.getType(), event.getOrderId());
}

---

Common Pitfalls

Pitfall	Symptom	Fix
No message key	Random partition assignment	Always set key for ordered entities
max.in.flight > 1 without idempotence	Reordering on retry	Enable idempotence or set to 1
Consumer rebalance	Processing restarts mid-sequence	Use sticky assignor, handle carefully
Key cardinality too low	Hot partitions	Use composite keys

---

Code Repository

Complete examples with Testcontainers:

GitHub: techyowls/techyowls-io-blog-public/kafka-message-ordering

git clone https://github.com/techyowls/techyowls-io-blog-public.git
cd techyowls-io-blog-public/kafka-message-ordering
./mvnw test  # Runs ordering tests with Testcontainers

---

Further Reading

• Spring AI MCP Guide - Event-driven AI
• Java 21 Virtual Threads - Scale Kafka consumers
• Kafka Documentation

---

Build event-driven systems that don’t lose order. Follow TechyOwls for more practical guides.