How to compare two Kafka topics

Create a diff of two Kafka topics, including messages, keys, headers, and configuration.

Problem Statement

When working with Apache Kafka, there are many scenarios where you need to verify that two topics are identical, contain the same data or at least consistent. This is especially important if you:

• Are validating replication between clusters (e.g., using MirrorMaker or Confluent Replicator)
• Have restored a topic from an external backup or another source and want to ensure data integrity
• Need to compare production and staging topics for debugging, compliance, or migration
• Want to check that compaction or retention policies have not caused unexpected data loss
• Are troubleshooting issues with message ordering, duplication, or partitioning

Manually comparing topics is tedious and error-prone, especially for large topics or when you need to check not just message values, but also keys, headers, and ordering. Differences in topic configuration, partitioning, or compaction can make this even more challenging.

Solution: kafka-topic-compare

Compare Topics

kafka-topic-compare is an open-source tool designed to solve this problem for developers and operators. It provides a robust, CLI-based way to compare two Kafka topics, highlighting differences in messages, keys, values, headers, and topic configuration.

Key Features

• Compares two Kafka topics (A and B) for differences in messages, keys, values, and headers
• Detects:
• Messages only in topic A or B
• Duplicate messages in either topic
• Messages with the same timestamp, key and value but different headers
• Out-of-order messages (with improved detection, tolerant to constant offset shifts and duplicates)
• Messages located on different partitions
• Support for time-based starting point for comparison (via --startTimestamp)
• Handles large topics and configurable message limits per partition (via --maxMessages)
• Output in CSV (default) or JSON format for easy parsing and analysis
• Compaction-aware: can compare only the latest version of each key for compacted topics
• Logs differences in topic properties/configuration before comparing messages
• Output is ordered by topicA, partitionA, offsetA for easier analysis
• Easy to run locally or in CI/CD pipelines

Getting Started

Installation

Requirements:

• Java 17 or newer

Or download the latest release directly:

curl -L https://github.com/spoud/kafka-topic-compare/releases/latest/download/kafka-topic-compare-1.0-SNAPSHOT-runner.jar -L -o kafka-topic-compare.jar

Building the project:

Requirements:

• Java 17 or newer
• Maven (or use the included mvnw wrapper)

Clone the repository and build the project:

git clone https://github.com/spoud/kafka-topic-compare.git
cd kafka-topic-compare
./mvnw clean package

The built JAR will be in target/quarkus-app/quarkus-run.jar.

Usage Example

To compare the first 100 messages per partition of two topics:

cp target/quarkus-app/quarkus-run.jar ./kafka-topic-compare.jar
java -jar kafka-topic-compare.jar \
  --bootstrapA localhost:9092 --topicA topicA \
  --bootstrapB localhost:9093 --topicB topicB \
  --maxMessages 100

Key Arguments

• --bootstrapA Kafka bootstrap servers for topic A (default: localhost:9092)
• --topicA Name of topic A (default: topicA)
• --bootstrapB Kafka bootstrap servers for topic B (default: localhost:9093)
• --topicB Name of topic B (default: topicB)
• --maxMessages Maximum number of messages to compare from each topic (default: 1000)
• --output or -o Output format: csv (default) or json
• --startTimestamp Only compare messages with timestamp >= this value
• --print-diff Print detailed differences for messages with the same key but different values/headers
• --skip-header Optional comma-separated list of header names to skip in diff (or empty to disable header comparison)
• --skip-missing-at-end Skip logging/reporting of differences of type MISSING_AT_END`
• --help Show help and exit
• --debug Enable detailed logging for troubleshooting

Example Output

CSV (default):

type,bootstrapA,topicA,partitionA,offsetA,bootstrapB,topicB,partitionB,offsetB
ONLY_IN_A,localhost:9092,topicA,0,10,localhost:9093,topicB,,
ONLY_IN_B,localhost:9092,topicA,,,localhost:9093,topicB,1,15
HEADER_DIFFERENCE,localhost:9092,topicA,0,12,localhost:9093,topicB,1,17
OUT_OF_ORDER,localhost:9092,topicA,0,20,localhost:9093,topicB,0,18
PARTITION_DIFFERENCE,localhost:9092,topicA,2,30,localhost:9093,topicB,3,30

JSON:

{"type":"ONLY_IN_A","bootstrapA":"localhost:9092","topicA":"topicA","partitionA":0,"offsetA":10,"bootstrapB":"localhost:9093","topicB":"topicB","partitionB":null,"offsetB":null}

Topic Configuration Diff

Before comparing messages, the tool logs a diff of all topic configuration properties to stderr, in the format:

property,bootstrapA,valueA,bootstrapB,valueB
cleanup.policy,localhost:9092,compact,localhost:9093,delete

This helps you spot differences in retention, compaction, or other critical settings that might explain data mismatches. The tool also checks for partition count differences.

Debugging and Logging

Use the --debug flag to enable detailed logging, which can help troubleshoot issues during comparison. The tool will also fail fast and provide clear error messages if it cannot connect to the specified bootstrap servers.

Contributing and Repository

The project is open source and welcomes contributions! Visit the repository for source code, issues, and more documentation:

👉 https://github.com/spoud/kafka-topic-compare

Conclusion

kafka-topic-compare is a powerful utility for anyone who needs to verify Kafka topic consistency, whether for replication, backup/restore, migration, or debugging. It’s easy to use, flexible, and provides deep insight into both data and configuration differences.

For the latest updates, features, and support, check out the GitHub repository. Happy comparing!