Kafka offset清理

Requirement

• jdk 1.8及以上
• Kafka 软件包 2.0及以上

执行路径

将Kafka软件包解压，进入到bin目录

[bin]# ls
connect-distributed.sh        kafka-consumer-perf-test.sh          kafka-reassign-partitions.sh        trogdor.sh
connect-mirror-maker.sh       kafka-delegation-tokens.sh           kafka-replica-verification.sh       windows
connect-standalone.sh         kafka-delete-records.sh              kafka-run-class.sh                  zookeeper-security-migration.sh
kafka-acls.sh                 kafka-dump-log.sh                    kafka-server-start.sh               zookeeper-server-start.sh
kafka-broker-api-versions.sh  kafka-leader-election.sh             kafka-server-stop.sh                zookeeper-server-stop.sh
kafka-configs.sh              kafka-log-dirs.sh                    kafka-streams-application-reset.sh  zookeeper-shell.sh
kafka-console-consumer.sh     kafka-mirror-maker.sh                kafka-topics.sh
kafka-console-producer.sh     kafka-preferred-replica-election.sh  kafka-verifiable-consumer.sh
kafka-consumer-groups.sh      kafka-producer-perf-test.sh          kafka-verifiable-producer.sh

操作步骤

以kafka地址为127.0.0.1:9092, topic名为test-h，消费组名为hhh举例

先查看offset

./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group hhh --describe
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
hhh             test-h          0          18              18              0               -               -               

如果LAG为0，那么也无需操作，没有积压，当LAG很大的时候，考虑进行规避操作。以LAG为5举例

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
hhh             test-h          0          18              23              5               -               -               -

那么指定offset进行consumer offset切换

./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group hhh --reset-offsets --to-offset 22 --topic test-h --execute

指定22，则LAG只剩下1条。

也可以直接指定最新offset进行切换

./kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --group hhh --reset-offsets --to-latest --topic test-h --execute