The correspondence between consumer groups and consumers is as follows:
A consumer can subscribe to multiple topics simultaneously.
A topic contains one or more partitions.
A partition can only be consumed by one consumer.
Therefore: The maximum number of consumers in a consumer group = Number of partitions in topic1 + Number of partitions in topic2 + ... + Number of partitions in topicN.
Regarding the definition of a consumer: At the code level, a consumer refers to a consumer object. Multiple consumers can exist on a single server. For example, multiple threads may be launched, each containing one consumer, and so on, as shown in the following code:
ConsumerRecords<String, String> records = consumer.poll(100);
}
The initial number of consumers can be deployed based on client resources. Then configure a backlog alarm for the consumer group. If a backlog occurs, scale out consumers. For how to configure an alarm, see Configuring Alarm Policies.
Why Are Consumer Details Invisible?
Consumer details include the topics consumed by the consumer, partition progress, and clients. The information is stored by the server after the consumer processes messages and commits offsets to the server. The client will not automatically commit offset storage requests to the server in the following scenarios:
1. A custom partition consumption mode is used.
2. Flink and TDMQ for CKafka (CKafka) are used for big data scenarios. Since the Flink cluster does not commit offsets or perform a partition rebalance via a coordinator, the server lacks the information about the corresponding consumption status and the IP address of the consumption Pod.
What Should I Do If the Details of the Consumer Group List Are Missing?
Phenomenon Overview
In the consumer group list of CKafka, the consumer group name is displayed, but clicking Details shows no consumption details. For example, the consumer group CR in the figure below does not display details.
Possible Causes
Kafka supports two data consumption modes: consumer group mode and custom partition consumption mode.
When consuming in consumer group mode, the client coordinates consumption through a consumer coordinator. After completing data consumption, it commits an offset storage request to the server. The server then stores information such as the consumed topics, partition progress, and clients.
When consuming in custom partition consumption mode, the client does not automatically commit an offset storage request to the server. If the client does not proactively initiate an offset commit request, the server cannot see the relevant consumption information.
After access control lists (ACLs) have been set for a topic, certain instances may be unable to view consumer group details. Therefore, if the details cannot be viewed, first check for ACLs. If you find ACLs, submit a ticket for assistance.
Note: This will not show information about old Zookeeper-based consumers.
You will find that this consumer group has no details. This indicates that the consumer client did not use the consumerGroup mechanism to consume data. Since the client did not submit consumption details to the server, and the server did not store consumption data, the information is not displayed normally.
3. Determine whether the issue is with the server.
4. Specify the consumer group name test1 for consumption by running the native consumer group command, as shown below:
]$ bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --from-beginning --topic test --group test1
For consumer groups that are displayed normally in the console, you can view their details by running the --describe command, as shown below:
Can Subscribe-Type Consumer Groups and Assign-Type Consumer Groups Reuse the Same Group ID?
In CKafka, the management methods for subscribe-type and assign-type consumer groups are entirely different. The information of subscribe-type consumer groups is fully managed by the CKafka server, including consumer metadata, committed offsets, and rebalance management. In contrast, for assign-type consumer groups, the logic of partition assignment for each consumer is entirely managed by the client itself, with the server only storing the offsets committed by the assign-type consumer group. Therefore, if subscribe-type and assign-type consumer groups reuse the same group ID, the following unexpected behaviors may occur:
1. If a subscribe-type consumer group is in a stable status and consuming, starting an assign-type consumer group with the same group ID may cause offset commit failures for the assign-type consumer group.
2. If an assign-type consumer group is consuming and committing offsets, starting a subscribe-type consumer group with the same group ID may cause subsequently committed offsets from the subscribe-type consumer group to overwrite those of the assign-type consumer group. This can lead to offset confusions and commit failures for the assign-type consumer group.
3. If a subscribe-type consumer group stops consumption after being used and enters an empty status, starting an assign-type consumer group with the same group ID may allow normal consumption and offset commits. However, since the server stores this group ID as a subscribe-type group including its consumer metadata, when the metadata expires (7 days by default), the consumer group will be directly deleted via subscribe-type expiration cleanup. This can cause synchronous deletion of the consumer group's offsets, and the ongoing assign-type consumer group may encounter issues due to offset loss.
Therefore, avoid this improper usage. If you must use the same group ID name while the consumer group types differ before and after, you need to delete the consumer group in the console first and then enable a new type of consumer group for consumption.
What Should I Do If the Consumer Group Continuously Stays in the PreparingRebalance Status?
Possible Causes
1. A new consumer joins the consumer group.
2. A running consumer stops functioning and leaves the consumer group. Common situations include a consumer restart, consumer application crash, and heartbeat timeout reported by the consumer process (for details, see Configuring Client Parameters).
3. The number of partitions changes (due to addition or deletion).
Solutions
Two situations, 1 and 3, cannot avoid a rebalance. Under normal conditions, a rebalance completes within 30 seconds. If a long-term rebalance occurs, submit a ticket for assistance.
If the issue is caused by a heartbeat timeout or an excessive interval between poll calls, you can adjust the following parameters:
# Consumer timeout when the Kafka consumer group mechanism is used. When the broker does not receive a heartbeat from the consumer within this time, the consumer is considered to have failed, and the broker restarts the rebalance process. Currently, this value must be set in the broker, with group.min.session.timeout.ms set to 6000 and group.max.session.timeout.ms set to 300000.
session.timeout.ms=10000
# Interval at which consumers send heartbeats when the Kafka consumer group mechanism is used. This value must be less than session.timeout.ms, typically less than one-third of that value.
heartbeat.interval.ms=3000
# Maximum interval allowed between successive poll calls when the Kafka consumer group mechanism is used. If the poll is not called again within this period, the consumer is considered to have failed, and the broker will restart the rebalance process to assign its partitions to other consumers.
max.poll.interval.ms=300000
If a consumer subscribes to multiple topics, you can try to reduce the number of topics subscribed to by the group.
