Why Does the Pulsar Server Suddenly Stop Pushing Messages to Clients, with Partial or All Partitions Showing Zero Consumption Speed?
To balance performance across different topics and clients, Pulsar sets a default upper limit of unacknowledged messages (number of unack messages) per partition subscription for a topic. The default value is 5,000, as specified in the Use Limits. To check if a subscription has reached this upper limit, observe whether the number of unacknowledged messages for the subscription has hit the quota limit. When the total number of unacknowledged messages for a subscription on a partition exceeds 5,000, the server will stop pushing messages to consumers of the partition's subscription. Unacknowledged messages refer to messages that have been pushed but not acknowledged by the client through ack(), negativeAcknowledge(), or reconsumeLater(). Common solutions include:
1. First, determine whether the messages are unacknowledged due to inherently long business processing times, or whether logical exceptions exist in the business code where messages are not promptly acknowledged, causing message gaps. To do so, obtain the msgId of gap messages from the topic partition subscription and then query the message content and push records in the message trace. For detailed information, see Gap Messages. 2. If the messages involve long processing times, gap messages are normal. As the upper limit of unacknowledged messages has been reached, horizontal scaling of consumers is required to enhance consumption capacity. In this case, it is recommended to reduce the ReceiverQueueSize configuration for individual clients, with a suggested value of 2,500 divided by the number of consumers.
3. If analysis of gap messages reveals that the business code of consumers failed to acknowledge messages in certain scenarios, you need to restart the consumer clients corresponding to the unacknowledged messages after you fix the acknowledgment-related logic in the consumer code and deploy a new version. After restarting, the server will automatically push the unacknowledged messages again.