Partition consumer hangs when reading messages from beginning of compacted topic

[akamensky]

Versions

Sarama	Kafka	Go
1.27.2	2.6.0	1.15.4

Configuration

What configuration values are you using for Sarama and Kafka?

	// Common settings
	config := sarama.NewConfig()
	config.Version = sarama.V2_6_0_0
	config.ClientID = "ksnap"

	// Consumer settings
	config.Consumer.Offsets.AutoCommit.Enable = false
	config.Consumer.Offsets.Initial = sarama.OffsetOldest

Logs

logs: CLICK ME

The log below is a log output as seen when enabling logging in Sarama. It hangs at the last line and keeps doing-nothing indefinitely.

[Sarama] 2020/08/19 14:40:37 Initializing new client
[Sarama] 2020/08/19 14:40:37 client/metadata fetching metadata for all topics from broker kafka04:9092
[Sarama] 2020/08/19 14:40:37 Connected to broker at kafka04:9092 (unregistered)
[Sarama] 2020/08/19 14:40:37 client/brokers registered new broker #5 at kafka05:9092
[Sarama] 2020/08/19 14:40:37 client/brokers registered new broker #4 at kafka04:9092
[Sarama] 2020/08/19 14:40:37 client/brokers registered new broker #6 at kafka06:9092
[Sarama] 2020/08/19 14:40:37 client/brokers registered new broker #2 at kafka02:9092
[Sarama] 2020/08/19 14:40:37 client/brokers registered new broker #3 at kafka03:9092
[Sarama] 2020/08/19 14:40:37 Successfully initialized new client
[Sarama] 2020/08/19 14:40:37 Connected to broker at kafka03:9092 (registered as #3)
[Sarama] 2020/08/19 14:40:37 consumer/broker/3 added subscription to TopicName/0

Code

https://github.com/EclipseTrading/ksnap

Problem Description

We are using sarama to read messages from beginning of partition until last message, most topics work fine, but we have a set of topics which are being republished every morning (compact topics with unique keys, keys being republished every morning to overwrite values). These topics always (every time) hang on reading very first message. I've ran a debugger and inspected the routines -- the one that i found appears to wait indefinitely for response from server (which may or may not have arrived already, as I could not see that in debugger).

We have two other applications reading from this topic (one in Java using standard client and one in C++ using librdkafka), both applications have no problems of consuming this partition from the very first offset.

[akamensky]

Anyone every looking at those issues here? I see almost no comments/reaction from contributors/maintainers.

Anyway, I've been digging into this issue and found following:

• the issue happens on compact only topics that are produced to by clients using librdkafka (which does something weird with publishing tombstones)
• the hang ALWAYS happens in the same offset -- where messages go from 0 len value to len value > 0 (see screenshot below in that case last consumed message before hang is always offset 8676711)
• if I detect the hang and increase offset manually -- it still hangs on ALL offsets between 8676711 and 8699311
• if I set next offset to consume to 8699312 it can consume the rest of the topic without any issues
• Java clients have no such problem as can be seen by the fact that kafkatool can read this topic w/o any issue

Last point above makes me think that sarama implements something differently than expected and that needs to be fixed in sarama. Unfortunately I can't say yet what it is that needs to be fixed.

[SladeThe]

I have a similar issue. Though the reason could be different.

I have a producer, which is subscribed to a few partitions, but it gets messages only from some of them.
In some cases it gets a few messages from a partition, then stops.
In other cases it completely ignores a partition.

I checked another Go-client, works fine.

[akamensky]

@SladeThe I tried multiple including confluent one (that uses hated cgo). But they all hung there for me. Which client you tried that worked for you?

[SladeThe]

@akamensky segmentio/kafka-go

[akamensky]

Still same issue with 1.28.0 no one from maintainers ever looking at these issues?

[xfzhu01]

Still same issue with 1.29.1

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[hamilton-earthscope]

I'm running into this same issue while trying to use Sarama with Goka. Have any of you found a workaround for this?

Using 1.42.1 and go 1.20

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.