fix(ci): Make integration tests on Linux arm64 not flaky#4313
Conversation
| "message.max.bytes": 50000000, | ||
| "socket.timeout.ms": 1000, | ||
| "socket.timeout.ms": 10000, # Timeout for individual socket operations (send/recv) | ||
| "request.timeout.ms": 30000, # Max time to wait for a broker response before failing |
There was a problem hiding this comment.
From logs:
uptime-results-1 | %4|1778036634.157|SESSTMOUT|rdkafka#consumer-1| [thrd:main]: Consumer group session timed out (in join-state steady) after 30400 ms without a successful response from the group coordinator (broker 1001, last error was Success): revoking assignment and rejoining group
searching "Consumer group session timed out" shows a couple more kafka timeouts after 30s as well, maybe we should increase kafka timeout even further?
There was a problem hiding this comment.
To be honest, I didn't catch this myself, it's been a while since I also use AI to review stuff :)
There was a problem hiding this comment.
But the problem persist. I have no idea other to blame github's arm64 runners.
There was a problem hiding this comment.
Yeah, they are too slow, this happens to me as well in other projects.
I searched "Consumer group session timed out" in latest run:
process-spans-1 | %4|1779437870.346|SESSTMOUT|rdkafka#consumer-1| [thrd:main]: Consumer group session timed out (in join-state steady) after 31691 ms without a successful response from the group coordinator (broker -1, last error was Success): revoking assignment and rejoining group
Seems like they still have 30 seconds timeout.
There was a problem hiding this comment.
This is according to AI:
Root cause summary:
- Silent config drop (Bug 1): KAFKA_CLUSTERS["default"] = DEFAULT_KAFKA_OPTIONS uses the legacy flat format. get_kafka_consumer_cluster_options() detects this and, because it's called with only_bootstrap=True, extracts only bootstrap.servers — all other settings including
session.timeout.ms and heartbeat.interval.ms are silently discarded. - HACK override (Bug 2): Even if the settings survived, build_consumer_config in consumers/init.py unconditionally overrides session.timeout.ms to match max_poll_interval_ms when that value is < 45000ms. Since --max-poll-interval-ms defaults to 30000 in run.py,
session.timeout.ms is force-set to 30s — which is exactly what the ~31.7s timeout error reflects.
Recommended fix (no code change required): Pass --max-poll-interval-ms 300000 (or any value ≥ 45000) to the sentry run consumer process-spans command in the self-hosted docker-compose.yml. This disables the HACK and allows the session timeout to stay at the safe default of
45s.
Let me try to mess things up again
I'm happy already, but let me rerun the test one more time |
|
re-running it here: https://github.com/getsentry/self-hosted/actions/runs/26268331023 |
sigh, still flaky |
3 participants
Closes SELF-93