Replies: 2 comments 6 replies
-
|
@malmetom according to the logs, all replicas report a missing local file. This is not a known problem. Perhaps the stream could never finish replicating even the initial "chunk". Can you share data directories of these nodes? That's the only way to suggest anything specific. |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin thanks for reaching out I got the issue again, see below for directories before(everything looks fine) and after one node has been isolated Before partition: Node 1: Node 2: Node 3: Leader was on node 58 58 was isolated and leader moved to node 57 After partition: log getting repeated errors: Node 2: user@57: Log getting repeated warnings: Sorry for the strikethroughs in the attachd text above, shouldn't be there Also those are the feature flags in use: Flag: classic_mirrored_queue_version, state: enabled Seems like directory exists, but files inside are different and even increased on the new leader. The stream uses x-max-length-bytes: 3072 and x-stream-max-segment-size-bytes: 256 |
Beta Was this translation helpful? Give feedback.
-
Yes most definitely. It is on oversight on our behalf to allow such small I would strongly recommend re-declaring the stream with at least 16MB Related: rabbitmq/osiris#170 |
Beta Was this translation helpful? Give feedback.
-
|
Ok, thanks, will change. Got same issue with 4.0.9. I can also see that there is a period of only one node up I had a case with node 58 being the leader for the stream and in the log I see 2025-04-17 10:09:54.025656+02:00 [info] <0.419.0> rabbit on node 'rabbit@56.rabbitmq.service.consul' down then starts to get a few warnings about replication 2025-04-17 10:11:54.561324+02:00 [warning] <0.23956.0> rabbit_stream_coordinator: Error while starting replica for __rssi_ble_stream_1744874840828243762 on node rabbit@57.rabbitmq.service.consul in 2 : {{bad_return_value,missing_file},{gen_server,call,[<14766.452.0>,await,infinity]}} Before 57 comes up again 2025-04-17 10:11:54.646499+02:00 [info] <0.419.0> rabbit on node 'rabbit@57.rabbitmq.service.consul' up So seems like there is a period of ~24s with only leader up and running, should it be possible for the stream to survive that as long as the max-length-bytes and max-segment-size-bytes have reasonable values? |
Beta Was this translation helpful? Give feedback.
-
|
I haven't spent a lot of time investigating but try a larger segment size and see if it works better. I'm not sure why you would have such a tiny segment size anyway. |
Beta Was this translation helpful? Give feedback.
-
|
we're considering shipping a change to claim segment sizing to a reasonable size if configuration requested is too small. |
Beta Was this translation helpful? Give feedback.
-
|
@malmetom also, if you can reproduce this behavior reliably, please use |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin Will give it a try with a newer rabbitmq, and come back with result |
Beta Was this translation helpful? Give feedback.
-
Community Support Policy
RabbitMQ version used
4.0.8
Erlang version used
27.3.x
Operating system (distribution) used
Ubuntu 22.04
How is RabbitMQ deployed?
Community Docker image
rabbitmq-diagnostics status output
See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics
Logs from node 1 (with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 2 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 3 (if applicable, with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
rabbitmq.conf
See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location
Steps to deploy RabbitMQ cluster
Three node cluster
Steps to reproduce the behavior in question
advanced.config
See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location
Application code
# PASTE CODE HERE, BETWEEN BACKTICKSKubernetes deployment file
What problem are you trying to solve?
I run a 3-node setup with a stream created with node 2 as leader. After network partition and restoring to 3 nodes again, the leader cannot create replica on the non-leader nodes. Is this a known issue, what's might be missing in the setup to handle a situation like this?
Beta Was this translation helpful? Give feedback.
All reactions