Message groups - any plans for this?

[alekbarszczewski]

I read this issue #262 which is open since 5 years. Does it mean there are no plans adding this feature at all? If yes, what are the reasons that it won't/can't be added?

In my opinion this is one of most useful features, both Kafka and ActiveMQ support this. It allows processing messages in order with some degree of concurrency. For now the only way to preserve message order is to either use single active consumer with prefetch_count = 1, which allows only one message processing at the same time (even if some of the messages could be processed in parallel) or use rabbitmq_consistent_hash_exchange which complicates topology a lot I think.

Here is how it works in ActiveMQ: https://activemq.apache.org/message-groups

If there is RoundRobin message distribution between consumers in RabbitMQ then is it complicated to add hash-based distribution?

[kjnilsson]

You don’t have to use prefetch of 1 for single active consumer ordering. Any prefetch is fine and it will retain the input order. SAC with consistent hash exchange is a good but underused pattern to follow IMO.

On Sun, 5 Jun 2022 at 11:13, Aleksander Barszczewski < ***@***.***> wrote: I read this issue #262 <#262> which is open since 5 years. Does it mean there are no plans adding this feature at all? If yes, what are the reasons that it won't/can't be added? In my opinion this is one of most useful features, both Kafka and ActiveMQ support this. It allows processing messages in order with some degree of concurrency. For now the only way to preserve message order is to either use single active consumer with prefetch_count = 1, which allows only one message processing at the same time (even if some of the messages could be processed in parallel) or use rabbitmq_consistent_hash_exchange <https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_consistent_hash_exchange> which complicates topology a lot I think. Here is how it works in ActiveMQ: https://activemq.apache.org/message-groups If there is RoundRobin message distribution between consumers in RabbitMQ then is it complicated to add hash-based distribution? — Reply to this email directly, view it on GitHub <#4974>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFCTAAOOHENHEBAK3X3VNR4VBANCNFSM5X42TPZA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- *Karl Nilsson*

[michaelklishin]

The reason is as simple as "there are other priorities and the team is small".

[acogoluegnes]

We are working on single active consumer on streams and super streams for RabbitMQ 3.11, which combined provide the same kind of semantics and scalability of message groups (ActiveMQ) or consumer groups (Kafka). #3754 implements the bulk of it and you are welcome to try it out (Docker and JDK 11+ required).

[vincentjames501]

This is definitely something we've also wanted and is the sole reason we have to use SQS in production (for their FIFO message groups). We need to queue jobs by customer that MUST run in order but we also need to parallelize across multiple nodes/consumers.

We can do consistent hashing and SAC as @alekbarszczewski , but it pushes the complexity of topology onto the developer trying to balance consumers by number of queues (something Kafka, SQS, etc to automatically).

@acogoluegnes , the streams SAC is neat, though the semantics of a stream are a bit different than that of a classic queue especially for managing long running tasks.

I'm also interested in @alekbarszczewski question. Are there any technical reasons that it can't be added? Major challenges?

[milanovdx]

@kjnilsson Indeed SAC and the consistent hash exchange plugin is a great combination. We've been using it in production and it's very reliable. One problem with that setup is the difficulty in horizontally scaling the consumers, specifically of different service instances. I've described it in more detail in https://groups.google.com/g/rabbitmq-users/c/v5BsdWd9AlQ.

[ansd]

@milanovdx the problem you described is solved in RabbitMQ 4.0. See https://groups.google.com/g/rabbitmq-users/c/v5BsdWd9AlQ/m/EAsp9I0fAgAJ

[ansd]

I read this issue #262 which is open since 5 years. Does it mean there are no plans adding this feature at all? If yes, what are the reasons that it won't/can't be added?

#262 is implemented in RabbitMQ 4.1, see #262 (comment)