4.0.7: questions about disk and memory footprint of classic queues with larger messages

[ronmegini]

Environment Details

• Upgrade Path: Upgraded from a single-node RabbitMQ 3.11 to a 3-node cluster (v4.0.5).
• Deployment Method: Using Bitnami Helm Chart v15.2.1 on Kubernetes v1.30.
• Cluster Configuration:
  • Nodes: 3 (each running RabbitMQ)
  • Resources per Node: 4 vCPU, 28 GiB memory, 250 GiB storage.
• Workload:
  • Message Throughput: ~400 messages/sec.
  • Message Size: ~0.5 MB per message.
  • Queue Type: Classic queues.
• Other Clusters: We operate multiple RabbitMQ clusters across different regions, all with identical configurations and workloads, but only this one exhibits the issue.

Issue Observed

One of the RabbitMQ pods in the cluster is consuming unexpectedly high memory and storage, while other pods in the same cluster and across different environments do not exhibit this issue.

Potential Storage Leak

• The total queue storage (message body size) on the affected pod is ~90 GiB.
• However, the actual storage usage on the node has grown to 240 GiB.
• Question: Where is the extra 150 GiB of storage consumption coming from?

Potential Memory Leak

• RabbitMQ Overview Dashboard reports 15 GiB free memory. (with 0.6 memory watermark)
• However, Kubernetes metrics show only ~0.5 GiB free.
• Erlang Memory Allocators Dashboard indicates RabbitMQ itself is using only ~1 GiB.
• Pod-level memory consumption is ~27 GiB (far higher than expected).
• Question: Why is the pod consuming so much memory when RabbitMQ reports low usage?

Additional Observations

• This issue only affects one pod in the cluster—all other pods in the same cluster and in other regions are operating normally. (potentially happen less significantly in one of the qa clusters)
• All configurations (memory settings, persistence, message TTL, etc.) are identical across clusters.
• The workload is consistent across all environments, making this behavior unexpected.
• The affected pod eventually hits the storage watermark, yet storage usage continues increasing until the node crashes due to OOM (Out of Memory).
• After a restart, the excess storage is cleared, but the high, unexplained memory and storage usage persist—even though RabbitMQ itself does not report abnormal resource consumption.

Reproduction steps

rabbitmq:
  global:
    security:
      allowInsecureImages: true

  image:
    registry: ***********.dkr.ecr.eu-central-1.amazonaws.com
    repository: rabbitmq
    tag: 4.0.5-debian-12-r0
  
  auth:
    username: admin
    existingPasswordSecret: rabbitmq-password-secret
    tls:
      enabled: true
      autoGenerated: false
      failIfNoPeerCert: false
      sslOptionsVerify: verify_none
      existingSecret: rabbitmq-amqp-cert
      existingSecretFullChain: true

  memoryHighWatermark:
    enabled: true
    type: "relative"
    value: 0.6

  clustering:
    enabled: true

  extraConfiguration: |-
    disk_free_limit.absolute = 10GB

  replicaCount: 3

  podAntiAffinityPreset: hard

  tolerations:
  - key: rabbitmq-on-demand
    operator: Exists
  
  resources:
    limits:
      memory: 28Gi
    requests:
      cpu: "4"
      memory: 28Gi

  persistence:
    enabled: true
    size: 250Gi

  service:
    portEnabled: false
    annotations:
      external-dns.alpha.kubernetes.io/hostname: amqp-int.prv.prod.ec1.cyner.io

  ingress:
    enabled: true
    hostname: rabbitmq.prv.prod.ec1.cyner.io

  metrics:
    enabled: true
    plugins: "rabbitmq_prometheus"
    serviceMonitor:
      enabled: true
      namespace: cross
      labels:
        release: kube-prometheus-stack
      default:
        enabled: true
      detailed:
        enabled: true
        family: [queue_metrics, queue_coarse_metrics, queue_delivery_metrics]
    prometheusRule:
      enabled: true
      namespace: cross
      labels:
        release: kube-prometheus-stack

Expected Behavior

• Memory reported in RabbitMQ's dashboard should match actual pod memory usage.
• Storage usage should not exceed the total queue size unless justified (e.g., logs, persistence overhead).

[michaelklishin]

@ronmegini I don't see any evidence of a leak. This should have been a discussion, not an issue.

Messages are stored on disk with a fair amount of metadata. When messages are consumed and confirmed, they are not deleted immediately, neither by CQs nor by QQs and certainly not streams (where consumptions is non-destructive).

Quorum queues delete entire segment files, not individual messages and CQs compact segment files when more than half of two files have "holes" (messages marked for deletion).

The explanation for one pod having an outsized footprint is almost always as simple as: that pod hosts the majority of replicas, or, in your case, the majority of CQs which in 4.x are a non-replicated type.

The runtime allocates memory without using it immediately, which is explained in the Reasoning About Memory Use guide.

There is kernel page cache which Kubernetes before a certain version (1.25 or so) or certain cgroups version (v1) is considered to be a part of the process' footprint even thought that's completely incorrect, which our docs also mention.

[ronmegini]

@michaelklishin Thanks for the explanation!
Please let me know if these numbers aligned:

Memory allocation - 27GiB cached:

I have no name!@rabbitmq-cluster-1:/$ rabbitmq-diagnostics memory_breakdown
Reporting memory breakdown on node rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local...
binary: 0.8483 gb (60.12%)
allocated_unused: 0.3112 gb (22.06%)
reserved_unallocated: 0.0589 gb (4.17%)
msg_index: 0.0401 gb (2.84%)
other_proc: 0.0277 gb (1.96%)
code: 0.0246 gb (1.75%)
other_system: 0.0214 gb (1.52%)
connection_other: 0.0143 gb (1.01%)
mgmt_db: 0.0132 gb (0.94%)
queue_procs: 0.0105 gb (0.74%)
plugins: 0.0085 gb (0.6%)
connection_channels: 0.0081 gb (0.58%)
other_ets: 0.0069 gb (0.49%)
connection_writers: 0.0068 gb (0.48%)
connection_readers: 0.0059 gb (0.42%)
metrics: 0.0022 gb (0.15%)
atom: 0.0012 gb (0.08%)
mnesia: 0.0008 gb (0.05%)
metadata_store: 0.0002 gb (0.01%)
quorum_ets: 0.0 gb (0.0%)
metadata_store_ets: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
quorum_queue_dlx_procs: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)

I have no name!@rabbitmq-cluster-1:/$ free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi       2.3Gi       850Mi        21Mi        27Gi        28Gi
Swap:             0B          0B          0B

Storage allocation - 77GiB of messages stored in disk, while the total queues size stored on this pod is 25GiB:

I have no name!@rabbitmq-cluster-1:/var/lib/rabbitmq/mnesia$ du -sh /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/*
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/cluster_nodes.config
52K     /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/coordination
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/DECISION_TAB.LOG
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/LATEST.LOG
77G     /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/msg_stores
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/nodes_running_at_shutdown
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/node-type.txt
28K     /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/quorum
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_durable_exchange.DCD
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_durable_exchange.DCL
44K     /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_durable_queue.DCD
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_durable_queue.DCL
32K     /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_durable_route.DCD
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_runtime_parameters.DCD
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_runtime_parameters.DCL
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_serial
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_topic_permission.DCD
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_user.DCD
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_user_permission.DCD
4.0K    /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/rabbit_vhost.DCD
32K     /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-cluster-1.rabbitmq-cluster-headless.cross.svc.cluster.local/schema.DAT

[michaelklishin]

So 92% of the memory footprint comes down to "reserved by the runtime", "allocated but unused" or belonging to the binary heap that is reference-counted and in practice, much lazier to be cleaned up. This is neither a leak nor a very common scenario to see.

See

rabbitmqctl help force_gc

It might help with the binary heap part since I cannot guess what specifically is referencing those messages.

[michaelklishin]

msg_stores is the directory where classic queue message stores reside. There are two per virtual host, and once transient queues are gone later in 4.x, there will be one per virtual host.

You have accumulated a backlog of messages in CQs and kept it there without consumers. If this is your typical workload, perhaps use a stream or a quorum queue, in the latter case tuning the segment entry count according to the message size specifically to avoid gigantic segment files that might only have one alive messages in them each.

[ronmegini]

We keep seeing this "storage leak" behavior every few days. For example here you can see the total storage of the queues is only around 40 GiB, but the actual disk usage exceeds 300 GiB.

A few key observations:

• This issue only occurs on one node in the cluster.
• Once this excessive storage consumption happens, purging or deleting queues has no effect - the disk space does not decrease.
• Storage usage keeps increasing even after hitting the storage watermark
• The only solution is deleting and recreating the pod, which clears the excess storage.
• This repeats every few days, we cant fine any relation

Is there any way to force RabbitMQ to reclaim disk space without restarting the pod?
Any ideas or leads would be greatly appreciated.

[mkuratczyk]

@ronmegini are you sure these logs cover the correct time period? The logs cover 22:33-22:52, which, even assuming a 2 hour time difference, corresponds to the time starting a few minutes after you drained the queues.

Assuming the logs are correct, it does seem indeed like it can be a case where the compaction happens but can't keep up. The logs show that all nodes are performing compaction.

It seems like there's a lot of compactions going on. That means that some messages cannot be deleted. When you consumed the messages, did you perhaps leave some queues untouched? That would explain why there are files in the message store that can be compacted but cannot be deleted (they contain at least 1 valid message). It's pretty much how we tested the compaction process while making changes to the message store in 4.0 - we would publish to many queues and consume from all but 1 queue, so that the message store would need to perform a lot of compactions.

Things that would help us diagnose the problem further:

• pause publishing to see if the compaction process starts catching up (and how quickly) - this would be disruptive to your system, so up to you whether you can try it
• try to identify what happens when the issue starts; there doesn't have to be a specific trigger so it may be hard to find it. It could be that a minor event leads to the compaction process taking a bit longer at some point and this destabilizies the system (in terms of queuing theory / Little's Law); but it could be that this can be correlated to the consumer timeout mentioned before, a spike in messages published or something else
• collecting flame graphs https://www.rabbitmq.com/blog/2022/05/31/flame-graphs

Potential workarounds:

• make sure that messages are consumed from all queues - given that many of the files are compacted and not deleted, it suggests that some of your queues keep messages longer than others - try to consume from all of them; this should allow the message store to just delete empty files
• there's a message store per vhost (technically two - one for persistent and one for transient messages); if you have some way to shard the flow of messages into a few vhosts, you would effectively parallelize the compaction process; in particular, if you indeed have some queues that are consumed more slowly - isolating them into a separate vhost should help
• you can try to increase the value of queue_index_embed_msgs_below. The default is 4096 (4kb) and means that messages above that threshold are stored in the message store, where they mix with messages from other queues; we have no experience of setting this to a value as high as 400kb for example, but we see no obvious reason why this wouldn't work. You can try this on 1 machine to see if it behaves better. With such a setting, each queue's messages would be stored independently (in a per-queue folder) and just deleted when they are consumed. Therefore, it should significantly reduce the message store workload and in particular - how many compactions the message store needs to perform.

[ronmegini]

@mkuratczyk, you’ve made some interesting points here. We will test stopping ingestion, and then we will know how to proceed with the workarounds.

[ronmegini]

We attempted to stop data ingestion to RabbitMQ, but couldn’t fully halt it for some reason. Since the disruption to production was taking too long, we had to stop the process.
In the meantime, I downgraded RabbitMQ from version 4.0.7 to 4.0.5, and the issue has been completely resolved for the past 5 days.

[mkuratczyk]

thanks for the update. Your original report was for 4.0.5 though. You did mention that the problem got worse after upgrading to 4.0.7. I guess it's better with 4.0.5 but you still expect the issue to occur at some point, right?

[ronmegini]

Yes. The original problem was with 4.0.5 and occurred once in few weeks. With 4.0.7 it constantly happened. For now no problems with 4.0.5.
Important to mention that our second production environment still running with 4.0.7 for two weeks already and no problems there.