AMQP: 0-9-1: OAuth2 Token Size Limitation Causes Connection Failure (frame_too_large)

[psavva]

Describe the bug

When using OAuth2 authentication with RabbitMQ, if the access token is too large, the connection fails with the following error:

RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable
 ---> RabbitMQ.Client.Exceptions.PossibleAuthenticationFailureException: Possibly caused by authentication failure
 ---> RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541, text='Unexpected Exception: Unable to read data from the transport connection: Connection reset by peer.', classId=0, methodId=0

and

2025-03-15 05:55:21.689185+00:00 [info] <0.2771.0> accepting AMQP connection <0.2771.0> (10.244.136.164:45024 -> 10.244.136.141:5672)
2025-03-15 05:55:24.745906+00:00 [error] <0.2771.0> closing AMQP connection <0.2771.0> (10.244.136.164:45024 -> 10.244.136.141:5672):
2025-03-15 05:55:24.745906+00:00 [error] <0.2771.0> {handshake_error,starting,0,
2025-03-15 05:55:24.745906+00:00 [error] <0.2771.0>                  {amqp_error,frame_error,
2025-03-15 05:55:24.745906+00:00 [error] <0.2771.0>                              "type 1, all octets = <<>>: {frame_too_large,6307,4088}",
2025-03-15 05:55:24.745906+00:00 [error] <0.2771.0>                              none}}

Reducing the token size resolves the issue, which suggests that RabbitMQ has a hard limit on OAuth2 token length that is causing an authentication failure.

Environment
RabbitMQ Version: 3.13.7
Erlang Version:
Deployment: Kubernetes (RabbitmqCluster CRD)
Authentication Method: OAuth2
Workaround
Using a smaller OAuth2 token allows authentication to succeed.

Suggested Fix
Either increase the frame_max limit for authentication packets or provide a configurable setting.
Improve error messaging to clearly indicate token size exceeded the limit instead of a generic frame_too_large error.

Reproduction steps

Create a token that is too largeConfigure RabbitMQ OAuth2 authentication.
Use an OAuth2 provider that issues large tokens (e.g., with many scopes, claims, or large signing keys).
Attempt to connect with a client using the oversized token.
Observe the frame_too_large error and the connection failure.

Expected behavior

RabbitMQ should either:
Accept larger OAuth2 tokens (or provide a configurable limit).
Fail with a clearer error message that explicitly states the OAuth token exceeded the supported size.

Additional context

No response

[michaelklishin]

@psavva RabbitMQ 3.13 is out of community support.

There is no such thing as an "AMQP 0-9-1 token" and "a token that is too large" is not a reproduction step. We will not guess how long is "too long".

What is really going on in your case: AMQP 0-9-1 has a limit on the length of the password field (likely of 255 bytes but it's been a long time since I've looked at that part of the codebase), so
indeed a frame with an exceeding length will be rejected as too large. It's been the case for over 18 years now.

Such validation belongs to client libraries. You are welcome to contribute it to the .NET client, and more specifically to its ConnectionFactory, that would not require any protocol parser changes.

[michaelklishin]

@psavva well, there are only two possible options:

1. See if the field can be expanded from its current limit of 255 bytes in a backwards-compatible way
2. Add a documentation note and validations to client libraries

We have over a decade of evidence that many users do not read any documentation until they run into issues. So adding a note to the OAuth 2 Troubleshooting guide seems inevitable.

Adding validation to client libraries is an easy solution easily accessible to users like you.

Whether option 1 is feasible, the core team (or you) would have to investigate. Changing the generated protocol parser is an option but only if it can be done in a backwards-compatible way.

[psavva]

I think the easiest way around this, given, was probably the first to see this issue (and maybe the only ones that will ever see it), and it's in a very specific case, we could just update the troubleshooting oAuth2 docs.

Perhaps this is the best course of action.

[michaelklishin]

@psavva I'll update the docs in a bit rabbitmq/rabbitmq-website#2212.

Option 1 will be investigated in #13541, likely after RabbitMQ 4.1.0 ships late next month.

[psavva]

Much appreciated

[michaelklishin]

Done rabbitmq/rabbitmq-website@4520c95

[michaelklishin]

Instead of calling up your colleagues @constantinos-yiasemi @chriscosta @icanci01 to an upvote fest, you'd be better off explaining what tooling produces those large JWT tokens and in what circumstances.

A very rare to see issue (in the few years that the OAuth 2 support it's been available) suddenly attracts a few upvotes in just nine hours. Seems legit 🤣

We support and test against five different IDPs and there are users who have adopted OAuth 2 at a very large scale with multiple IDPs, and yet somehow they do not run into this password field length limitation.

[chriscosta]

hello dear friiiend.
To further add to what @psavva already described is that the Token belonged to a RabbitMQ producer which had to be subscribed to multiple exchanges and queues that needed multiple service account roles defined.
We never thought that too many would be a problem and spent hours trying to figure out what the issue was.
But the issue was caused by this, removing old/unused service account roles from the client fixed the issue.

Aaand, why not upvote something I was a part of?

[michaelklishin]

@chriscosta do you know how long were the original token and the token that the node has accepted, in bytes?

[chriscosta]

the faulty token size is described in the issue above (6307 bytes)

{amqp_error,frame_error,
"type 1, all octets = <<>>: {frame_too_large,6307,4088}",
none}}

the later (fixed one) 3671 bytes

[chriscosta]

we do have a workaround, ofcourse, we can create such service account roles that can fit to a broader selection of permissions for RabbitMQ
e.g.: rabbitmq.configure:[vhost]/[exchange]/* (each role for us is around 150 bytes btw depending ofcourse on the naming of the exchange/queues)
but we just wanted to point out that this IS an issue that when encountered is really hard to find.

[michaelklishin]

@chriscosta fair enough, this scenario is now documented.

[michaelklishin]

@psavva @chriscosta the response field length is not limited, see my findings in #13541.

Your clients or rabbitmq.conf likely override the connection's allowed frame_max, that's the best hypothesis I have. The default AMQP 0-9-1 frame_max is 128 kB, and client libraries usually use the same default (set over a decade ago).

Which might explain why there were only two reports of JWT tokens running over the limit in a few years of OAuth 2 support.

See rabbitmq-diagnostics environment | grep frame_max and your .NET client's ConnectionFactory settings. Depending on the protocol plugins enabled on the node, you might see multiple values reported, one per protocol (AMQP 0-9-1 and the RabbitMQ Stream Protocol specifically).

[michaelklishin]

It's a value negotiated with the client: https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/bdbfa3b6333bbcc3d1247f9de46264c9826fe417/projects/RabbitMQ.Client/Impl/Connection.Commands.cs#L178.

The .NET client seems to use a uint for default frame size.

I do not see any other limits on the password/secret/SASL response in the AMQP 0-9-1 parser, and it is specifically the parser that throws that exception.

[chriscosta]

looking at the code you posted and going back to the initial bug details posted by @psavva , the exception:

---> RabbitMQ.Client.Exceptions.PossibleAuthenticationFailureException: Possibly caused by authentication failure

is thrown before setiting max frame size (line 171), which validates my hypothesis that there is a default size somewhere before the error is thrown.

[michaelklishin]

decode_method_fields('connection.start_ok', <<F0Len:32/unsigned, F0Tab:F0Len/binary, F1Len:8/unsigned, F1:F1Len/binary, F2Len:32/unsigned, F2:F2Len/binary, F3Len:8/unsigned, F3:F3Len/binary>>) ->
  F0 = rabbit_binary_parser:parse_table(F0Tab),
  rabbit_binary_parser:assert_utf8(F1),
  rabbit_binary_parser:assert_utf8(F3),
  #'connection.start_ok'{client_properties = F0, mechanism = F1, response = F2, locale = F3};

The F2 value length is parsed as a 32-bit unsigned integer, so up to 2^32-1 bytes will be accepted.

So the initial pre-authenticated state value turned out to be

InitialFrameMax = application:get_env(rabbit, initial_frame_max, ?FRAME_MIN_SIZE),

where FRAME_MIN_SIZE is set to

-define(FRAME_MIN_SIZE, 4096).

[michaelklishin]

#13542

[michaelklishin]

Plus #13549, apparently the constant value was produced by the parser code generator.

[michaelklishin]

A higher pre-authentication frame max default will ship in 4.1.0: #13542, #13549, 18533d4.

It can in theory affect some clients with exotic configurations, so it won't be backported to 4.0.x, at least that's the decision for now.

Starting with 4.0.0, the initial frame size limit can be configured, including via rabbitmq.conf (even though the setting is fairly advanced).

This is now documented in the Troubleshooting OAuth 2 guide.

amqplib Users Will Have to Upgrade Their Client Library

JavaScript's amqplib users would have to upgrade to a new release when it comes out or that client won't be able to connect to a RabbitMQ 4.1.0 node due to a very low initial frame_max (set at least 10 years ago and unnoticed prior to this change).