Skip to content

Bug: Hardcoded GRPC limits in store gateway client prevents data from quering #10295

Open
@moskitone

Description

@moskitone

What is the bug?

When trying to query data with a large time range, one can face the GRPC error in the mimir-quirier.
"rpc error: code = ResourceExhausted desc = grpc: received message larger than max (133997632 vs. 104857600)"

Increasing grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size does nothing because it is hardcoded in the
https://github.com/grafana/mimir/blob/main/pkg/querier/store_gateway_client.go#L80

How to reproduce it?

It fails only on a limited set of queries, and it is not easy to reproduce without having the same metrics data stored in storage.

What did you think would happen?

The data size reached hardcoded limits.

What was your environment?

Kubernetes, mimir-distributed helm chart, GCS bucket.

Any additional context to share?

Image

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions