Open
Description
What is the bug?
When trying to query data with a large time range, one can face the GRPC error in the mimir-quirier.
"rpc error: code = ResourceExhausted desc = grpc: received message larger than max (133997632 vs. 104857600)"
Increasing grpc_server_max_recv_msg_size and grpc_server_max_send_msg_size does nothing because it is hardcoded in the
https://github.com/grafana/mimir/blob/main/pkg/querier/store_gateway_client.go#L80
How to reproduce it?
It fails only on a limited set of queries, and it is not easy to reproduce without having the same metrics data stored in storage.
What did you think would happen?
The data size reached hardcoded limits.
What was your environment?
Kubernetes, mimir-distributed helm chart, GCS bucket.