Description
What is the problem you are trying to solve?
Max lookback of a query is limited (among other) by compactor_blocks_retention_period for tenants included in a query. There is a risk of user assuming that the situation he is querying for did not occur in the past, while it might be that that part of data were not available for evaluation in case there are tenants with smaller compactor_blocks_retention_period than the maximum.
Consider for example the following query:
(
label_replace(
avg_over_time(scrape_duration_seconds[10m]) > 5,
"pod",
"$1",
"instance",
"(.*)"
)
and on (pod, namespace, cluster)
kube_pod_status_ready == 1
)
* on (pod, namespace, cluster) group_left (label_app)
kube_pod_labels{label_app=~"foo",namespace=~"bar-production"}
It's an alert, so typically it's evaluated over current data, but in case user would want to check if this happened in the past and use this as either a subquery or range query, it might get an empty result in case kube_ metrics have shorter retention to scrape_duration_seconds metrics -> false impression that it did not occur in the past.
In our setup scrape_duration_seconds belongs to a tenant who owns the scrape configuration, while kube-state-metrics metrics belong to k8s tenant, who have much smaller retency than the other tenants.
Note: we came across this while debugging #9727 after an upgrade
Which solution do you envision (roughly)?
Query result should include a warning that retention for tenants included in the query was not aligned. This could be then displayed in an UI. Similar to when there is a histogram in which monotonicity had to be fixed while evaluating query was evaluated.
Have you considered any alternatives?
No response
Any additional context to share?
No response
How long do you think this would take to be developed?
Not sure
What are the documentation dependencies?
No response
Proposer?
No response