Bug: mimir-distributed helm chart with ServerSideApply fails - invalid value null

[jkleinlercher]

What is the bug?

With the latest mimir-distributed helm chart update we needed to activate serversideapply in argocd, but the the installation fails with:

message: 'LogsInstance.monitoring.grafana.com "mimir-meta-monitoring" is
          invalid: spec.clients: Invalid value: "null": spec.clients in body must
          be of type array: "null"'

and

message: 'MetricsInstance.monitoring.grafana.com "mimir-meta-monitoring"
          is invalid: spec.remoteWrite[0].basicAuth: Invalid value: "null": spec.remoteWrite[0].basicAuth
          in body must be of type object: "null"'

my values are:

metaMonitoring:
  dashboards:
    enabled: true
  serviceMonitor:
    enabled: true
  grafanaAgent:
    enabled: true

According to argocd diff the resource which would be applied is like this

> apiVersion: monitoring.grafana.com/v1alpha1
> kind: LogsInstance
> metadata:
>   annotations:
>     argocd.argoproj.io/tracking-id: sx-mimir:monitoring.grafana.com/LogsInstance:mimir/sx-mimir-meta-monitoring
>   labels:
>     app.kubernetes.io/component: meta-monitoring
>     app.kubernetes.io/instance: sx-mimir
>     app.kubernetes.io/managed-by: Helm
>     app.kubernetes.io/name: mimir
>     app.kubernetes.io/version: 2.15.0
>     helm.sh/chart: mimir-5.6.0
>   name: sx-mimir-meta-monitoring
>   namespace: mimir
> spec:
>   clients: null
>   podLogsNamespaceSelector: {}
>   podLogsSelector:
>     matchLabels:
>       app.kubernetes.io/instance: sx-mimir
>       app.kubernetes.io/name: mimir

> apiVersion: monitoring.grafana.com/v1alpha1
> kind: MetricsInstance
> metadata:
>   annotations:
>     argocd.argoproj.io/tracking-id: sx-mimir:monitoring.grafana.com/MetricsInstance:mimir/sx-mimir-meta-monitoring
>   labels:
>     app.kubernetes.io/component: meta-monitoring
>     app.kubernetes.io/instance: sx-mimir
>     app.kubernetes.io/managed-by: Helm
>     app.kubernetes.io/name: mimir
>     app.kubernetes.io/version: 2.15.0
>     helm.sh/chart: mimir-5.6.0
>   name: sx-mimir-meta-monitoring
>   namespace: mimir
> spec:
>   remoteWrite:
>   - basicAuth: null
>     headers:
>       X-Scope-OrgID: metamonitoring
>     url: [http://sx-mimir-nginx.mimir.svc:80/api/v1/push](http://sx-mimir-nginx.mimir.svc/api/v1/push)
>   serviceMonitorNamespaceSelector: {}
>   serviceMonitorSelector:
>     matchLabels:
>       app.kubernetes.io/instance: sx-mimir
>       app.kubernetes.io/name: mimir

How to reproduce it?

Deploy latest helm chart with values above and serversideapply option in argocd

What did you think would happen?

Installation should just work

What was your environment?

Mimir 5.6.0 with argocd 2.14 and SSA enabled

Any additional context to share?

No response

[narqo]

message: 'LogsInstance.monitoring.grafana.com "mimir-meta-monitoring" is
         invalid: spec.clients: Invalid value: "null": spec.clients in body must
         be of type array: "null"'

The chart expects you to specify the metaMonitoring.grafanaAgent.logs.remote.url values, as shown in the docs.

---

As for this one,

'MetricsInstance.monitoring.grafana.com "mimir-meta-monitoring"
         is invalid: spec.remoteWrite[0].basicAuth: Invalid value: "null": spec.remoteWrite[0].basicAuth
         in body must be of type object: "null"'

Although it's the same underlying problem as above, it's surprising to see the error in this case: remoteWrite.[].basicAuth isn't required. It seems related to this Kubernetes issue kubernetes/kubernetes#117447

I think we can fix it by updating the mimir.metaMonitoring.metrics.remoteWrite helper (code) to not yield the basicAuth field if nothing in the .auth values is set.

[jkleinlercher]

thanks @narqo ! I added the URLs to LogsInstance and MetricsInstance with the following values:

  metaMonitoring:
    dashboards:
      enabled: true
    serviceMonitor:
      enabled: true
    grafanaAgent:
      enabled: true
      logs:
        remote:
          url: "http://test.com"
          auth:
            username: ""
            passwordSecretName: ""
            passwordSecretKey: ""
      metrics:
        remote:
          url: "http://test.com"
          auth:
            username: ""
            passwordSecretName: ""
            passwordSecretKey: ""

Now I get the following error:

'one or more objects failed to apply, reason: MetricsInstance.monitoring.grafana.com
      "sx-mimir-meta-monitoring" is invalid: spec.remoteWrite[0].basicAuth: Invalid
      value: "null": spec.remoteWrite[0].basicAuth in body must be of type object:
      "null",LogsInstance.monitoring.grafana.com "sx-mimir-meta-monitoring" is invalid:
      spec.clients[0].basicAuth: Invalid value: "null": spec.clients[0].basicAuth
      in body must be of type object: "null"'

I compared the LogsInstance and MetricsInstance between client-side-apply and server-side-apply. In LogsInstance and MetricsInstance you will see basicAuth: in client-side-apply and basicAuth: null in server-side-apply.

Client-Side-Apply:

apiVersion: monitoring.grafana.com/v1alpha1
kind: LogsInstance
metadata:
  name: sx-mimir-meta-monitoring
  namespace: mimir
  labels:
    helm.sh/chart: mimir-5.6.0
    app.kubernetes.io/name: mimir
    app.kubernetes.io/instance: sx-mimir
    app.kubernetes.io/component: meta-monitoring
    app.kubernetes.io/version: "2.15.0"
    app.kubernetes.io/managed-by: Helm
spec:
  clients:

      - url: http://test.com
        basicAuth:
        externalLabels:
          cluster: sx-mimir
  podLogsNamespaceSelector: {}

  podLogsSelector:
    matchLabels:
      app.kubernetes.io/name: mimir
      app.kubernetes.io/instance: sx-mimir
---
apiVersion: monitoring.grafana.com/v1alpha1
kind: LogsInstance
metadata:
  name: sx-mimir-meta-monitoring
  namespace: mimir
  labels:
    helm.sh/chart: mimir-5.6.0
    app.kubernetes.io/name: mimir
    app.kubernetes.io/instance: sx-mimir
    app.kubernetes.io/component: meta-monitoring
    app.kubernetes.io/version: "2.15.0"
    app.kubernetes.io/managed-by: Helm
spec:
  clients:

      - url: http://test.com
        basicAuth:
        externalLabels:
          cluster: "release-name"
  podLogsNamespaceSelector: {}

  podLogsSelector:
    matchLabels:
      app.kubernetes.io/name: mimir
      app.kubernetes.io/instance: sx-mimir

Server-Side-Apply:

apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
  labels:
    app.kubernetes.io/component: meta-monitoring
    app.kubernetes.io/instance: sx-mimir
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mimir
    app.kubernetes.io/version: 2.15.0
    helm.sh/chart: mimir-5.6.0
  name: sx-mimir-meta-monitoring
  namespace: mimir
spec:
  remoteWrite:
  - basicAuth: null
    url: http://test.com/
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      app.kubernetes.io/instance: sx-mimir
      app.kubernetes.io/name: mimir
---
apiVersion: monitoring.grafana.com/v1alpha1
kind: LogsInstance
metadata:
  labels:
    app.kubernetes.io/component: meta-monitoring
    app.kubernetes.io/instance: sx-mimir
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: mimir
    app.kubernetes.io/version: 2.15.0
    helm.sh/chart: mimir-5.6.0
  name: sx-mimir-meta-monitoring
  namespace: mimir
spec:
  clients:
  - basicAuth: null
    externalLabels:
      cluster: sx-mimir
    url: http://test.com/
  podLogsNamespaceSelector: {}
  podLogsSelector:
    matchLabels:
      app.kubernetes.io/instance: sx-mimir
      app.kubernetes.io/name: mimir