Skip to content

changes in initContainer's command doesn't trigger statefulset sync #2259

New issue
New issue
Open
Open
changes in initContainer's command doesn't trigger statefulset sync#2259
Labels
questionwontfix
@artesby

Description

@artesby
artesby
opened on Mar 10, 2023

Please, answer some short questions which should help us to understand your problem / question better?

  • Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.8.1
  • Where do you run it - cloud or metal? Kubernetes or OpenShift? aws k8s v1.23
  • Are you running Postgres Operator in production? yes
  • Type of issue? bug report

Hi,

We are using initContainer to tune some kernel settings for nodes running postgres. postgresql manifest looks like this:

apiVersion: acid.zalan.do/v1
kind: postgresql
...
spec:
  ...
  additionalVolumes:
  - mountPath: /host-sys
    name: host-sys
    volumeSource:
      hostPath:
        path: /sys
  initContainers:
  - command:
    - /bin/sh
    - -c
    - |-
      sysctl -w vm.dirty_background_bytes=1600000
      sysctl -w vm.dirty_bytes=3200000
      sysctl -w vm.swappiness=1
      sysctl -w kernel.sched_autogroup_enabled=0
      sysctl -w kernel.sched_migration_cost_ns=5000000
    image: busybox
    name: sysctl-container-revision-1
    securityContext:
      privileged: true
      runAsUser: 0
    volumeMounts:
    - mountPath: /host-sys
      name: host-sys

We noticed that changes in "command" section does not trigger update of statefulset. For example, change of first line to sysctl -w vm.dirty_background_bytes=1600001 produces following log in postgres-operator (and doesn't trigger "syncing statefulsets"):

time="2023-03-10T13:58:33Z" level=info msg="UPDATE event has been queued" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T13:58:33Z" level=info msg="update of the cluster started" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T13:58:33Z" level=debug msg="-    resourceVersion: 16999969," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="-    generation: 24," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+    resourceVersion: 17001340," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+    generation: 25," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="-          sysctl -w vm.dirty_background_bytes=1600000\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="+          sysctl -w vm.dirty_background_bytes=1600001\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="postgresql major version unchanged or smaller, no changes needed" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="initialize users" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="volume claims do not require changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:33Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=debug msg="syncing statefulsets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:34Z" level=debug msg="making GET http request: http://10.104.14.184:8008/config" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="making GET http request: http://10.104.15.187:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="making GET http request: http://10.104.14.184:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="syncing roles" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="closing database connection" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=debug msg="syncing connection pooler (master, replica) from (true, nil) to (true, nil)" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T13:58:46Z" level=info msg="cluster has been updated" cluster-name=island-published/island-postgres pkg=controller worker=0

To get your changes applied to sts you have to patch any other parameter of manifest, for example if I change initContainer's name to sysctl-container-revision-2 then i can see in operator's log that sts is being updated:

time="2023-03-10T14:01:27Z" level=info msg="UPDATE event has been queued" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T14:01:27Z" level=info msg="update of the cluster started" cluster-name=island-published/island-postgres pkg=controller worker=0
time="2023-03-10T14:01:27Z" level=debug msg="-    resourceVersion: 17001458," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="-    generation: 25," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+    resourceVersion: 17002746," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+    generation: 26," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="-        name: sysctl-container-revision-1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="+        name: sysctl-container-revision-2," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=info msg="postgresql major version unchanged or smaller, no changes needed" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="initialize users" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="team API is disabled" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=debug msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:27Z" level=info msg="syncing secrets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=debug msg="syncing volumes using \"pvc\" storage resize mode" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="volume claims do not require changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:28Z" level=debug msg="syncing statefulsets" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="Mount additional volumes: [{Name:host-sys MountPath:/host-sys SubPath: TargetContainers:[postgres] VolumeSource:{HostPath:&HostPathVolumeSource{Path:/sys,Type:nil,} EmptyDir:nil GCEPersistentDisk:nil AWSElasticBlockStore:nil GitRepo:nil Secret:nil NFS:nil ISCSI:nil Glusterfs:nil PersistentVolumeClaim:nil RBD:nil FlexVolume:nil Cinder:nil CephFS:nil Flocker:nil DownwardAPI:nil FC:nil AzureFile:nil ConfigMap:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil Projected:nil PortworxVolume:nil ScaleIO:nil StorageOS:nil CSI:nil Ephemeral:nil}}]" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="mark rolling update annotation for island-postgres-0: reason pod changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="mark rolling update annotation for island-postgres-1: reason pod changes" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="statefulset island-published/island-postgres is not in the desired state and needs to be updated" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            path: /sys," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            type: " cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+            path: /sys" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          name: sysctl-container-revision-1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+          name: sysctl-container-revision-2," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-            sysctl -w vm.dirty_background_bytes=1600000\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+            sysctl -w vm.dirty_background_bytes=1600001\\nsysctl -w vm.dirty_bytes=3200000\\nsysctl -w vm.swappiness=1\\nsysctl -w kernel.sched_autogroup_enabled=0\\nsysctl -w kernel.sched_migration_cost_ns=5000000" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          imagePullPolicy: IfNotPresent," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-              port: 8008," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-              scheme: HTTP" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+              port: 8008" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePath: /dev/termination-log," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-          terminationMessagePolicy: File," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      restartPolicy: Always," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      dnsPolicy: ClusterFirst," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      serviceAccount: default," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      schedulerName: default-scheduler," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      kind: PersistentVolumeClaim," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      apiVersion: v1," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      status: {" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-        phase: Pending" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-      }" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+      status: {}" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-  }," cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="-  revisionHistoryLimit: 10" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="+  }" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=info msg="reason: new statefulset initContainers's sysctl-container-revision-1 (index 0) name does not match the current one" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="updating statefulset" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:29Z" level=debug msg="making GET http request: http://10.104.14.184:8008/config" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="making GET http request: http://10.104.15.187:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="making GET http request: http://10.104.14.184:8008/patroni" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="performing rolling update" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=info msg="there are 2 pods in the cluster to recreate" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:01:41Z" level=debug msg="subscribing to pod \"island-published/island-postgres-1\"" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=info msg="pod \"island-published/island-postgres-1\" has been recreated" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="unsubscribing from pod \"island-published/island-postgres-1\" events" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="making GET http request: http://10.104.14.184:8008/cluster" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=error msg="could not sync statefulsets: could not recreate pods: skipping switchover: no switchover candidate found" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="syncing roles" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="closing database connection" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=debug msg="syncing connection pooler (master, replica) from (true, nil) to (true, nil)" cluster-name=island-published/island-postgres pkg=cluster
time="2023-03-10T14:02:06Z" level=info msg="cluster has been updated" cluster-name=island-published/island-postgres pkg=controller worker=0

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionwontfix

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions