Make operator report no longer need resources

[sdudoladov]

We have sometimes a situation when a pod/statefulset/service/etc survives a PG cluster to which it belongs. This situation may (a) indicate a bug, (b) interfere with operator from functioning correctly (for example the operator does not move pods of unknown clusters from unschedulable nodes), and (c) occupy resources.

To ease monitoring and debugging such cases, we may create a REST endpoint(s) that will return a list of resources that match operator label selection yet does not belong to any of the Postgres clusters known by the operator. It should also be straightforward to piggyback fetching relevant info on the Sync operation.

Please provide feedback and suggestions.

[alexeyklyukin]

The ListResources collect K8S objects, but only for a given cluster. Doing this globally could be rather expensive for a REST API call, especially if the cluster has multiple namespaces. Couldn't it be solved by a script running from the client-side that fetches the list of PG objects and all available labels for K8S resources and compares both, finding those labels that have no PG resources attached.

The other idea is to have a global rest API endpoint specifically for errors; perhaps we could also retry the whole create/update/delete operation on the resource if it has failed with some error code, instead of trying to retry just the Kubernetes call (that would make the retry code less complex).