Repository navigation
Closed
Labels
Description
Orphaned pods/endpoints are left in the cluster if the ADD event is followed by DELETE within a short time period for the same cluster.
This error manifests as
time="2019-04-29T12:17:52Z" level=info msg="\"ADD\" event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
...
time="2019-04-29T12:17:53Z" level=info msg="waiting for the cluster being ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2019-04-29T12:17:56Z" level=debug msg="Waiting for 2 pods to become ready" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2019-04-29T12:18:00Z" level=info msg="\"DELETE\" event has been queued" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
...
time="2019-04-29T12:18:29Z" level=info msg="statefulset \"default/acid-minimal-cluster\" has been deleted" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2019-04-29T12:18:29Z" level=debug msg="deleting pods" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2019-04-29T12:18:29Z" level=debug msg="no pods to delete" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
...
time="2019-04-29T12:18:29Z" level=debug msg="deleting PVCs" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2019-04-29T12:18:29Z" level=debug msg="no PVCs to delete" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
...
time="2019-04-29T12:18:32Z" level=debug msg="removing leftover Patroni objects (endpoints or configmaps)" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2019-04-29T12:18:32Z" level=warning msg="could not remove leftover patroni objects; could not fetch Patroni Endpoint \"/\": an empty namespace may not be set when a resource name is provided" cluster-name=default/acid-minimal-cluster pkg=cluster worker=0
time="2019-04-29T12:18:32Z" level=info msg="cluster has been deleted" cluster-name=default/acid-minimal-cluster pkg=controller worker=0
reproducible both with kind and actual k8s
this issue also prevents creating a new cluster with the same name afterwards
Reacted by Weilu Jia, Seyed Amirali Sobhgol and Zhikun Lao
Activity
FxKu commentedon May 23, 2019
Aside from finalizers #450, ownerReference #498 might also help.
Jan-M commentedon May 23, 2019
Both options need to be investigated with care, the only real delete we care about is the delete of the "postgresql" object. Other objects can be deleted (e.g. the statefulset) and no interruption or impact is expected.
davisford commentedon Aug 2, 2019
@Jan-M this is what I'm seeing. I have Terraform scripts that build and tear down the whole cluster, but the secondary read replica we spawn never gets removed even though the operator and the
postgresqlobject is removed.The
foo-cluster-1is left behind.... it is a PG db node that was part of the original cluster.Other resources are also left behind, secrets, services, endpoints, persistent volumes + claims.
What is the best way to handle this?
FYI, the logs for that orphaned pg cluster pod, just spewing this error over and over from Patroni:
It looks like the secret it had been using was deleted for the service account which causes it to just go into a continuous failed state:
EDIT -- I understand why the pv/pvc aren't deleted. That's undesirable for a
StatefulSet. Also noted reading the k8s docs onStatefulSet:This may be why the
-1cluster pod node is orphaned? Might it be possible for the operator to thus scale the set down to 0 prior to delete?davisford commentedon Aug 3, 2019
@Jan-M looking at the code, it appears that it is just attempting to delete the
StatefulSetas opposed to the recommended approach of scaling it down to zero prior to deletion.Jan-M commentedon Aug 5, 2019
But it is followed up by deletePods if I see this correctly.
But you are right, we will look into the right place of the docs on how to delete a statefulset. I did not know that the delete maybe does not delete pod.
Maybe got mixed up here, where it is mentioned that kubectl scales down too.
https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/
@sdudoladov