Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bitnami/rabbitmq] RabbitMQ: explain how to safely avoid a deployment deadlock #25931

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

michaelklishin
Copy link
Contributor

Explain the problem and a widely used solution
instead of recommending force booting nodes.

These practices are generally used by existing
RabbitMQ K8S Cluster Operators but this PR
intentionally does not recommend any of them.

References #25698, #16081, #25916.

Explain the problem and a widely used solution
instead of recommending force booting nodes.

These practices are generally used by existing
RabbitMQ K8S Cluster Operators but this PR
intentionally does not recommend any of them.

Signed-off-by: Michael Klishin <michael@clojurewerkz.org>
Signed-off-by: Michael Klishin <klishinm@vmware.com>
Signed-off-by: Michael Klishin <mikhail.klishin@broadcom.com>

Commit a suggested edit to bitnami/rabbitmq/README.md

Co-authored-by: Carlos Rodríguez Hernández <carlosrh@vmware.com>
Signed-off-by: Michael Klishin <michaelklishin@icloud.com>
@github-actions github-actions bot added rabbitmq triage Triage is needed labels May 15, 2024
@github-actions github-actions bot requested a review from javsalgar May 15, 2024 15:24
@javsalgar javsalgar changed the title RabbitMQ: explain how to safely avoid a deployment deadlock [bitnami/rabbitmq] RabbitMQ: explain how to safely avoid a deployment deadlock May 16, 2024
@javsalgar javsalgar requested review from carrodher and removed request for javsalgar May 16, 2024 07:42
@javsalgar javsalgar assigned carrodher and unassigned javsalgar May 16, 2024
@github-actions github-actions bot removed the triage Triage is needed label May 17, 2024
@github-actions github-actions bot removed the request for review from carrodher May 17, 2024 06:36
@github-actions github-actions bot requested a review from dgomezleon May 17, 2024 06:36
@carrodher carrodher removed the request for review from dgomezleon May 17, 2024 06:37
@michaelklishin
Copy link
Contributor Author

@rafariossaa @carrodher @javsalgar is there anything else I can do to help push this forward? Not having this aspect documented is a big deal for RabbitMQ users.

@michaelklishin
Copy link
Contributor Author

Bump. This is still as relevant and important to document as before.

@carrodher
Copy link
Member

Thanks @michaelklishin and sorry for the delay. Our team will review and provide feedback.
Your contribution is greatly appreciated!


This happens if the pod management policy of the statefulset is not `Parallel` and the last pod to be running wasn't the first pod of the statefulset. If that happens, update the pod management policy to recover a healthy state:
The following combination of deployment settings avoids the problem:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi,
Is there any kind of process that user could perform to once he is in this situation ?.
I mean, I am not sure if the user has already deployed without the parameters you indicate here he could avoid the issue by just upgrading the deployment with the parameter or if he may need to execute other steps.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rafariossaa change to the recommended deployment parameters and change the readiness probe. The absolute minimum required would be using rabbitmq-diagnostics ping for readiness probe but both recommendations can be applied at the same time in a single deployment update.

The goal of this PR is to recommended the safe setup from the start.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants