-
Notifications
You must be signed in to change notification settings - Fork 106
Add minor update target-stage gate annotation #1913
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -596,4 +596,7 @@ const ( | |||||
|
|
||||||
| // OpenStackVersionMinorUpdateAvailableMessage | ||||||
| OpenStackVersionMinorUpdateAvailableMessage = "update available" | ||||||
|
|
||||||
| // OpenStackVersionMinorUpdateReadyGatedMessage - format string; arg is the target stage name | ||||||
| OpenStackVersionMinorUpdateReadyGatedMessage = "Minor update progression stopped after stage: %s . Set annotation to next stage after %s to resume OpenStack update" | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It doesn't have to be the next stage only, right? Can be any of the stages that follow? |
||||||
| ) | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -34,6 +34,28 @@ const ( | |
| MinorUpdateControlPlane string = "Minor Update Controlplane In Progress" | ||
| // MinorUpdateComplete - | ||
| MinorUpdateComplete string = "Complete" | ||
|
|
||
| // MinorUpdateTargetStageAnnotation - specifies the update stage after which the minor update | ||
| // should pause. All stages up to and including the named stage will be completed; subsequent | ||
| // stages will be blocked until the annotation is removed or updated to a later stage. | ||
| // Valid values: "ovn-controlplane", "ovn-dataplane", "rabbitmq", "mariadb", "memcached", | ||
| // "keystone", "controlplane". Remove the annotation to let the update proceed to completion. | ||
| MinorUpdateTargetStageAnnotation string = "core.openstack.org/minor-update-target-stage" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. wondering if we could shorten it to |
||
|
|
||
| // MinorUpdateStageOVNControlplane - stage name for OVN controlplane update | ||
| MinorUpdateStageOVNControlplane string = "ovn-controlplane" | ||
| // MinorUpdateStageOVNDataplane - stage name for OVN dataplane update | ||
| MinorUpdateStageOVNDataplane string = "ovn-dataplane" | ||
| // MinorUpdateStageRabbitMQ - stage name for RabbitMQ update | ||
| MinorUpdateStageRabbitMQ string = "rabbitmq" | ||
| // MinorUpdateStageMariaDB - stage name for MariaDB update | ||
| MinorUpdateStageMariaDB string = "mariadb" | ||
| // MinorUpdateStageMemcached - stage name for Memcached update | ||
| MinorUpdateStageMemcached string = "memcached" | ||
| // MinorUpdateStageKeystone - stage name for Keystone update | ||
| MinorUpdateStageKeystone string = "keystone" | ||
| // MinorUpdateStageControlplane - stage name for full controlplane update | ||
| MinorUpdateStageControlplane string = "controlplane" | ||
| ) | ||
|
|
||
| // OpenStackVersionSpec - defines the desired state of OpenStackVersion | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,272 @@ | ||
| # Performing a Staged Minor Update of OpenStack | ||
|
|
||
| A minor update of OpenStack environment in a fixed sequence of stages. By default the | ||
| update runs all stages automatically. The `core.openstack.org/minor-update-target-stage` | ||
| annotation lets you pause the update after any stage so you can validate the environment, | ||
| coordinate maintenance windows, or simply advance one stage at a time. | ||
|
|
||
| ## Examples to use staged rollouts | ||
|
|
||
| - You want to verify OVN networking is healthy before allowing the rest of the update to | ||
| proceed. | ||
| - Your organisation requires a sign-off after each major component is updated. | ||
| - You are performing the update in phases across a maintenance window and need to stop at a | ||
| known safe point. | ||
|
|
||
| ## Understanding the update pipeline | ||
|
|
||
| The update always runs stages in this order. Each stage must complete before the next one | ||
| starts. | ||
|
|
||
| | Stage | What gets updated | Requires manual action? | | ||
| |--------------------|-----------------------------------------|---------------------------------------------------------| | ||
| | `ovn-controlplane` | OVN control plane images | No | | ||
| | `ovn-dataplane` | OVN controller data plane images on compute nodes | **Yes** — create an OVN `OpenStackDataPlaneDeployment` | | ||
| | `rabbitmq` | RabbitMQ images | No | | ||
| | `mariadb` | MariaDB/Galera images | No | | ||
| | `memcached` | Memcached images | No | | ||
| | `keystone` | Keystone API images | No | | ||
| | `controlplane` | All remaining control-plane services | No | | ||
| | *(completion)* | Data-plane services on compute nodes | **Yes** — create a full `OpenStackDataPlaneDeployment` | | ||
|
|
||
| > **Note:** Two stages require you to create an `OpenStackDataPlaneDeployment` manually. | ||
| > The `ovn-dataplane` stage and the final data-plane completion step do not self-drive — | ||
| > the controller waits for the corresponding deployment to finish before advancing. | ||
| > See [Required manual deployments](#required-manual-deployments) below. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - A running cluster with a deployed OpenStack environment. | ||
| - `OpenStackControlPlane` and `OpenStackVersion` are both `Ready`. | ||
| - `status.deployedVersion` is set on the `OpenStackVersion` CR. | ||
| - A newer version is available: `status.availableVersion` differs from | ||
| `status.deployedVersion`. | ||
|
|
||
| The examples below use: | ||
| - Namespace: `openstack` | ||
| - `OpenStackVersion` CR name: `openstack` | ||
|
|
||
| --- | ||
|
|
||
| ## Performing a fully staged update | ||
|
|
||
| The recommended approach is to set the annotation to the first stage before bumping | ||
| `targetVersion`, then advance the annotation one stage at a time after you have validated | ||
| each step. | ||
|
|
||
| ### Step 1 — Confirm an update is available | ||
|
|
||
| ```bash | ||
| oc get openstackversion openstack -n openstack \ | ||
| -o jsonpath='Available: {.status.availableVersion} Deployed: {.status.deployedVersion}{"\n"}' | ||
| ``` | ||
|
|
||
| Note the `availableVersion` value — this is `<new-version>` in the commands below. | ||
|
|
||
| ### Step 2 — Set the initial pause point | ||
|
|
||
| Choose the stage after which you want the first pause. To pause after OVN control-plane: | ||
|
|
||
| ```bash | ||
| oc annotate openstackversion openstack \ | ||
| core.openstack.org/minor-update-target-stage=ovn-controlplane \ | ||
| -n openstack | ||
| ``` | ||
|
|
||
| ### Step 3 — Start the update | ||
|
|
||
| ```bash | ||
| oc patch openstackversion openstack -n openstack \ | ||
| --type=merge -p '{"spec":{"targetVersion":"<new-version>"}}' | ||
| ``` | ||
|
|
||
| The update begins immediately. The controller runs the `ovn-controlplane` stage and then | ||
| pauses. The `MinorUpdateOVNControlplane` condition becomes `True` and the | ||
| `MinorUpdateOVNDataplane` condition shows: | ||
|
|
||
| ``` | ||
| Minor update progression stopped after stage: ovn-controlplane . | ||
| Set annotation to next stage after ovn-controlplane to resume OpenStack update | ||
| ``` | ||
|
|
||
| ### Step 4 — Validate and advance stage by stage | ||
|
|
||
| After each pause, check the environment is healthy, then advance to the next stage. | ||
|
|
||
| #### Checking the current update status | ||
|
|
||
| ```bash | ||
| oc get openstackversion openstack -n openstack \ | ||
| -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}' \ | ||
| | grep MinorUpdate | ||
| ``` | ||
|
|
||
| Completed stages show `True`. The currently blocked stage shows `False` with a message | ||
| telling you which stage just finished and what to set next. | ||
|
|
||
| #### Advancing to the next stage | ||
|
|
||
| Update the annotation value to the stage you want to run next. For example, after | ||
| validating the OVN control-plane, advance to `ovn-dataplane`: | ||
|
|
||
| > **Before advancing to `ovn-dataplane`**, create the OVN dataplane deployment first — | ||
| > see [Required manual deployments](#required-manual-deployments). | ||
|
|
||
| ```bash | ||
| oc annotate openstackversion openstack \ | ||
| core.openstack.org/minor-update-target-stage=ovn-dataplane \ | ||
| --overwrite -n openstack | ||
| ``` | ||
|
|
||
| Continue advancing through the remaining stages as needed: | ||
|
|
||
| | To run through… | Set annotation to… | | ||
| |--------------------|--------------------| | ||
| | RabbitMQ | `rabbitmq` | | ||
| | MariaDB | `mariadb` | | ||
| | Memcached | `memcached` | | ||
| | Keystone | `keystone` | | ||
| | Full control-plane | `controlplane` | | ||
|
|
||
| ### Step 5 — Complete the update | ||
|
|
||
| When you are ready to run the final data-plane update on compute nodes, first create the | ||
| full dataplane deployment (see [Required manual deployments](#required-manual-deployments)), | ||
| then remove the annotation to let the update finish: | ||
|
|
||
| ```bash | ||
| oc annotate openstackversion openstack \ | ||
| core.openstack.org/minor-update-target-stage- \ | ||
| -n openstack | ||
| ``` | ||
|
|
||
| > The trailing `-` removes the annotation entirely. | ||
|
|
||
| The controller runs the remaining stages and, once complete, sets | ||
| `status.deployedVersion` to the new version. | ||
|
|
||
| ### Step 6 — Confirm completion | ||
|
|
||
| ```bash | ||
| oc get openstackversion openstack -n openstack \ | ||
| -o jsonpath='{.status.deployedVersion}' | ||
| ``` | ||
|
|
||
| The output should show `<new-version>`. | ||
|
|
||
| --- | ||
|
|
||
| ## Required manual deployments | ||
|
|
||
| Two stages in the process do not self-start. You must create an | ||
| `OpenStackDataPlaneDeployment` before (or at the same time as) advancing past each of them. | ||
|
|
||
| ### OVN data-plane deployment | ||
|
|
||
| Required before the `ovn-dataplane` stage can complete. This deployment updates only the | ||
| OVN-related services on compute nodes. | ||
|
|
||
| ```yaml | ||
| apiVersion: dataplane.openstack.org/v1beta1 | ||
| kind: OpenStackDataPlaneDeployment | ||
| metadata: | ||
| name: edpm-deployment-ovn-update | ||
| namespace: openstack | ||
| spec: | ||
| nodeSets: | ||
| - openstack-edpm-ipam | ||
| servicesOverride: | ||
| - ovn | ||
| ``` | ||
|
|
||
| ```bash | ||
| oc apply -f edpm-deployment-ovn-update.yaml | ||
| ``` | ||
|
|
||
| ### Full data-plane update deployment | ||
|
|
||
| Required before the final completion step can finish. This deployment updates all remaining | ||
| services on compute nodes. | ||
|
|
||
| ```yaml | ||
| apiVersion: dataplane.openstack.org/v1beta1 | ||
| kind: OpenStackDataPlaneDeployment | ||
| metadata: | ||
| name: edpm-deployment-update | ||
| namespace: openstack | ||
| spec: | ||
| nodeSets: | ||
| - openstack-edpm-ipam | ||
| servicesOverride: | ||
| - update | ||
| ``` | ||
|
|
||
| ```bash | ||
| oc apply -f edpm-deployment-update.yaml | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Pausing a running update | ||
|
|
||
| If you need to pause an update that is already in progress, add the annotation at any time. | ||
| The controller completes whichever stage is currently running, then stops after the stage you | ||
| named. | ||
|
|
||
| ```bash | ||
| oc annotate openstackversion openstack \ | ||
| core.openstack.org/minor-update-target-stage=<stage> \ | ||
| -n openstack | ||
| ``` | ||
|
|
||
| Replace `<stage>` with the name of the last stage you want to run before pausing. | ||
|
|
||
| --- | ||
|
|
||
| ## Running the full update without pausing | ||
|
|
||
| If you do not need staged control, omit the annotation entirely and let the controller run | ||
| all stages automatically. You still need to create both dataplane deployments at the right | ||
| time: | ||
|
|
||
| 1. Create the OVN dataplane deployment before or immediately after starting the update. | ||
| 2. Create the dataplane update deployment before the final completion step. | ||
|
|
||
| ```bash | ||
| oc patch openstackversion openstack -n openstack \ | ||
| --type=merge -p '{"spec":{"targetVersion":"<new-version>"}}' | ||
| ``` | ||
|
|
||
| --- | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### The update appears stuck | ||
|
|
||
| Check whether the blocked condition message contains `"stopped after stage"`. If it does, | ||
| the update is intentionally paused — advance or remove the annotation to continue. | ||
|
|
||
| ```bash | ||
| oc get openstackversion openstack -n openstack -o json | \ | ||
| jq '[.status.conditions[] | select(.reason=="Requested" and .status=="False")]' | ||
| ``` | ||
|
|
||
| ### `MinorUpdateOVNDataplane` or `MinorUpdateDataplane` stays `False` | ||
|
|
||
| These stages wait for an `OpenStackDataPlaneDeployment` to complete. Check whether the | ||
| required deployment exists and is running: | ||
|
|
||
| ```bash | ||
| oc get openstackdataplanedeployment -n openstack | ||
| ``` | ||
|
|
||
| If the deployment is missing, create it as described in | ||
| [Required manual deployments](#required-manual-deployments). | ||
|
|
||
| ### Checking overall update progress | ||
|
|
||
| ```bash | ||
| watch -n 5 "oc get openstackversion openstack -n openstack \ | ||
| -o jsonpath='{range .status.conditions[*]}{.type}{\"\t\"}{.status}{\"\t\"}{.message}{\"\n\"}{end}' \ | ||
| | grep MinorUpdate" | ||
| ``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you could also remove the annotation entirely, so I think the message should be something like
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what we also could do is to name the next possible stages, like if one did set it to rabbitmq, it would show