parallelvirtualcluster
/
pvc-ansible
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
pvc-ansible/oneshot
History
Joshua Boniface 7fe682aa60 Handle freshness for all 3 types separately 
If microcode was missing, checking the other two would be UNKN and thus
not restart. But, if microcode *is* present, we want to restart for
either of the other two as well.

So separate into 3 distinct checks and restart if any one is changed.
 		2023-10-24 10:41:47 -04:00
..
README.md Remove pvc-flush references 2023-09-01 15:42:28 -04:00
reboot-pvc-cluster.yml Move to new maintenance mode and check legacy first 2023-09-01 15:42:29 -04:00
roles Add Debian 10 -> Debian 11 upgrade playbook 2023-09-01 15:42:27 -04:00
update-pvc-cluster.yml Handle freshness for all 3 types separately 2023-10-24 10:41:47 -04:00
update-pvc-daemons.yml Ensure PVC daemons are updated before reboots 2023-09-01 15:42:30 -04:00
upgrade-pvc-cluster_deb11.yml Move to new maintenance mode and check legacy first 2023-09-01 15:42:29 -04:00
upgrade-pvc-cluster_deb12.yml Properly disable autoscaling PGs 2023-09-01 15:42:30 -04:00

README.md

PVC Oneshot Playbooks

This directory contains playbooks to assist in automating day-to-day maintenance of a PVC cluster. These playbooks can be used independent of the main pvc.yml and roles setup to automate tasks.

update-pvc-cluster.yml and reboot-pvc-cluster.yml

This playbook performs a sequential full upgrade on all nodes in a PVC cluster.

The reboot-pvc-cluster.yml does the same shutdown and restart steps as update-pvc-cluster.yml, but forced for all hosts, and without the update part.

Running the Playbook

$ ansible-playbook -i hosts -l [cluster] update-pvc-cluster.yml

Caveats, Warnings and Notes

  • Ensure the cluster is in Optimal health before executing this playbook; all nodes should be up and reachable and operating normally

  • Be prepared to intervene if step 9 times out; OOB access may be required

  • This playbook is safe to run against a given host multiple times (e.g. to rerun after a failure); if a reboot is not required, it will not be performed

Process and Steps

For each host in the cluster sequentially, do:

  1. Enable cluster maintenance mode

  2. Perform a full apt update, upgrade, autoremove, and clean

  3. Clean up obsolete kernels (kernel-cleanup.sh), packages/updated configuration files (dpkg-cleanup.sh), and the apt archive

  4. Verify library freshness and kernel version; if these produce no warnings, go to step 14 (skip reboot)

  5. Secondary the node, then wait 30 seconds

  6. Flush the node, wait for all VMs to migrate, then wait 15 seconds

  7. Stop the pvcnoded and zookeeper daemons, then wait 15 seconds

  8. Set Ceph OSD noout and stop all Ceph OSD, monitor, and manager processes, then wait 30 seconds

  9. Reboot the system and wait for it to come back up (maximum wait time of 1800 seconds)

  10. Ensure all OSDs become active and all PGs recover, then unset Ceph OSD noout

  11. Unflush the node, wait for all VMs to migrate, then wait 30 seconds

  12. Reset any systemd failures

  13. Disable cluster maintenance mode, then wait 30 seconds

upgrade-pvc-daemons.yml

This playbook performs a sequential upgrade of the PVC software daemons via apt on all nodes in a PVC cluster. This is a less invasive update process than the update-pvc-cluster.yml playbook as it does not flush or reboot the nodes, but does restart all PVC daemons (pvcnoded, pvcapid, and pvcapid-worker).

Running the Playbook

$ ansible-playbook -i hosts -l [cluster] upgrade-pvc-daemons.yml

Caveats, Warnings, and Notes

  • Ensure the cluster is in Optimal health before executing this playbook; all nodes should be up and reachable and operating normally

  • This playbook is safe to run against a given host multiple times; if service restarts are not required, they will not be performed

  • This playbook should only be used in exceptional circumstances when performing a full update-pvc-cluster.yml would be too disruptive; it is always preferable to update all packages and reboot the nodes instead

Process and Steps

For each node in the cluster sequentially, do:

  1. Enable cluster maintenance mode

  2. Perform an apt update, and install the 4 PVC packages (pvc-client-cli, pvc-daemon-common, pvc-daemon-api, pvc-daemon-node)

  3. Clean up the apt archive

  4. Verify if packages changed; if not, go to step 8 (skip restarts)

  5. Secondary the node, then wait 30 seconds

  6. Restart both active PVC daemons (pvcapid-worker, pvcnoded), then wait 60 seconds; since the node is not the primary coordinator, pvcapid will not be running

  7. Verify daemons are running

  8. Disble cluster maintenance mode, then wait 30 seconds