Bump version to 0.9.82

For whatever reason, a Celery worker on <5.2.x was not picking these up.
Move them back to the root of the module so they are properly picked up
on these older versions but still prevents calling the routing functions
during an API doc generation.

.gitignore

@@ -1,6 +1,8 @@
 *.pyc
 *.tmp
 *.swp
+# Ignore swagger output (for pvc-docs instead)
+swagger.json
 # Ignore build artifacts
 debian/pvc-*/
 debian/*.log

.version

@@ -1 +1 @@
-0.9.80
+0.9.82

CHANGELOG.md

@@ -1,5 +1,25 @@
 ## PVC Changelog
 
+###### [v0.9.82](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.82)
+
+  * [API Daemon] Fixes a bug where the Celery result_backend was not loading properly on Celery <5.2.x (Debian 10/11).
+
+###### [v0.9.81](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.81)
+
+  **Breaking Changes:** This large release features a number of major changes. While these should all be a seamless transition, the behaviour of several commands and the backend system for handling them has changed significantly, along with new dependencies from PVC Ansible. A full cluster configuration update via `pvc.yml` is recommended after installing this version. Redis is replaced with KeyDB on coordinator nodes as a Celery backend; this transition will be handled gracefully by the `pvc-ansible` playbooks, though note that KeyDB will be exposed on the Upstream interface. The Celery worker system is renamed `pvcworkerd`, is now active on all nodes (coordinator and non-coordinator), and is expanded to encompass several commands that previously used a similar, custom setup within the node daemons, including "pvc vm flush-locks" and all "pvc storage osd" tasks. The previously-mentioned CLI commands now all feature "--wait"/"--no-wait" flags, with wait showing a progress bar and status output of the task run. The "pvc cluster task" command can now used for viewing all task types, replacing the previously-custom/specific "pvc provisioner status" command. All example provisioner scripts have been updated to leverage new helper functions in the Celery system; while updating these is optional, an administrator is recommended to do so for optimal log output behaviour.
+
+  * [CLI Client] Fixes "--live" argument handling and duplicate restart prompts.
+  * [All] Adds support for multiple OSDs on individual disks (NVMe workloads).
+  * [All] Corrects and updates OSD replace, refresh, remove, and add functionality; replace no longer purges.
+  * [All] Switches to KeyDB (multi-master) instead of Redis and adds node monitoring plugin.
+  * [All] Replaces Zookeeper/Node Daemon-based message passing and task handling with pvcworkerd Celery workers on all nodes; increases worker concurrency to 3 (per node).
+  * [All] Moves all task-like functions to Celery and updates existing Celery tasks to use new helpers and ID system.
+  * [CLI Client] Adds "--wait/--no-wait" options with progress bars to all Celery-based tasks, "--wait" default; adds a standardized task interface under "pvc cluster task".
+  * [Node Daemon] Cleans up the fencing handler and related functions.
+  * [Node Daemon] Fixes bugs with VM memory reporting during keepalives.
+  * [Node Daemon] Fixes a potential race condition during primary/secondary transition by backgrounding systemctl commands.
+  * [API Daemon] Updates example provisioner plugins to use new Celery functions.
+
 ###### [v0.9.80](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.80)
 
   * [CLI] Improves CLI performance by not loading "pkg_resources" until needed

api-daemon/pvcapid/Daemon.py

@@ -27,7 +27,7 @@ from ssl import SSLContext, TLSVersion
 from distutils.util import strtobool as dustrtobool
 
 # Daemon version
-version = "0.9.80"
+version = "0.9.82"
 
 # API version
 API_VERSION = 1.0

api-daemon/pvcapid/flaskapi.py

@@ -141,11 +141,15 @@ celery = Celery(
     result_backend=celery_task_uri,
     result_extended=True,
 )
+app.config["broker_url"] = celery_task_uri
+app.config["result_backend"] = celery_task_uri
+celery.conf.update(app.config)
 
 
 def celery_startup():
-    app.config["CELERY_broker_url"] = celery_task_uri
-    app.config["result_backend"] = celery_task_uri
+    """
+    Runs when the API daemon starts, but not the Celery workers or the API doc generator
+    """
     app.config["task_queues"] = tuple(
         [Queue(h, routing_key=f"{h}.#") for h in get_all_nodes()]
     )

client-cli/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup
 
 setup(
     name="pvc",
-    version="0.9.80",
+    version="0.9.82",
     packages=["pvc.cli", "pvc.lib"],
     install_requires=[
         "Click",

debian/changelog

@@ -1,3 +1,27 @@
+pvc (0.9.82-0) unstable; urgency=high
+
+  * [API Daemon] Fixes a bug where the Celery result_backend was not loading properly on Celery <5.2.x (Debian 10/11).
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sat, 25 Nov 2023 15:38:50 -0500
+
+pvc (0.9.81-0) unstable; urgency=high
+
+  **Breaking Changes:** This large release features a number of major changes. While these should all be a seamless transition, the behaviour of several commands and the backend system for handling them has changed significantly, along with new dependencies from PVC Ansible. A full cluster configuration update via `pvc.yml` is recommended after installing this version. Redis is replaced with KeyDB on coordinator nodes as a Celery backend; this transition will be handled gracefully by the `pvc-ansible` playbooks, though note that KeyDB will be exposed on the Upstream interface. The Celery worker system is renamed `pvcworkerd`, is now active on all nodes (coordinator and non-coordinator), and is expanded to encompass several commands that previously used a similar, custom setup within the node daemons, including "pvc vm flush-locks" and all "pvc storage osd" tasks. The previously-mentioned CLI commands now all feature "--wait"/"--no-wait" flags, with wait showing a progress bar and status output of the task run. The "pvc cluster task" command can now used for viewing all task types, replacing the previously-custom/specific "pvc provisioner status" command. All example provisioner scripts have been updated to leverage new helper functions in the Celery system; while updating these is optional, an administrator is recommended to do so for optimal log output behaviour.
+
+  * [CLI Client] Fixes "--live" argument handling and duplicate restart prompts.
+  * [All] Adds support for multiple OSDs on individual disks (NVMe workloads).
+  * [All] Corrects and updates OSD replace, refresh, remove, and add functionality; replace no longer purges.
+  * [All] Switches to KeyDB (multi-master) instead of Redis and adds node monitoring plugin.
+  * [All] Replaces Zookeeper/Node Daemon-based message passing and task handling with pvcworkerd Celery workers on all nodes; increases worker concurrency to 3 (per node).
+  * [All] Moves all task-like functions to Celery and updates existing Celery tasks to use new helpers and ID system.
+  * [CLI Client] Adds "--wait/--no-wait" options with progress bars to all Celery-based tasks, "--wait" default; adds a standardized task interface under "pvc cluster task".
+  * [Node Daemon] Cleans up the fencing handler and related functions.
+  * [Node Daemon] Fixes bugs with VM memory reporting during keepalives.
+  * [Node Daemon] Fixes a potential race condition during primary/secondary transition by backgrounding systemctl commands.
+  * [API Daemon] Updates example provisioner plugins to use new Celery functions.
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Fri, 17 Nov 2023 01:29:41 -0500
+
 pvc (0.9.80-0) unstable; urgency=high
 
   * [CLI] Improves CLI performance by not loading "pkg_resources" until needed

gen-api-migrations

@@ -7,7 +7,7 @@ VERSION="$( head -1 debian/changelog | awk -F'[()-]' '{ print $2 }' )"
 pushd $( git rev-parse --show-toplevel ) &>/dev/null
 pushd api-daemon &>/dev/null
 export PVC_CONFIG_FILE="./pvcapid.sample.yaml"
-./pvcapid-manage.py db migrate -m "PVC version ${VERSION}"
-./pvcapid-manage.py db upgrade
+./pvcapid-manage_flask.py db migrate -m "PVC version ${VERSION}"
+./pvcapid-manage_flask.py db upgrade
 popd &>/dev/null
 popd &>/dev/null

node-daemon/pvcnoded/Daemon.py

@@ -49,7 +49,7 @@ import re
 import json
 
 # Daemon version
-version = "0.9.80"
+version = "0.9.82"
 
 
 ##########################################################

test-cluster.sh

@@ -78,9 +78,9 @@ _pvc vm tag get testx
 _pvc vm list --tag mytag
 _pvc vm tag remove testx mytag
 _pvc vm network get testx
-_pvc vm vcpu set --no-restart testx 4
+_pvc vm vcpu set --no-restart testx 1
 _pvc vm vcpu get testx
-_pvc vm memory set --no-restart testx 4096
+_pvc vm memory set --no-restart testx 1024
 _pvc vm memory get testx
 _pvc vm vcpu set --no-restart testx 2
 _pvc vm memory set testx 2048 --restart --yes