Allow similar for IPMI hostnames

This is printed in the startup logo block anyways.

.gitignore

@@ -1,6 +1,8 @@
 *.pyc
 *.tmp
 *.swp
+# Ignore swagger output (for pvc-docs instead)
+swagger.json
 # Ignore build artifacts
 debian/pvc-*/
 debian/*.log

.version

@@ -1 +1 @@
-0.9.81
+0.9.82

CHANGELOG.md

@@ -1,5 +1,9 @@
 ## PVC Changelog
 
+###### [v0.9.82](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.82)
+
+  * [API Daemon] Fixes a bug where the Celery result_backend was not loading properly on Celery <5.2.x (Debian 10/11).
+
 ###### [v0.9.81](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.81)
 
   **Breaking Changes:** This large release features a number of major changes. While these should all be a seamless transition, the behaviour of several commands and the backend system for handling them has changed significantly, along with new dependencies from PVC Ansible. A full cluster configuration update via `pvc.yml` is recommended after installing this version. Redis is replaced with KeyDB on coordinator nodes as a Celery backend; this transition will be handled gracefully by the `pvc-ansible` playbooks, though note that KeyDB will be exposed on the Upstream interface. The Celery worker system is renamed `pvcworkerd`, is now active on all nodes (coordinator and non-coordinator), and is expanded to encompass several commands that previously used a similar, custom setup within the node daemons, including "pvc vm flush-locks" and all "pvc storage osd" tasks. The previously-mentioned CLI commands now all feature "--wait"/"--no-wait" flags, with wait showing a progress bar and status output of the task run. The "pvc cluster task" command can now used for viewing all task types, replacing the previously-custom/specific "pvc provisioner status" command. All example provisioner scripts have been updated to leverage new helper functions in the Celery system; while updating these is optional, an administrator is recommended to do so for optimal log output behaviour.

api-daemon/pvcapid.sample.yaml

@@ -1,80 +0,0 @@
----
-# pvcapid configuration file example
-#
-# This configuration file specifies details for the PVC API daemon running on
-# this machine. Default values are not supported; the values in this sample
-# configuration are considered defaults and can be used as-is.
-#
-# Copy this example to /etc/pvc/pvcapid.conf and edit to your needs
-
-pvc:
-    # debug: Enable/disable API debug mode
-    debug: True
-    # coordinators: The list of cluster coordinator hostnames
-    coordinators:
-      - pvchv1
-      - pvchv2
-      - pvchv3
-    # api: Configuration of the API listener
-    api:
-        # listen_address: IP address(es) to listen on; use 0.0.0.0 for all interfaces
-        listen_address: "127.0.0.1"
-        # listen_port: TCP port to listen on, usually 7370
-        listen_port: "7370"
-        # authentication: Authentication and security settings
-        authentication:
-            # enabled: Enable or disable authentication (True/False)
-            enabled: False
-            # secret_key: Per-cluster secret key for API cookies; generate with uuidgen or pwgen
-            secret_key: ""
-            # tokens: a list of authentication tokens; leave as an empty list to disable authentication
-            tokens:
-                  # description: token description for management
-                - description: "testing"
-                  # token: random token for authentication; generate with uuidgen or pwgen
-                  token: ""
-        # ssl: SSL configuration
-        ssl:
-            # enabled: Enabled or disable SSL operation (True/False)
-            enabled: False
-            # cert_file: SSL certificate file
-            cert_file: ""
-            # key_file: SSL certificate key file
-            key_file: ""
-    # provisioner: Configuration of the Provisioner API listener
-    provisioner:
-        # database: Backend database configuration
-        database:
-            # host: PostgreSQL hostname, usually 'localhost'
-            host: localhost
-            # port: PostgreSQL port, invariably '5432'
-            port: 5432
-            # name: PostgreSQL database name, invariably 'pvcapi'
-            name: pvcapi
-            # user: PostgreSQL username, invariable 'pvcapi'
-            user: pvcapi
-            # pass: PostgreSQL user password, randomly generated
-            pass: pvcapi
-        # queue: Celery backend queue using the PVC Zookeeper cluster
-        queue:
-            # host: Redis hostname, usually 'localhost'
-            host: localhost
-            # port: Redis port, invariably '6279'
-            port: 6379
-            # path: Redis queue path, invariably '/0'
-            path: /0
-        # ceph_cluster: Information about the Ceph storage cluster
-        ceph_cluster:
-            # storage_hosts: The list of hosts that the Ceph monitors are valid on; if empty (the default),
-            #                uses the list of coordinators
-            storage_hosts:
-              - pvchv1
-              - pvchv2
-              - pvchv3
-            # storage_domain: The storage domain name, concatenated with the coordinators list names
-            #                 to form monitor access strings
-            storage_domain: "pvc.storage"
-            # ceph_monitor_port: The port that the Ceph monitor on each coordinator listens on
-            ceph_monitor_port: 6789
-            # ceph_storage_secret_uuid: Libvirt secret UUID for Ceph storage access
-            ceph_storage_secret_uuid: ""

api-daemon/pvcapid.service

@@ -8,7 +8,7 @@ After = network-online.target
 Type = simple
 WorkingDirectory = /usr/share/pvc
 Environment = PYTHONUNBUFFERED=true
-Environment = PVC_CONFIG_FILE=/etc/pvc/pvcapid.yaml
+Environment = PVC_CONFIG_FILE=/etc/pvc/pvc.conf
 ExecStart = /usr/share/pvc/pvcapid.py
 Restart = on-failure
 

api-daemon/pvcapid/Daemon.py

@@ -27,7 +27,7 @@ from ssl import SSLContext, TLSVersion
 from distutils.util import strtobool as dustrtobool
 
 # Daemon version
-version = "0.9.81"
+version = "0.9.82"
 
 # API version
 API_VERSION = 1.0
@@ -54,67 +54,158 @@ def strtobool(stringv):
 ##########################################################
 
 # Parse the configuration file
+config_file = None
 try:
-    pvcapid_config_file = os.environ["PVC_CONFIG_FILE"]
+    _config_file = "/etc/pvc/pvcapid.yaml"
+    if not os.path.exists(_config_file):
+        raise
+    config_file = _config_file
+    config_type = "legacy"
 except Exception:
+    pass
+try:
+    _config_file = os.environ["PVC_CONFIG_FILE"]
+    if not os.path.exists(_config_file):
+        raise
+    config_file = _config_file
+    config_type = "current"
+except Exception:
+    pass
+
+if not config_file:
     print(
         'Error: The "PVC_CONFIG_FILE" environment variable must be set before starting pvcapid.'
     )
     exit(1)
 
-print('Loading configuration from file "{}"'.format(pvcapid_config_file))
 
-# Read in the config
+def load_configuration_file(config_file):
-try:
+    print('Loading configuration from file "{}"'.format(config_file))
-    with open(pvcapid_config_file, "r") as cfgfile:
-        o_config = yaml.load(cfgfile, Loader=yaml.BaseLoader)
-except Exception as e:
-    print("ERROR: Failed to parse configuration file: {}".format(e))
-    exit(1)
 
-try:
+    # Read in the config
-    # Create the config object
+    try:
-    config = {
+        with open(config_file, "r") as cfgfile:
-        "debug": strtobool(o_config["pvc"]["debug"]),
+            o_config = yaml.load(cfgfile, Loader=yaml.BaseLoader)
-        "coordinators": o_config["pvc"]["coordinators"],
+    except Exception as e:
-        "listen_address": o_config["pvc"]["api"]["listen_address"],
+        print("ERROR: Failed to parse configuration file: {}".format(e))
-        "listen_port": int(o_config["pvc"]["api"]["listen_port"]),
+        exit(1)
-        "auth_enabled": strtobool(o_config["pvc"]["api"]["authentication"]["enabled"]),
-        "auth_secret_key": o_config["pvc"]["api"]["authentication"]["secret_key"],
-        "auth_tokens": o_config["pvc"]["api"]["authentication"]["tokens"],
-        "ssl_enabled": strtobool(o_config["pvc"]["api"]["ssl"]["enabled"]),
-        "ssl_key_file": o_config["pvc"]["api"]["ssl"]["key_file"],
-        "ssl_cert_file": o_config["pvc"]["api"]["ssl"]["cert_file"],
-        "database_host": o_config["pvc"]["provisioner"]["database"]["host"],
-        "database_port": int(o_config["pvc"]["provisioner"]["database"]["port"]),
-        "database_name": o_config["pvc"]["provisioner"]["database"]["name"],
-        "database_user": o_config["pvc"]["provisioner"]["database"]["user"],
-        "database_password": o_config["pvc"]["provisioner"]["database"]["pass"],
-        "queue_host": o_config["pvc"]["provisioner"]["queue"]["host"],
-        "queue_port": o_config["pvc"]["provisioner"]["queue"]["port"],
-        "queue_path": o_config["pvc"]["provisioner"]["queue"]["path"],
-        "storage_hosts": o_config["pvc"]["provisioner"]["ceph_cluster"][
-            "storage_hosts"
-        ],
-        "storage_domain": o_config["pvc"]["provisioner"]["ceph_cluster"][
-            "storage_domain"
-        ],
-        "ceph_monitor_port": o_config["pvc"]["provisioner"]["ceph_cluster"][
-            "ceph_monitor_port"
-        ],
-        "ceph_storage_secret_uuid": o_config["pvc"]["provisioner"]["ceph_cluster"][
-            "ceph_storage_secret_uuid"
-        ],
-    }
 
-    # Use coordinators as storage hosts if not explicitly specified
+    return o_config
-    if not config["storage_hosts"]:
-        config["storage_hosts"] = config["coordinators"]
 
-except Exception as e:
-    print("ERROR: Failed to load configuration: {}".format(e))
-    exit(1)
 
+def get_configuration_current(config_file):
+    o_config = load_configuration_file(config_file)
+    try:
+        # Create the config object
+        config = {
+            "debug": strtobool(o_config["logging"].get("debug_logging", "False")),
+            "all_nodes": o_config["cluster"]["all_nodes"],
+            "coordinators": o_config["cluster"]["coordinator_nodes"],
+            "listen_address": o_config["api"]["listen"]["address"],
+            "listen_port": int(o_config["api"]["listen"]["port"]),
+            "auth_enabled": strtobool(
+                o_config["api"]["authentication"].get("enabled", "False")
+            ),
+            "auth_secret_key": o_config["api"]["authentication"]["secret_key"],
+            "auth_source": o_config["api"]["authentication"]["source"],
+            "ssl_enabled": strtobool(o_config["api"]["ssl"].get("enabled", "False")),
+            "ssl_cert_file": o_config["api"]["ssl"]["certificate"],
+            "ssl_key_file": o_config["api"]["ssl"]["private_key"],
+            "database_port": o_config["database"]["postgres"]["port"],
+            "database_host": o_config["database"]["postgres"]["hostname"],
+            "database_name": o_config["database"]["postgres"]["credentials"]["api"][
+                "database"
+            ],
+            "database_user": o_config["database"]["postgres"]["credentials"]["api"][
+                "username"
+            ],
+            "database_password": o_config["database"]["postgres"]["credentials"]["api"][
+                "password"
+            ],
+            "queue_port": o_config["database"]["keydb"]["port"],
+            "queue_host": o_config["database"]["keydb"]["hostname"],
+            "queue_path": o_config["database"]["keydb"]["path"],
+            "storage_domain": o_config["cluster"]["networks"]["storage"]["domain"],
+            "storage_hosts": o_config["ceph"].get("monitor_hosts", None),
+            "ceph_monitor_port": o_config["ceph"]["monitor_port"],
+            "ceph_storage_secret_uuid": o_config["ceph"]["secret_uuid"],
+        }
+
+        # Use coordinators as storage hosts if not explicitly specified
+        if not config["storage_hosts"] or len(config["storage_hosts"]) < 1:
+            config["storage_hosts"] = config["coordinators"]
+
+        # Set up our token list if specified
+        if config["auth_source"] == "token":
+            config["auth_tokens"] = o_config["api"]["token"]
+        else:
+            if config["auth_enabled"]:
+                print(
+                    "WARNING: No authentication method provided; disabling authentication."
+                )
+                config["auth_enabled"] = False
+
+    except Exception as e:
+        print(f"ERROR: Failed to load configuration: {e}")
+        exit(1)
+
+    return config
+
+
+def get_configuration_legacy(config_file):
+    o_config = load_configuration_file(config_file)
+    try:
+        # Create the config object
+        config = {
+            "debug": strtobool(o_config["pvc"]["debug"]),
+            "coordinators": o_config["pvc"]["coordinators"],
+            "listen_address": o_config["pvc"]["api"]["listen_address"],
+            "listen_port": int(o_config["pvc"]["api"]["listen_port"]),
+            "auth_enabled": strtobool(
+                o_config["pvc"]["api"]["authentication"]["enabled"]
+            ),
+            "auth_secret_key": o_config["pvc"]["api"]["authentication"]["secret_key"],
+            "auth_tokens": o_config["pvc"]["api"]["authentication"]["tokens"],
+            "ssl_enabled": strtobool(o_config["pvc"]["api"]["ssl"]["enabled"]),
+            "ssl_key_file": o_config["pvc"]["api"]["ssl"]["key_file"],
+            "ssl_cert_file": o_config["pvc"]["api"]["ssl"]["cert_file"],
+            "database_host": o_config["pvc"]["provisioner"]["database"]["host"],
+            "database_port": int(o_config["pvc"]["provisioner"]["database"]["port"]),
+            "database_name": o_config["pvc"]["provisioner"]["database"]["name"],
+            "database_user": o_config["pvc"]["provisioner"]["database"]["user"],
+            "database_password": o_config["pvc"]["provisioner"]["database"]["pass"],
+            "queue_host": o_config["pvc"]["provisioner"]["queue"]["host"],
+            "queue_port": o_config["pvc"]["provisioner"]["queue"]["port"],
+            "queue_path": o_config["pvc"]["provisioner"]["queue"]["path"],
+            "storage_hosts": o_config["pvc"]["provisioner"]["ceph_cluster"][
+                "storage_hosts"
+            ],
+            "storage_domain": o_config["pvc"]["provisioner"]["ceph_cluster"][
+                "storage_domain"
+            ],
+            "ceph_monitor_port": o_config["pvc"]["provisioner"]["ceph_cluster"][
+                "ceph_monitor_port"
+            ],
+            "ceph_storage_secret_uuid": o_config["pvc"]["provisioner"]["ceph_cluster"][
+                "ceph_storage_secret_uuid"
+            ],
+        }
+
+        # Use coordinators as storage hosts if not explicitly specified
+        if not config["storage_hosts"]:
+            config["storage_hosts"] = config["coordinators"]
+
+    except Exception as e:
+        print("ERROR: Failed to load configuration: {}".format(e))
+        exit(1)
+
+    return config
+
+
+if config_type == "legacy":
+    config = get_configuration_legacy(config_file)
+else:
+    config = get_configuration_current(config_file)
 
 ##########################################################
 # Entrypoint
@@ -134,25 +225,25 @@ def entrypoint():
 
     # Print our startup messages
     print("")
-    print("|----------------------------------------------------------|")
+    print("|------------------------------------------------------------|")
-    print("|                                                          |")
+    print("|                                                            |")
-    print("|           ███████████ ▜█▙      ▟█▛ █████ █ █ █           |")
+    print("|            ███████████ ▜█▙      ▟█▛ █████ █ █ █            |")
-    print("|                    ██  ▜█▙    ▟█▛  ██                    |")
+    print("|                     ██  ▜█▙    ▟█▛  ██                     |")
-    print("|           ███████████   ▜█▙  ▟█▛   ██                    |")
+    print("|            ███████████   ▜█▙  ▟█▛   ██                     |")
-    print("|           ██             ▜█▙▟█▛    ███████████           |")
+    print("|            ██             ▜█▙▟█▛    ███████████            |")
-    print("|                                                          |")
+    print("|                                                            |")
-    print("|----------------------------------------------------------|")
+    print("|------------------------------------------------------------|")
-    print("| Parallel Virtual Cluster API daemon v{0: <19} |".format(version))
+    print("| Parallel Virtual Cluster API daemon v{0: <21} |".format(version))
-    print("| Debug: {0: <49} |".format(str(config["debug"])))
+    print("| Debug: {0: <51} |".format(str(config["debug"])))
-    print("| API version: v{0: <42} |".format(API_VERSION))
+    print("| API version: v{0: <44} |".format(API_VERSION))
     print(
-        "| Listen: {0: <48} |".format(
+        "| Listen: {0: <50} |".format(
             "{}:{}".format(config["listen_address"], config["listen_port"])
         )
     )
-    print("| SSL: {0: <51} |".format(str(config["ssl_enabled"])))
+    print("| SSL: {0: <53} |".format(str(config["ssl_enabled"])))
-    print("| Authentication: {0: <40} |".format(str(config["auth_enabled"])))
+    print("| Authentication: {0: <42} |".format(str(config["auth_enabled"])))
-    print("|----------------------------------------------------------|")
+    print("|------------------------------------------------------------|")
     print("")
 
     pvc_api.celery_startup()

api-daemon/pvcapid/flaskapi.py

@@ -141,11 +141,15 @@ celery = Celery(
     result_backend=celery_task_uri,
     result_extended=True,
 )
+app.config["broker_url"] = celery_task_uri
+app.config["result_backend"] = celery_task_uri
+celery.conf.update(app.config)
 
 
 def celery_startup():
-    app.config["CELERY_broker_url"] = celery_task_uri
+    """
-    app.config["result_backend"] = celery_task_uri
+    Runs when the API daemon starts, but not the Celery workers or the API doc generator
+    """
     app.config["task_queues"] = tuple(
         [Queue(h, routing_key=f"{h}.#") for h in get_all_nodes()]
     )

api-daemon/pvcworkerd.service

@@ -8,7 +8,7 @@ After = network-online.target
 Type = simple
 WorkingDirectory = /usr/share/pvc
 Environment = PYTHONUNBUFFERED=true
-Environment = PVC_CONFIG_FILE=/etc/pvc/pvcapid.yaml
+Environment = PVC_CONFIG_FILE=/etc/pvc/pvc.conf
 ExecStart = /usr/share/pvc/pvcworkerd.sh
 Restart = on-failure
 

client-cli/autobackup.sample.yaml

@@ -1,52 +0,0 @@
----
-# Root level configuration key
-autobackup:
-
-  # Backup root path on the node, used as the remote mountpoint
-  # Must be an absolute path beginning with '/'
-  # If remote_mount is enabled, the remote mount will be mounted on this directory
-  # If remote_mount is enabled, it is recommended to use a path under `/tmp` for this
-  # If remote_mount is disabled, a real filesystem must be mounted here (PVC system volumes are small!)
-  backup_root_path: "/tmp/backups"
-
-  # Suffix to the backup root path, used to allow multiple PVC systems to write to a single root path
-  # Must begin with '/'; leave empty to use the backup root path directly
-  # Note that most remote mount options can fake this if needed, but provided to ensure local compatability
-  backup_root_suffix: "/mycluster"
-
-  # VM tag(s) to back up
-  # Only VMs with at least one of the given tag(s) will be backed up; all others will be skipped
-  backup_tags:
-    - "backup"
-    - "mytag"
-
-  # Backup schedule: when and what format to take backups
-  backup_schedule:
-    full_interval:  7  # Number of total backups between full backups; others are incremental
-                       # > If this number is 1, every backup will be a full backup and no incremental
-                       #   backups will be taken
-                       # > If this number is 2, every second backup will be a full backup, etc.
-    full_retention: 2  # Keep this many full backups; the oldest will be deleted when a new one is
-                       # taken, along with all child incremental backups of that backup
-                       # > Should usually be at least 2 when using incrementals (full_interval > 1) to
-                       #   avoid there being too few backups after cleanup from a new full backup
-
-  # Automatic mount settings
-  # These settings permit running an arbitrary set of commands, ideally a "mount" command or similar, to
-  # ensure that a remote filesystem is mounted on the backup root path
-  # While the examples here show absolute paths, that is not required; they will run with the $PATH of the
-  # executing environment (either the "pvc" command on a CLI or a cron/systemd timer)
-  # A "{backup_root_path}" f-string/str.format type variable MAY be present in any cmds string to represent
-  # the above configured root backup path, which is interpolated at runtime
-  # If multiple commands are given, they will be executed in the order given; if no commands are given,
-  # nothing is executed, but the keys MUST be present
-  auto_mount:
-    enabled: no  # Enable automatic mount/unmount support
-    # These commands are executed at the start of the backup run and should mount a filesystem
-    mount_cmds:
-      # This example shows an NFS mount leveraging the backup_root_path variable
-      - "/usr/sbin/mount.nfs -o nfsvers=3 10.0.0.10:/backups {backup_root_path}"
-    # These commands are executed at the end of the backup run and should unmount a filesystem
-    unmount_cmds:
-      # This example shows a generic umount leveraging the backup_root_path variable
-      - "/usr/bin/umount {backup_root_path}"

client-cli/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup
 
 setup(
     name="pvc",
-    version="0.9.81",
+    version="0.9.82",
     packages=["pvc.cli", "pvc.lib"],
     install_requires=[
         "Click",

debian/changelog

@@ -1,3 +1,9 @@
+pvc (0.9.82-0) unstable; urgency=high
+
+  * [API Daemon] Fixes a bug where the Celery result_backend was not loading properly on Celery <5.2.x (Debian 10/11).
+
+ -- Joshua M. Boniface <joshua@boniface.me>  Sat, 25 Nov 2023 15:38:50 -0500
+
 pvc (0.9.81-0) unstable; urgency=high
 
   **Breaking Changes:** This large release features a number of major changes. While these should all be a seamless transition, the behaviour of several commands and the backend system for handling them has changed significantly, along with new dependencies from PVC Ansible. A full cluster configuration update via `pvc.yml` is recommended after installing this version. Redis is replaced with KeyDB on coordinator nodes as a Celery backend; this transition will be handled gracefully by the `pvc-ansible` playbooks, though note that KeyDB will be exposed on the Upstream interface. The Celery worker system is renamed `pvcworkerd`, is now active on all nodes (coordinator and non-coordinator), and is expanded to encompass several commands that previously used a similar, custom setup within the node daemons, including "pvc vm flush-locks" and all "pvc storage osd" tasks. The previously-mentioned CLI commands now all feature "--wait"/"--no-wait" flags, with wait showing a progress bar and status output of the task run. The "pvc cluster task" command can now used for viewing all task types, replacing the previously-custom/specific "pvc provisioner status" command. All example provisioner scripts have been updated to leverage new helper functions in the Celery system; while updating these is optional, an administrator is recommended to do so for optimal log output behaviour.

debian/pvc-client-cli.install

@@ -1 +0,0 @@
-client-cli/autobackup.sample.yaml usr/share/pvc

debian/pvc-daemon-api.install

@@ -1,7 +1,6 @@
 api-daemon/pvcapid.py usr/share/pvc
 api-daemon/pvcapid-manage*.py usr/share/pvc
 api-daemon/pvc-api-db-upgrade usr/share/pvc
-api-daemon/pvcapid.sample.yaml usr/share/pvc
 api-daemon/pvcapid usr/share/pvc
 api-daemon/pvcapid.service lib/systemd/system
 api-daemon/pvcworkerd.service lib/systemd/system

debian/pvc-daemon-api.postinst

@@ -15,9 +15,6 @@ if systemctl is-active --quiet pvcworkerd.service; then
     systemctl start pvcworkerd.service
 fi
 
-if [ ! -f /etc/pvc/pvcapid.yaml ]; then
+if [ ! -f /etc/pvc/pvc.conf ]; then
-    echo "NOTE: The PVC client API daemon (pvcapid.service) and the PVC Worker daemon (pvcworkerd.service) have not been started; create a config file at /etc/pvc/pvcapid.yaml, then run the database configuration (/usr/share/pvc/pvc-api-db-upgrade) and start them manually."
+    echo "NOTE: The PVC client API daemon (pvcapid.service) and the PVC Worker daemon (pvcworkerd.service) have not been started; create a config file at /etc/pvc/pvc.conf, then run the database configuration (/usr/share/pvc/pvc-api-db-upgrade) and start them manually."
 fi
-
-# Clean up any old sample configs
-rm /etc/pvc/pvcapid.sample.yaml || true

debian/pvc-daemon-node.install

@@ -1,5 +1,4 @@
 node-daemon/pvcnoded.py usr/share/pvc
-node-daemon/pvcnoded.sample.yaml usr/share/pvc
 node-daemon/pvcnoded usr/share/pvc
 node-daemon/pvcnoded.service lib/systemd/system
 node-daemon/pvc.target lib/systemd/system

debian/pvc-daemon-node.postinst

@@ -12,8 +12,5 @@ systemctl enable /lib/systemd/system/pvc.target
 if systemctl is-active --quiet pvcnoded.service; then
     echo "NOTE: The PVC node daemon (pvcnoded.service) has not been restarted; this is up to the administrator."
 else
-    echo "NOTE: The PVC node daemon (pvcnoded.service) has not been started; create a config file at /etc/pvc/pvcnoded.yaml then start it."
+    echo "NOTE: The PVC node daemon (pvcnoded.service) has not been started; create a config file at /etc/pvc/pvc.conf then start it."
 fi
-
-# Clean up any old sample configs
-rm /etc/pvc/pvcnoded.sample.yaml || true

node-daemon/pvcnoded.sample.yaml

@@ -1,216 +0,0 @@
----
-# pvcnoded configuration file example
-#
-# This configuration file specifies details for this node in PVC. Multiple node
-# blocks can be added but only the one matching the current system nodename will
-# be used by the local daemon. Default values are not supported; the values in
-# this sample configuration are considered defaults and, with adjustment of the
-# nodename section and coordinators list, can be used as-is on a Debian system.
-#
-# Copy this example to /etc/pvc/pvcnoded.conf and edit to your needs
-
-pvc:
-  # node: The (short) hostname of the node, set during provisioning
-  node: pvchv1
-  # debug: Enable or disable debug output
-  debug: False
-  # functions: The daemon functions to enable
-  functions:
-    # enable_hypervisor: Enable or disable hypervisor functionality
-    # This should never be False except in very advanced usecases
-    enable_hypervisor: True
-    # enable_networking: Enable or disable virtual networking and routing functionality
-    enable_networking: True
-    # enable_storage: Enable or disable Ceph storage management functionality
-    enable_storage: True
-    # enable_api: Enable or disable the API client, if installed, when node is Primary
-    enable_api: True
-  # cluster: Cluster-level configuration
-  cluster:
-    # coordinators: The list of cluster coordinator hostnames
-    coordinators:
-      - pvchv1
-      - pvchv2
-      - pvchv3
-    # networks: Cluster-level network configuration
-    # OPTIONAL if enable_networking: False
-    networks:
-      # upstream: Upstream routed network for in- and out-bound upstream networking
-      upstream:
-        # domain: Upstream domain name, may be None
-        domain: "mydomain.net"
-        # network: Upstream network block
-        network: "1.1.1.0/24"
-        # floating_ip: Upstream floating IP address for the primary coordinator
-        floating_ip: "1.1.1.10/24"
-        # gateway: Upstream static default gateway, if applicable
-        gateway: "1.1.1.1"
-      # cluster: Cluster internal network for node communication and client virtual networks
-      cluster:
-        # domain: Cluster internal domain name
-        domain: "pvc.local"
-        # network: Cluster internal network block
-        network: "10.255.0.0/24"
-        # floating_ip: Cluster internal floating IP address for the primary coordinator
-        floating_ip: "10.255.0.254/24"
-      # storage: Cluster internal network for storage traffic
-      storage:
-        # domain: Cluster storage domain name
-        domain: "pvc.storage"
-        # network: Cluster storage network block
-        network: "10.254.0.0/24"
-        # floating_ip: Cluster storage floating IP address for the primary coordinator
-        floating_ip: "10.254.0.254/24"
-  # coordinator: Coordinator-specific configuration
-  # OPTIONAL if enable_networking: False
-  coordinator:
-    # dns: DNS aggregator subsystem
-    dns:
-      # database: Patroni PostgreSQL database configuration
-      database:
-        # host: PostgreSQL hostname, invariably 'localhost'
-        host: localhost
-        # port: PostgreSQL port, invariably 'localhost'
-        port: 5432
-        # name: PostgreSQL database name, invariably 'pvcdns'
-        name: pvcdns
-        # user: PostgreSQL username, invariable 'pvcdns'
-        user: pvcdns
-        # pass: PostgreSQL user password, randomly generated
-        pass: pvcdns
-    # metadata: Metadata API subsystem
-    metadata:
-      # database: Patroni PostgreSQL database configuration
-      database:
-        # host: PostgreSQL hostname, invariably 'localhost'
-        host: localhost
-        # port: PostgreSQL port, invariably 'localhost'
-        port: 5432
-        # name: PostgreSQL database name, invariably 'pvcapi'
-        name: pvcapi
-        # user: PostgreSQL username, invariable 'pvcapi'
-        user: pvcapi
-        # pass: PostgreSQL user password, randomly generated
-        pass: pvcapi
-  # system: Local PVC instance configuration
-  system:
-    # intervals: Intervals for keepalives and fencing
-    intervals:
-      # vm_shutdown_timeout: Number of seconds for a VM to 'shutdown' before being forced off
-      vm_shutdown_timeout: 180
-      # keepalive_interval: Number of seconds between keepalive/status updates
-      keepalive_interval: 5
-      # monitoring_interval: Number of seconds between monitoring check updates
-      monitoring_interval: 60
-      # fence_intervals: Number of keepalive_intervals to declare a node dead and fence it
-      fence_intervals: 6
-      # suicide_intervals: Numer of keepalive_intervals before a node considers itself dead and self-fences, 0 to disable 
-      suicide_intervals: 0
-    # fencing: Node fencing configuration
-    fencing:
-      # actions: Actions to take after a fence trigger
-      actions:
-        # successful_fence: Action to take after successfully fencing a node, options: migrate, None
-        successful_fence: migrate
-        # failed_fence: Action to take after failing to fence a node, options: migrate, None
-        failed_fence: None
-      # ipmi: Local system IPMI options
-      ipmi:
-        # host: Hostname/IP of the local system's IPMI interface, must be reachable
-        host: pvchv1-lom
-        # user: Local system IPMI username
-        user: admin
-        # pass: Local system IPMI password
-        pass: Passw0rd
-    # migration: Migration option configuration
-    migration:
-      # target_selector: Criteria to select the ideal migration target, options: mem, memprov, load, vcpus, vms
-      target_selector: mem
-    # configuration: Local system configurations
-    configuration:
-      # directories: PVC system directories
-      directories:
-        # plugin_directory: Directory containing node monitoring plugins
-        plugin_directory: "/usr/share/pvc/plugins"
-        # dynamic_directory: Temporary in-memory directory for active configurations
-        dynamic_directory: "/run/pvc"
-        # log_directory: Logging directory
-        log_directory: "/var/log/pvc"
-        # console_log_directory: Libvirt console logging directory
-        console_log_directory: "/var/log/libvirt"
-      # logging: PVC logging configuration
-      logging:
-        # file_logging: Enable or disable logging to files under log_directory
-        file_logging: True
-        # stdout_logging: Enable or disable logging to stdout (i.e. journald)
-        stdout_logging: True
-        # zookeeper_logging: Enable ot disable logging to Zookeeper (for `pvc node log` functionality)
-        zookeeper_logging: True
-        # log_colours: Enable or disable ANSI colours in log output
-        log_colours: True
-        # log_dates: Enable or disable date strings in log output
-        log_dates: True
-        # log_keepalives: Enable or disable keepalive logging
-        log_keepalives: True
-        # log_keepalive_cluster_details: Enable or disable node status logging during keepalive
-        log_keepalive_cluster_details: True
-        # log_keepalive_plugin_details: Enable or disable node health plugin logging during keepalive
-        log_keepalive_plugin_details: True
-        # console_log_lines: Number of console log lines to store in Zookeeper per VM
-        console_log_lines: 1000
-        # node_log_lines: Number of node log lines to store in Zookeeper per node
-        node_log_lines: 2000
-      # networking: PVC networking configuration
-      # OPTIONAL if enable_networking: False
-      networking:
-        # bridge_device: Underlying device to use for bridged vLAN networks; usually the device of <cluster>
-        bridge_device: ens4
-        # bridge_mtu: The MTU of the underlying device used for bridged vLAN networks, and thus the maximum
-        #             MTU of the overlying bridge devices.
-        bridge_mtu: 1500
-        # sriov_enable: Enable or disable (default if absent) SR-IOV network support
-        sriov_enable: False
-        # sriov_device: Underlying device(s) to use for SR-IOV networks; can be bridge_device or other NIC(s)
-        sriov_device:
-            # The physical device name
-          - phy: ens1f1 
-            # The preferred MTU of the physical device; OPTIONAL - defaults to the interface default if unset
-            mtu: 9000
-            # The number of VFs to enable on this device
-            # NOTE: This defines the maximum number of VMs which can be provisioned on this physical device; VMs
-            #       are allocated to these VFs manually by the administrator and thus all nodes should have the
-            #       same number
-            # NOTE: This value cannot be changed at runtime on Intel(R) NICs; the node will need to be restarted
-            #       if this value changes
-            vfcount: 8
-        # upstream: Upstream physical interface device
-        upstream:
-          # device: Upstream interface device name
-          device: ens4
-          # mtu: Upstream interface MTU; use 9000 for jumbo frames (requires switch support)
-          mtu: 1500
-          # address: Upstream interface IP address, options: by-id, <static>/<mask>
-          address: by-id
-        # cluster: Cluster (VNIC) physical interface device
-        cluster:
-          # device: Cluster (VNIC) interface device name
-          device: ens4
-          # mtu: Cluster (VNIC) interface MTU; use 9000 for jumbo frames (requires switch support)
-          mtu: 1500
-          # address: Cluster (VNIC) interface IP address, options: by-id, <static>/<mask>
-          address: by-id
-        # storage: Storage (Ceph OSD) physical interface device
-        storage:
-          # device: Storage (Ceph OSD) interface device name
-          device: ens4
-          # mtu: Storage (Ceph OSD) interface MTU; use 9000 for jumbo frames (requires switch support)
-          mtu: 1500
-          # address: Storage (Ceph OSD) interface IP address, options: by-id, <static>/<mask>
-          address: by-id
-      # storage; PVC storage configuration
-      # OPTIONAL if enable_storage: False
-      storage:
-        # ceph_config_file: The config file containing the Ceph cluster configuration
-        ceph_config_file: "/etc/ceph/ceph.conf"
-        # ceph_admin_keyring: The file containing the Ceph client admin keyring
-        ceph_admin_keyring: "/etc/ceph/ceph.client.admin.keyring"

node-daemon/pvcnoded.service

@@ -10,7 +10,7 @@ PartOf = pvc.target
 Type = simple
 WorkingDirectory = /usr/share/pvc
 Environment = PYTHONUNBUFFERED=true
-Environment = PVCD_CONFIG_FILE=/etc/pvc/pvcnoded.yaml
+Environment = PVC_CONFIG_FILE=/etc/pvc/pvc.conf
 ExecStartPre = /bin/sleep 2
 ExecStart = /usr/share/pvc/pvcnoded.py
 ExecStopPost = /bin/sleep 2

node-daemon/pvcnoded/Daemon.py

@@ -49,7 +49,7 @@ import re
 import json
 
 # Daemon version
-version = "0.9.81"
+version = "0.9.82"
 
 
 ##########################################################
@@ -65,11 +65,6 @@ def entrypoint():
     config = pvcnoded.util.config.get_configuration()
     config["pvcnoded_version"] = version
 
-    # Set some useful booleans for later (fewer characters)
-    debug = config["debug"]
-    if debug:
-        print("DEBUG MODE ENABLED")
-
     # Create and validate our directories
     pvcnoded.util.config.validate_directories(config)
 
@@ -78,26 +73,26 @@ def entrypoint():
 
     # Print our startup message
     logger.out("")
-    logger.out("|----------------------------------------------------------|")
+    logger.out("|------------------------------------------------------------|")
-    logger.out("|                                                          |")
+    logger.out("|                                                            |")
-    logger.out("|           ███████████ ▜█▙      ▟█▛ █████ █ █ █           |")
+    logger.out("|            ███████████ ▜█▙      ▟█▛ █████ █ █ █            |")
-    logger.out("|                    ██  ▜█▙    ▟█▛  ██                    |")
+    logger.out("|                     ██  ▜█▙    ▟█▛  ██                     |")
-    logger.out("|           ███████████   ▜█▙  ▟█▛   ██                    |")
+    logger.out("|            ███████████   ▜█▙  ▟█▛   ██                     |")
-    logger.out("|           ██             ▜█▙▟█▛    ███████████           |")
+    logger.out("|            ██             ▜█▙▟█▛    ███████████            |")
-    logger.out("|                                                          |")
+    logger.out("|                                                            |")
-    logger.out("|----------------------------------------------------------|")
+    logger.out("|------------------------------------------------------------|")
-    logger.out("| Parallel Virtual Cluster node daemon v{0: <18} |".format(version))
+    logger.out("| Parallel Virtual Cluster node daemon v{0: <20} |".format(version))
-    logger.out("| Debug: {0: <49} |".format(str(config["debug"])))
+    logger.out("| Debug: {0: <51} |".format(str(config["debug"])))
-    logger.out("| FQDN: {0: <50} |".format(config["node_fqdn"]))
+    logger.out("| FQDN: {0: <52} |".format(config["node_fqdn"]))
-    logger.out("| Host: {0: <50} |".format(config["node_hostname"]))
+    logger.out("| Host: {0: <52} |".format(config["node_hostname"]))
-    logger.out("| ID: {0: <52} |".format(config["node_id"]))
+    logger.out("| ID: {0: <54} |".format(config["node_id"]))
-    logger.out("| IPMI hostname: {0: <41} |".format(config["ipmi_hostname"]))
+    logger.out("| IPMI hostname: {0: <43} |".format(config["ipmi_hostname"]))
-    logger.out("| Machine details:                                         |")
+    logger.out("| Machine details:                                           |")
-    logger.out("|   CPUs: {0: <48} |".format(config["static_data"][0]))
+    logger.out("|   CPUs: {0: <50} |".format(config["static_data"][0]))
-    logger.out("|   Arch: {0: <48} |".format(config["static_data"][3]))
+    logger.out("|   Arch: {0: <50} |".format(config["static_data"][3]))
-    logger.out("|   OS: {0: <50} |".format(config["static_data"][2]))
+    logger.out("|   OS: {0: <52} |".format(config["static_data"][2]))
-    logger.out("|   Kernel: {0: <46} |".format(config["static_data"][1]))
+    logger.out("|   Kernel: {0: <48} |".format(config["static_data"][1]))
-    logger.out("|----------------------------------------------------------|")
+    logger.out("|------------------------------------------------------------|")
     logger.out("")
     logger.out(f'Starting pvcnoded on host {config["node_fqdn"]}', state="s")
 

node-daemon/pvcnoded/dnsmasq-zookeeper-leases.py

@@ -70,12 +70,12 @@ def get_client_id():
 def connect_zookeeper():
     # We expect the environ to contain the config file
     try:
-        pvcnoded_config_file = os.environ["PVCD_CONFIG_FILE"]
+        pvc_config_file = os.environ["PVC_CONFIG_FILE"]
     except Exception:
         # Default place
-        pvcnoded_config_file = "/etc/pvc/pvcnoded.yaml"
+        pvc_config_file = "/etc/pvc/pvc.conf"
 
-    with open(pvcnoded_config_file, "r") as cfgfile:
+    with open(pvc_config_file, "r") as cfgfile:
         try:
             o_config = yaml.load(cfgfile, yaml.SafeLoader)
         except Exception as e:
@@ -87,7 +87,7 @@ def connect_zookeeper():
 
     try:
         zk_conn = kazoo.client.KazooClient(
-            hosts=o_config["pvc"]["cluster"]["coordinators"]
+            hosts=o_config["cluster"]["coordinator_nodes"]
         )
         zk_conn.start()
     except Exception as e:

node-daemon/pvcnoded/objects/VXNetworkInstance.py

@@ -743,10 +743,27 @@ add rule inet filter forward ip6 saddr {netaddr6} counter jump {vxlannic}-out
             )
 
             # Recreate the environment we need for dnsmasq
-            pvcnoded_config_file = os.environ["PVCD_CONFIG_FILE"]
+            pvc_config_file = None
+            try:
+                _config_file = "/etc/pvc/pvcnoded.yaml"
+                if not os.path.exists(_config_file):
+                    raise
+                pvc_config_file = _config_file
+            except Exception:
+                pass
+            try:
+                _config_file = os.environ["PVC_CONFIG_FILE"]
+                if not os.path.exists(_config_file):
+                    raise
+                pvc_config_file = _config_file
+            except Exception:
+                pass
+            if pvc_config_file is None:
+                raise Exception
+
             dhcp_environment = {
                 "DNSMASQ_BRIDGE_INTERFACE": self.bridge_nic,
-                "PVCD_CONFIG_FILE": pvcnoded_config_file,
+                "PVC_CONFIG_FILE": pvc_config_file,
             }
 
             # Define the dnsmasq config fragments

node-daemon/pvcnoded/util/config.py

@@ -70,12 +70,30 @@ def get_static_data():
 
 
 def get_configuration_path():
+    config_file = None
     try:
-        return os.environ["PVCD_CONFIG_FILE"]
+        _config_file = "/etc/pvc/pvcnoded.yaml"
-    except KeyError:
+        if not os.path.exists(_config_file):
-        print('ERROR: The "PVCD_CONFIG_FILE" environment variable must be set.')
+            raise
+        config_file = _config_file
+        config_type = "legacy"
+    except Exception:
+        pass
+    try:
+        _config_file = os.environ["PVC_CONFIG_FILE"]
+        if not os.path.exists(_config_file):
+            raise
+        config_file = _config_file
+        config_type = "current"
+    except Exception:
+        pass
+
+    if not config_file:
+        print('ERROR: The "PVC_CONFIG_FILE" environment variable must be set.')
         os._exit(1)
 
+    return config_file, config_type
+
 
 def get_hostname():
     node_fqdn = gethostname()
@@ -119,12 +137,244 @@ def validate_floating_ip(config, network):
     return True, ""
 
 
-def get_configuration():
+def get_configuration_current(config_file):
-    """
+    print('Loading configuration from file "{}"'.format(config_file))
-    Parse the configuration of the node daemon.
-    """
-    pvcnoded_config_file = get_configuration_path()
 
+    with open(config_file, "r") as cfgfh:
+        try:
+            o_config = yaml.load(cfgfh, Loader=yaml.SafeLoader)
+        except Exception as e:
+            print(f"ERROR: Failed to parse configuration file: {e}")
+            os._exit(1)
+
+    config = dict()
+
+    node_fqdn, node_hostname, node_domain, node_id = get_hostname()
+
+    config_thisnode = {
+        "node": node_hostname,
+        "node_hostname": node_hostname,
+        "node_fqdn": node_fqdn,
+        "node_domain": node_domain,
+        "node_id": node_id,
+    }
+    config = {**config, **config_thisnode}
+
+    try:
+        o_path = o_config["path"]
+        config_path = {
+            "plugin_directory": o_path.get(
+                "plugin_directory", "/usr/share/pvc/plugins"
+            ),
+            "dynamic_directory": o_path["dynamic_directory"],
+            "log_directory": o_path["system_log_directory"],
+            "console_log_directory": o_path["console_log_directory"],
+            "ceph_directory": o_path["ceph_directory"],
+        }
+        # Define our dynamic directory schema
+        config_path["dnsmasq_dynamic_directory"] = (
+            config_path["dynamic_directory"] + "/dnsmasq"
+        )
+        config_path["pdns_dynamic_directory"] = (
+            config_path["dynamic_directory"] + "/pdns"
+        )
+        config_path["nft_dynamic_directory"] = config_path["dynamic_directory"] + "/nft"
+        # Define our log directory schema
+        config_path["dnsmasq_log_directory"] = config_path["log_directory"] + "/dnsmasq"
+        config_path["pdns_log_directory"] = config_path["log_directory"] + "/pdns"
+        config_path["nft_log_directory"] = config_path["log_directory"] + "/nft"
+        config = {**config, **config_path}
+
+        o_subsystem = o_config["subsystem"]
+        config_subsystem = {
+            "enable_hypervisor": o_subsystem.get("enable_hypervisor", True),
+            "enable_networking": o_subsystem.get("enable_networking", True),
+            "enable_storage": o_subsystem.get("enable_storage", True),
+            "enable_worker": o_subsystem.get("enable_worker", True),
+            "enable_api": o_subsystem.get("enable_api", True),
+        }
+        config = {**config, **config_subsystem}
+
+        o_cluster = o_config["cluster"]
+        config_cluster = {
+            "cluster_name": o_cluster["name"],
+            "all_nodes": o_cluster["all_nodes"],
+            "coordinators": o_cluster["coordinator_nodes"],
+        }
+        config = {**config, **config_cluster}
+
+        o_cluster_networks = o_cluster["networks"]
+        for network_type in ["cluster", "storage", "upstream"]:
+            o_cluster_networks_specific = o_cluster_networks[network_type]
+            config_cluster_networks_specific = {
+                f"{network_type}_domain": o_cluster_networks_specific["domain"],
+                f"{network_type}_dev": o_cluster_networks_specific["device"],
+                f"{network_type}_mtu": o_cluster_networks_specific["mtu"],
+                f"{network_type}_network": o_cluster_networks_specific["ipv4"][
+                    "network_address"
+                ]
+                + "/"
+                + str(o_cluster_networks_specific["ipv4"]["netmask"]),
+                f"{network_type}_floating_ip": o_cluster_networks_specific["ipv4"][
+                    "floating_address"
+                ]
+                + "/"
+                + str(o_cluster_networks_specific["ipv4"]["netmask"]),
+                f"{network_type}_node_ip_selection": o_cluster_networks_specific[
+                    "node_ip_selection"
+                ],
+            }
+
+            if (
+                o_cluster_networks_specific["ipv4"].get("gateway_address", None)
+                is not None
+            ):
+                config[f"{network_type}_gateway"] = o_cluster_networks_specific["ipv4"][
+                    "gateway_address"
+                ]
+
+            result, msg = validate_floating_ip(
+                config_cluster_networks_specific, network_type
+            )
+            if not result:
+                raise MalformedConfigurationError(msg)
+
+            network = ip_network(
+                config_cluster_networks_specific[f"{network_type}_network"]
+            )
+
+            if (
+                config_cluster_networks_specific[f"{network_type}_node_ip_selection"]
+                == "by-id"
+            ):
+                address_id = int(node_id) - 1
+            else:
+                # This roundabout solution ensures the given IP is in the subnet and is something valid
+                address_id = [
+                    idx
+                    for idx, ip in enumerate(list(network.hosts()))
+                    if str(ip)
+                    == config_cluster_networks_specific[
+                        f"{network_type}_node_ip_selection"
+                    ]
+                ][0]
+
+            config_cluster_networks_specific[
+                f"{network_type}_dev_ip"
+            ] = f"{list(network.hosts())[address_id]}/{network.prefixlen}"
+
+            config = {**config, **config_cluster_networks_specific}
+
+        o_database = o_config["database"]
+        config_database = {
+            "zookeeper_port": o_database["zookeeper"]["port"],
+            "keydb_port": o_database["keydb"]["port"],
+            "keydb_host": o_database["keydb"]["hostname"],
+            "keydb_path": o_database["keydb"]["path"],
+            "metadata_postgresql_port": o_database["postgres"]["port"],
+            "metadata_postgresql_host": o_database["postgres"]["hostname"],
+            "metadata_postgresql_dbname": o_database["postgres"]["credentials"]["api"][
+                "database"
+            ],
+            "metadata_postgresql_user": o_database["postgres"]["credentials"]["api"][
+                "username"
+            ],
+            "metadata_postgresql_password": o_database["postgres"]["credentials"][
+                "api"
+            ]["password"],
+            "pdns_postgresql_port": o_database["postgres"]["port"],
+            "pdns_postgresql_host": o_database["postgres"]["hostname"],
+            "pdns_postgresql_dbname": o_database["postgres"]["credentials"]["dns"][
+                "database"
+            ],
+            "pdns_postgresql_user": o_database["postgres"]["credentials"]["dns"][
+                "username"
+            ],
+            "pdns_postgresql_password": o_database["postgres"]["credentials"]["dns"][
+                "password"
+            ],
+        }
+        config = {**config, **config_database}
+
+        o_timer = o_config["timer"]
+        config_timer = {
+            "vm_shutdown_timeout": int(o_timer.get("vm_shutdown_timeout", 180)),
+            "keepalive_interval": int(o_timer.get("keepalive_interval", 5)),
+            "monitoring_interval": int(o_timer.get("monitoring_interval", 60)),
+        }
+        config = {**config, **config_timer}
+
+        o_fencing = o_config["fencing"]
+        config_fencing = {
+            "disable_on_ipmi_failure": o_fencing["disable_on_ipmi_failure"],
+            "fence_intervals": int(o_fencing["intervals"].get("fence_intervals", 6)),
+            "suicide_intervals": int(o_fencing["intervals"].get("suicide_interval", 0)),
+            "successful_fence": o_fencing["actions"].get("successful_fence", None),
+            "failed_fence": o_fencing["actions"].get("failed_fence", None),
+            "ipmi_hostname": o_fencing["ipmi"]["hostname"].format(node_id=node_id),
+            "ipmi_username": o_fencing["ipmi"]["username"],
+            "ipmi_password": o_fencing["ipmi"]["password"],
+        }
+        config = {**config, **config_fencing}
+
+        o_migration = o_config["migration"]
+        config_migration = {
+            "migration_target_selector": o_migration.get("target_selector", "mem"),
+        }
+        config = {**config, **config_migration}
+
+        o_logging = o_config["logging"]
+        config_logging = {
+            "debug": o_logging.get("debug_logging", False),
+            "file_logging": o_logging.get("file_logging", False),
+            "stdout_logging": o_logging.get("stdout_logging", False),
+            "zookeeper_logging": o_logging.get("zookeeper_logging", False),
+            "log_colours": o_logging.get("log_colours", False),
+            "log_dates": o_logging.get("log_dates", False),
+            "log_keepalives": o_logging.get("log_keepalives", False),
+            "log_keepalive_cluster_details": o_logging.get(
+                "log_cluster_details", False
+            ),
+            "log_keepalive_plugin_details": o_logging.get(
+                "log_monitoring_details", False
+            ),
+            "console_log_lines": o_logging.get("console_log_lines", False),
+            "node_log_lines": o_logging.get("node_log_lines", False),
+        }
+        config = {**config, **config_logging}
+
+        o_guest_networking = o_config["guest_networking"]
+        config_guest_networking = {
+            "bridge_dev": o_guest_networking["bridge_device"],
+            "bridge_mtu": o_guest_networking["bridge_mtu"],
+            "enable_sriov": o_guest_networking.get("sriov_enable", False),
+            "sriov_device": o_guest_networking.get("sriov_device", list()),
+        }
+        config = {**config, **config_guest_networking}
+
+        o_ceph = o_config["ceph"]
+        config_ceph = {
+            "ceph_config_file": config["ceph_directory"]
+            + "/"
+            + o_ceph["ceph_config_file"],
+            "ceph_admin_keyring": config["ceph_directory"]
+            + "/"
+            + o_ceph["ceph_keyring_file"],
+            "ceph_monitor_port": o_ceph["monitor_port"],
+            "ceph_secret_uuid": o_ceph["secret_uuid"],
+        }
+        config = {**config, **config_ceph}
+
+        # Add our node static data to the config
+        config["static_data"] = get_static_data()
+
+    except Exception as e:
+        raise MalformedConfigurationError(e)
+
+    return config
+
+
+def get_configuration_legacy(pvcnoded_config_file):
     print('Loading configuration from file "{}"'.format(pvcnoded_config_file))
 
     with open(pvcnoded_config_file, "r") as cfgfile:
@@ -168,6 +418,7 @@ def get_configuration():
         "enable_hypervisor": o_functions.get("enable_hypervisor", False),
         "enable_networking": o_functions.get("enable_networking", False),
         "enable_storage": o_functions.get("enable_storage", False),
+        "enable_worker": o_functions.get("enable_worker", True),
         "enable_api": o_functions.get("enable_api", False),
     }
 
@@ -417,6 +668,20 @@ def get_configuration():
     return config
 
 
+def get_configuration():
+    """
+    Parse the configuration of the node daemon.
+    """
+    pvc_config_file, pvc_config_type = get_configuration_path()
+
+    if pvc_config_type == "legacy":
+        config = get_configuration_legacy(pvc_config_file)
+    else:
+        config = get_configuration_current(pvc_config_file)
+
+    return config
+
+
 def validate_directories(config):
     if not os.path.exists(config["dynamic_directory"]):
         os.makedirs(config["dynamic_directory"])

node-daemon/pvcnoded/util/services.py

@@ -70,14 +70,16 @@ def start_ceph_mgr(logger, config):
 
 
 def start_keydb(logger, config):
-    if config["enable_api"] and config["daemon_mode"] == "coordinator":
+    if (config["enable_api"] or config["enable_worker"]) and config[
+        "daemon_mode"
+    ] == "coordinator":
         logger.out("Starting KeyDB daemon", state="i")
         # TODO: Move our handling out of Systemd and integrate it directly as a subprocess?
         common.run_os_command("systemctl start keydb-server.service")
 
 
-def start_api_worker(logger, config):
+def start_worker(logger, config):
-    if config["enable_api"]:
+    if config["enable_worker"]:
         logger.out("Starting Celery Worker daemon", state="i")
         # TODO: Move our handling out of Systemd and integrate it directly as a subprocess?
         common.run_os_command("systemctl start pvcworkerd.service")
@@ -91,7 +93,7 @@ def start_system_services(logger, config):
     start_ceph_mon(logger, config)
     start_ceph_mgr(logger, config)
     start_keydb(logger, config)
-    start_api_worker(logger, config)
+    start_worker(logger, config)
 
     logger.out("Waiting 10 seconds for daemons to start", state="s")
     sleep(10)

pvc.sample.conf

@@ -0,0 +1,452 @@
+---
+# PVC system configuration - example file
+#
+# This configuration file defines the details of a PVC cluster.
+# It is used by several daemons on the system, including pvcnoded, pvcapid, pvcworkerd, and pvchealthd.
+#
+# This file will normally be written by the PVC Ansible framework; this example is provided for reference
+
+# Paths configuration
+path:
+
+  # Plugin directory
+  plugin_directory: "/usr/share/pvc/plugins"
+
+  # Dynamic directory
+  dynamic_directory: "/run/pvc"
+
+  # System log directory
+  system_log_directory: "/var/log/pvc"
+
+  # VM Console log directory (set by Libvirt)
+  console_log_directory: "/var/log/libvirt"
+
+  # Ceph configuration directory (set by Ceph/Ansible)
+  ceph_directory: "/etc/ceph"
+
+# Subsystem configuration
+# Changing these values can be used to turn on or off various parts of PVC
+# Normally, all should be enabled ("yes") except in very custom clusters
+subsystem:
+
+    # Enable or disable hypervisor functionality
+    enable_hypervisor: yes
+
+    # Enable or disable virtual networking and routing functionality
+    enable_networking: yes
+
+    # Enable or disable Ceph storage management functionality
+    enable_storage: yes
+
+    # Enable or disable the worker client
+    enable_worker: yes
+
+    # Enable or disable the API client, if installed, when node is Primary
+    enable_api: yes
+
+# Cluster configuration
+cluster:
+
+  # The name of the cluster
+  name: pvc1
+
+  # The full list of nodes in this cluster
+  all_nodes:
+    - pvchv1
+    - pvchv2
+    - pvchv3
+
+  # The list of coorrdinator nodes in this cluster (subset of nodes)
+  coordinator_nodes:
+    - pvchv1
+    - pvchv2
+    - pvchv3
+
+  # Hardcoded networks (upstream/cluster/storage)
+  networks:
+
+    # Upstream network, used for inbound and outbound connectivity, API management, etc.
+    upstream:
+
+      # Domain name
+      domain: "mydomain.net"
+
+      # Device
+      device: ens4
+
+      # MTU
+      mtu: 1500
+
+      # IPv4 configuration
+      ipv4:
+
+        # CIDR netmask
+        netmask: 24
+
+        # Network address
+        network_address: 10.0.0.0
+
+        # Floating address
+        floating_address: 10.0.0.250
+
+        # Upstream/default gateway address
+        gateway_address: 10.0.0.254
+
+      # Node IP selection mechanism (either "by-id", or a static IP, no netmask, in the above network)
+      node_ip_selection: by-id
+
+    # Cluster network, used for inter-node communication (VM- and Network-layer), unrouted
+    cluster:
+
+      # Domain name
+      domain: "pvc.local"
+
+      # Device
+      device: ens4
+
+      # MTU
+      mtu: 1500
+
+      # IPv4 configuration
+      ipv4:
+      
+        # CIDR netmask
+        netmask: 24
+
+        # Network address
+        network_address: 10.0.1.0
+
+        # Floating address
+        floating_address: 10.0.1.250
+
+      # Node IP selection mechanism (either "by-id", or a static IP, no netmask, in the above network)
+      node_ip_selection: by-id
+
+    # Storage network, used for inter-node communication (Storage-layer), unrouted
+    storage:
+
+      # Domain name
+      domain: "storage.local"
+
+      # Device
+      device: ens4
+
+      # MTU
+      mtu: 1500
+
+      # IPv4 configuration
+      ipv4:
+
+        # CIDR netmask
+        netmask: 24
+
+        # Network address
+        network_address: 10.0.2.0
+
+        # Floating address
+        floating_address: 10.0.2.250
+
+      # Node IP selection mechanism (either "by-id", or a static IP, no netmask, in the above network)
+      node_ip_selection: by-id
+
+# Database configuration
+database:
+
+  # Zookeeper client configuration
+  zookeeper:
+
+    # Port number
+    port: 2181
+
+  # KeyDB/Redis client configuration
+  keydb:
+
+    # Port number
+    port: 6379
+
+    # Hostname; use `cluster` network floating IP address
+    hostname: 10.0.1.250
+
+    # Path, usually "/0"
+    path: "/0"
+
+  # PostgreSQL client configuration
+  postgres:
+    
+    # Port number
+    port: 5432
+
+    # Hostname; use `cluster` network floating IP address
+    hostname: 10.0.1.250
+
+    # Credentials
+    credentials:
+    
+      # API database
+      api:
+
+        # Database name
+        database: pvcapi
+
+        # Username
+        username: pvcapi
+
+        # Password
+        password: pvcapiPassw0rd
+
+      # DNS database
+      dns:
+      
+        # Database name
+        database: pvcdns
+
+        # Username
+        username: pvcdns
+
+        # Password
+        password: pvcdnsPassw0rd
+
+# Timer information
+timer:
+
+  # VM shutdown timeout (seconds)
+  vm_shutdown_timeout: 180
+
+  # Node keepalive interval (seconds)
+  keepalive_interval: 5
+
+  # Monitoring interval (seconds)
+  monitoring_interval: 60
+
+# Fencing configuration
+fencing:
+
+  # Disable fencing or not on IPMI failure at startup
+  # Setting this value to "no" will allow fencing to be enabled even if does not respond during node daemon
+  # startup. This will allow future fencing to be attempted if it later recovers.
+  disable_on_ipmi_failure: no
+
+  # Fencing intervals
+  intervals:
+
+    # Fence intervals (number of keepalives)
+    fence_intervals: 6
+
+    # Suicide intervals (number of keepalives; 0 to disable)
+    suicide_intervals: 0
+
+  # Fencing actions
+  actions:
+  
+    # Successful fence action ("migrate" or "none")
+    successful_fence: migrate
+
+    # Failed fence action ("migrate" or "none")
+    failed_fence: none
+
+  # IPMI details
+  ipmi:
+
+    # Hostname format; use a "{node_id}" entry for a template, or a literal hostname
+    hostname: "pvchv{node_id}-lom.mydomain.tld"
+
+    # IPMI username
+    username: admin
+
+    # IPMI password
+    password: S3cur3IPMIP4ssw0rd
+
+
+# VM migration configuration
+migration:
+
+  # Target selection default value (mem, memprov, load, vcpus, vms)
+  target_selector: mem
+
+# Logging configuration
+logging:
+
+  # Enable or disable debug logging (all services)
+  debug_logging: yes
+
+  # Enable or disable file logging
+  file_logging: no
+
+  # Enable or disable stdout logging (to journald)
+  stdout_logging: yes
+
+  # Enable or disable Zookeeper logging (for "pvc node log" functionality)
+  zookeeper_logging: yes
+
+  # Enable or disable ANSI colour sequences in logs
+  log_colours: yes
+
+  # Enable or disable dates in logs
+  log_dates: yes
+
+  # Enale or disable keepalive event logging
+  log_keepalives: yes
+
+  # Enable or disable cluster detail logging during keepalive events
+  log_cluster_details: yes
+
+  # Enable or disable monitoring detail logging during keepalive events
+  log_monitoring_details: yes
+
+  # Number of VM console log lines to store in Zookeeper (per VM)
+  console_log_lines: 1000
+
+  # Number of node log lines to store in Zookeeper (per node)
+  node_log_lines: 2000
+
+# Guest networking configuration
+guest_networking:
+
+  # Bridge device for "bridged"-type networks
+  bridge_device: ens4
+
+  # Bridge device MTU
+  bridge_mtu: 1500
+
+  # Enable or disable SR-IOV functionality
+  sriov_enable: no
+
+  # SR-IOV configuration (list of PFs)
+  sriov_device:
+
+      # SR-IOV device; if this device isn't found, it is ignored on a given node
+    - device: ens1f1
+
+      # SR-IOV device MTU
+      mtu: 9000
+
+      # Number of VFs on this device
+      vfcount: 4
+
+# Ceph configuration
+ceph:
+
+  # Main config file name
+  ceph_config_file: "ceph.conf"
+
+  # Admin keyring file name
+  ceph_keyring_file: "ceph.client.admin.keyring"
+
+  # Monitor port, usually 6789
+  monitor_port: 6789
+
+  # Monitor host(s), enable only you want to use hosts other than the coordinators
+  #monitor_hosts:
+  #  - pvchv1
+  #  - pvchv2
+  #  - pvchv3
+
+  # Storage secret UUID, generated during Ansible cluster bootstrap
+  secret_uuid: ""
+
+# API configuration
+api:
+
+  # API listening configuration
+  listen:
+
+    # Listen address, usually upstream floating IP
+    address: 10.0.0.250
+
+    # Listen port, usually 7370
+    port: 7370
+
+  # Authentication configuration
+  authentication:
+  
+    # Enable or disable authentication
+    enabled: yes
+
+    # Secret key for API cookies (long and secure password or UUID)
+    secret_key: "1234567890abcdefghijklmnopqrstuvwxyz"
+
+    # Authentication source (token, others in future)
+    source: token
+
+  # Token configuration
+  token:
+
+      # A friendly description
+    - description: "testing"
+        
+      # The token (long and secure password or UUID)
+      token: "1234567890abcdefghijklmnopqrstuvwxyz"
+
+  # SSL configuration
+  ssl:
+
+    # Enable or disable SSL operation
+    enabled: no
+
+    # Certificate file path
+    certificate: ""
+
+    # Private key file path
+    private_key: ""
+
+# Automatic backups
+autobackup:
+
+  # Backup root path on the node, used as the remote mountpoint
+  # Must be an absolute path beginning with '/'
+  # If remote_mount is enabled, the remote mount will be mounted on this directory
+  # If remote_mount is enabled, it is recommended to use a path under `/tmp` for this
+  # If remote_mount is disabled, a real filesystem must be mounted here (PVC system volumes are small!)
+  backup_root_path: "/tmp/backups"
+
+  # Suffix to the backup root path, used to allow multiple PVC systems to write to a single root path
+  # Must begin with '/'; leave empty to use the backup root path directly
+  # Note that most remote mount options can fake this if needed, but provided to ensure local compatability
+  backup_root_suffix: "/mycluster"
+
+  # VM tag(s) to back up
+  # Only VMs with at least one of the given tag(s) will be backed up; all others will be skipped
+  backup_tags:
+    - "backup"
+    - "mytag"
+
+  # Backup schedule: when and what format to take backups
+  backup_schedule:
+
+    full_interval:  7  # Number of total backups between full backups; others are incremental
+                       # > If this number is 1, every backup will be a full backup and no incremental
+                       #   backups will be taken
+                       # > If this number is 2, every second backup will be a full backup, etc.
+
+    full_retention: 2  # Keep this many full backups; the oldest will be deleted when a new one is
+                       # taken, along with all child incremental backups of that backup
+                       # > Should usually be at least 2 when using incrementals (full_interval > 1) to
+                       #   avoid there being too few backups after cleanup from a new full backup
+
+  # Automatic mount settings
+  # These settings permit running an arbitrary set of commands, ideally a "mount" command or similar, to
+  # ensure that a remote filesystem is mounted on the backup root path
+  # While the examples here show absolute paths, that is not required; they will run with the $PATH of the
+  # executing environment (either the "pvc" command on a CLI or a cron/systemd timer)
+  # A "{backup_root_path}" f-string/str.format type variable MAY be present in any cmds string to represent
+  # the above configured root backup path, which is interpolated at runtime
+  # If multiple commands are given, they will be executed in the order given; if no commands are given,
+  # nothing is executed, but the keys MUST be present
+  auto_mount:
+
+    enabled: no  # Enable automatic mount/unmount support
+
+    # These commands are executed at the start of the backup run and should mount a filesystem
+    mount_cmds:
+
+      # This example shows an NFS mount leveraging the backup_root_path variable
+      - "/usr/sbin/mount.nfs -o nfsvers=3 10.0.0.10:/backups {backup_root_path}"
+
+    # These commands are executed at the end of the backup run and should unmount a filesystem
+    unmount_cmds:
+
+      # This example shows a generic umount leveraging the backup_root_path variable
+      - "/usr/bin/umount {backup_root_path}"
+
+# VIM modeline, requires "set modeline" in your VIMRC
+# vim: expandtab shiftwidth=2 tabstop=2 filetype=yaml