Fix incorrect error handling

Avoids reprovisioning existing nodes until they are manually removed
from the database.

README.md

@@ -1,46 +1,18 @@
-<p align="center">
-<img alt="Logo banner" src="https://docs.parallelvirtualcluster.org/en/latest/images/pvc_logo_black.png"/>
-<br/><br/>
-<a href="https://www.parallelvirtualcluster.org"><img alt="Website" src="https://img.shields.io/badge/visit-website-blue"/></a>
-<a href="https://github.com/parallelvirtualcluster/pvc/releases"><img alt="Latest Release" src="https://img.shields.io/github/release-pre/parallelvirtualcluster/pvc"/></a>
-<a href="https://docs.parallelvirtualcluster.org/en/latest/?badge=latest"><img alt="Documentation Status" src="https://readthedocs.org/projects/parallelvirtualcluster/badge/?version=latest"/></a>
-<a href="https://github.com/parallelvirtualcluster/pvc"><img alt="License" src="https://img.shields.io/github/license/parallelvirtualcluster/pvc"/></a>
-<a href="https://github.com/psf/black"><img alt="Code style: Black" src="https://img.shields.io/badge/code%20style-black-000000.svg"/></a>
-</p>
-
-## What is PVC?
-
-PVC is a Linux KVM-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Ganeti, Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.
-
-PVC is a complete HCI solution, built from well-known and well-trusted Free Software tools, to assist an administrator in creating and managing a cluster of servers to run virtual machines, as well as self-managing several important aspects including storage failover, node failure and recovery, virtual machine failure and recovery, and network plumbing. It is designed to act consistently, reliably, and unobtrusively, letting the administrator concentrate on more important things.
-
-PVC is highly scalable. From a minimum (production) node count of 3, up to 12 or more, and supporting many dozens of VMs, PVC scales along with your workload and requirements. Deploy a cluster once and grow it as your needs expand.
-
-As a consequence of its features, PVC makes administrating very high-uptime VMs extremely easy, featuring VM live migration, built-in always-enabled shared storage with transparent multi-node replication, and consistent network plumbing throughout the cluster. Nodes can also be seamlessly removed from or added to service, with zero VM downtime, to facilitate maintenance, upgrades, or other work.
-
-PVC also features an optional, fully customizable VM provisioning framework, designed to automate and simplify VM deployments using custom provisioning profiles, scripts, and CloudInit userdata API support.
-
-Installation of PVC is accomplished by two main components: a [Node installer ISO](https://github.com/parallelvirtualcluster/pvc-installer) which creates on-demand installer ISOs, and an [Ansible role framework](https://github.com/parallelvirtualcluster/pvc-ansible) to configure, bootstrap, and administrate the nodes. Installation can also be fully automated with a companion [cluster bootstrapping system](https://github.com/parallelvirtualcluster/pvc-bootstrap). Once up, the cluster is managed via an HTTP REST API, accessible via a Python Click CLI client ~~or WebUI~~ (eventually).
-
-Just give it physical servers, and it will run your VMs without you having to think about it, all in just an hour or two of setup time.
-
-More information about PVC, its motivations, the hardware requirements, and setting up and managing a cluster [can be found over at our docs page](https://docs.parallelvirtualcluster.org).
-
 # PVC Bootstrap System
 
 The PVC bootstrap system provides a convenient way to deploy PVC clusters. Rather than manual node installation, this system provides a fully-automated deployment from node powering to cluster readiness, based on pre-configured values. It is useful if an administrator will deploy several PVC clusters or for repeated re-deployment for testing purposes.
 
-# Setup
+## Setup
 
 Setting up the PVC bootstrap system manually is very complicated, and has thus been automated with an installer script instead of providing a Debian or PIP package.
 
-## Preparing to use the PVC Bootstrap system
+### Preparing to use the PVC Bootstrap system
 
 1. Prepare a Git repository to store cluster configurations. This can be done automatically with the `create-local-repo.sh` script in the [PVC Ansible](https://github.com/parallelvirtualcluster/pvc-ansible) repository.
 
 1. Create `group_vars` for each cluster you plan to bootstrap. Additionally, ensure you configure the `bootstrap.yml` file for each cluster with the relevant details of the hardware you will be using. This step can be repeated for each cluster in the future as new clusters are required, and the system will automatically pull changes to the local PVC repository once configured.
 
-## Preparing a PVC Bootstrap host
+### Preparing a PVC Bootstrap host
 
 1. The recommended OS for a PVC Bootstrap host is Debian GNU/Linux 10+. In terms of hardware, there are several supported options:
 
@@ -56,7 +28,7 @@ Setting up the PVC bootstrap system manually is very complicated, and has thus b
 
 1. Run the `./install-pvcbootstrapd.sh` script from the root of the repository to install the PVC Bootstrap system on the host. It will prompt for several configuration parameters. The final steps will take some time (up to 2 hours on a Raspberry Pi 4B) so be patient.
 
-## Networking for Bootstrap
+### Networking for Bootstrap
 
 When using the pvcbootstrapd system, a dedicated network is required to provide bootstrap DHCP and TFTP to the cluster. This network can either have a dedicated, upstream router that does not provide DHCP, or the network can be routed with network address translation (NAT) through the bootstrap host. By default, the installer will configure the latter automatically using a second NIC separate from the upstream NIC of the bootstrap host, or via a vLAN on top of the single NIC.
 
@@ -76,7 +48,7 @@ Consider the following diagram for reference:
 
 ![Overall Network Topology](/docs/images/pvcbootstrapd-net.png)
 
-## Deploying a Cluster with PVC Bootstrap - Redfish
+### Deploying a Cluster with PVC Bootstrap - Redfish
 
 Redfish is an industry-standard RESTful API for interfacing with the BMC (baseband management controller, or out-of-band network management system) on modern (post ~2015) servers from most vendors, including Dell iDRAC, HP iLO, Cisco CIMC, Lenovo XCC, and Supermicro X10 and newer BMCs. Redfish allows remote management, data collection, and configuration from the BMC in a standardized way across server vendors.
 
@@ -92,7 +64,7 @@ The PVC Bootstrap system is designed to heavily leverage Redfish in its automati
 
 1. Verify and power off the servers and put them into production; you may need to complete several post-install tasks (for instance setting the production BMC networking via `sudo ifup ipmi` on each node) before the cluster is completely finished.
 
-## Deploying a Cluster with PVC Bootstrap - Non-Redfish
+### Deploying a Cluster with PVC Bootstrap - Non-Redfish
 
 The PVC Bootstrap system can still handle nodes without Redfish support, for instance older servers or those from non-compliant vendors. There is however more manual setup in the process. The steps are thus:
 
@@ -116,7 +88,7 @@ The PVC Bootstrap system can still handle nodes without Redfish support, for ins
 
 1. Verify and power off the servers and put them into production; you may need to complete several post-install tasks (for instance setting the production BMC networking via `sudo ifup ipmi` on each node) before the cluster is completely finished.
 
-### `host-MAC.ipxe`
+#### `host-MAC.ipxe`
 
 ```
 #1ipxe
@@ -134,7 +106,7 @@ The PVC Bootstrap system can still handle nodes without Redfish support, for ins
 set imgargs-host ARGUMENTS
 ```
 
-### `host-MAC.preseed`
+#### `host-MAC.preseed`
 
 ```
 # The name of this file is "host-123456abcdef.preseed", where "123456abcdef" is the MAC address of the
@@ -155,9 +127,9 @@ set imgargs-host ARGUMENTS
 # This file is thus not designed to be used by humans, and its values are seeded via options in
 # the cluster-local Ansible group_vars, though it can be used as a manual template if required.
 
-##
+###
-## General definitions/overrides
+### General definitions/overrides
-##
+###
 # The Debian release to use (overrides the default)
 debrelease="bullseye"
 
@@ -171,9 +143,9 @@ addpkglist="ca-certificates"
 filesystem="ext4"
 
 
-##
+###
-## Per-host definitions (required)
+### Per-host definitions (required)
-##
+###
 
 # The hostname of the system (set per-run)
 target_hostname="hv1.example.tld"
@@ -181,15 +153,13 @@ target_hostname="hv1.example.tld"
 # The target system disk path; must be a single disk (mdadm/software RAID is not supported)
 # This will usually use a `detect` string. A "detect" string is a string in the form "detect:<NAME>:<HUMAN-SIZE>:<ID>".
 # Detect strings allow for automatic determination of Linux block device paths from known basic information
-# about disks by leveraging "lsscsi"/"nvme" on the target host.
+# about disks by leveraging "lsscsi" on the target host. The "NAME" should be some descriptive identifier,
-# The "NAME" should be some descriptive identifier that would be part of the device's Model information, for instance
+# for instance the manufacturer (e.g. "INTEL"), the "HUMAN-SIZE" should be the labeled human-readable size
-# the manufacturer (e.g. "INTEL") or a similar unique string (e.g. "BOSS" for Dell BOSS cards).
+# of the device (e.g. "480GB", "1.92TB"), and "ID" specifies the Nth 0-indexed device which matches the
-# The "HUMAN-SIZE" should be the labeled human-readable size of the device (e.g. "480GB", "1.92TB").
+# NAME" and "HUMAN-SIZE" values (e.g. "2" would match the third device with the corresponding "NAME" and
-# The "ID" specifies the Nth 0-indexed device which matches the NAME" and "HUMAN-SIZE" values (e.g. "2" would match the
+# "HUMAN-SIZE"). When matching against sizes, there is +/- 3% flexibility to account for base-1000 vs.
-# third device with the corresponding "NAME" and "HUMAN-SIZE").
+# base-1024 differences and rounding errors. The "NAME" may contain whitespace but if so the entire detect
-# When matching against sizes, there is +/- 3% flexibility to account for base-1000 vs. base-1024 differences and
+# string should be quoted, and is case-insensitive.
-# rounding errors.
-# The "NAME" may contain whitespace but if so the entire detect string should be quoted, and is case-insensitive.
 target_disk="detect:LOGICAL:146GB:0"
 
 # SSH key fetch method (usually tftp)
@@ -216,7 +186,7 @@ target_deploy_user="deploy"
 pvcbootstrapd_checkin_uri="http://10.255.255.1:9999/checkin/host"
 ```
 
-# Bootstrap Process
+## Bootstrap Process
 
 This diagram outlines the various states the nodes and clusters will be in throughout the setup process along with the individual steps for reference. Which node starts characterizing first can be random, but is shown as `node1` for clarity. For non-Redflish installs, the first several steps must be completed manually as referenced above.
 

bootstrap-daemon/pvcbootstrapd-worker.sh

@@ -27,9 +27,13 @@ case "$( cat /etc/debian_version )" in
     10.*)
         CELERY_ARGS="worker --app pvcbootstrapd.flaskapi.celery --concurrency 99 --pool gevent --loglevel DEBUG"
     ;;
-    *)
+    11.*)
         CELERY_ARGS="--app pvcbootstrapd.flaskapi.celery worker --concurrency 99 --pool gevent --loglevel DEBUG"
     ;;
+    *)
+        echo "Invalid Debian version found!"
+        exit 1
+    ;;
 esac
 
 ${CELERY_BIN} ${CELERY_ARGS}

bootstrap-daemon/pvcbootstrapd.yaml.sample

@@ -64,11 +64,8 @@ pvc:
     # Replace "10.199.199.254" if you change "dhcp" -> "address" above
     mirror: http://10.199.199.254:3142/ftp.debian.org/debian
 
-    # Default Debian release for new clusters. Must be supported by PVC ("buster", "bullseye", "bookworm").
-    release: bookworm
-
   # PVC Ansible repository configuration
-  # Note: If "path" does not exist, "remote" will be cloned to it via Git using SSH private key "key_file".
+  # Note: If "path" does not exist, "remote" will be cloned to it via Git using SSH private key "keyfile".
   # Note: The VCS will be refreshed regularly via the API in response to webhooks.
   ansible:
     # Path to the VCS repository

bootstrap-daemon/pvcbootstrapd.yaml.template

@@ -23,14 +23,12 @@ pvc:
     host_path: "ROOT_DIRECTORY/tftp/host"
   repo:
     mirror: http://BOOTSTRAP_ADDRESS:3142/UPSTREAM_MIRROR
-    release: DEBIAN_RELEASE
   ansible:
     path: "ROOT_DIRECTORY/repo"
-    key_file: "ROOT_DIRECTORY/id_ed25519"
+    keyfile: "ROOT_DIRECTORY/id_ed25519"
     remote: "GIT_REMOTE"
     branch: "GIT_BRANCH"
     clusters_file: "clusters.yml"
-    lock_file: "/run/pvcbootstrapd.lock"
     cspec_files:
         base: "base.yml"
         pvc: "pvc.yml"

bootstrap-daemon/pvcbootstrapd/Daemon.py

@@ -180,7 +180,7 @@ def read_config():
             )
 
     # Get the Repo configuration
-    for key in ["mirror", "release"]:
+    for key in ["mirror"]:
         try:
             config[f"repo_{key}"] = o_repo[key]
         except Exception:

bootstrap-daemon/pvcbootstrapd/lib/git.py

@@ -70,7 +70,6 @@ def pull_repository(config):
             g.pull(rebase=True, env=dict(GIT_SSH_COMMAND=git_ssh_cmd))
             logger.debug("Performing git submodule update")
             g.submodule("update", "--init", env=dict(GIT_SSH_COMMAND=git_ssh_cmd))
-            g.submodule("update", env=dict(GIT_SSH_COMMAND=git_ssh_cmd))
         except Exception as e:
             logger.warn(e)
             notifications.send_webhook(config, "failure", "Failed to update Git repository")

bootstrap-daemon/pvcbootstrapd/lib/hooks.py

@@ -84,14 +84,13 @@ def run_hook_osd(config, targets, args):
         weight = args.get("weight", 1)
         ext_db_flag = args.get("ext_db", False)
         ext_db_ratio = args.get("ext_db_ratio", 0.05)
-        osd_count = args.get("osd_count", 1)
 
         logger.info(f"Creating OSD on node {node_name} device {device} weight {weight}")
 
         # Using a direct command on the target here is somewhat messy, but avoids many
         # complexities of determining a valid API listen address, etc.
         pvc_cmd_string = (
-            f"pvc storage osd add --yes {node_name} {device} --weight {weight} --osd-count {osd_count}"
+            f"pvc storage osd add --yes {node_name} {device} --weight {weight}"
         )
         if ext_db_flag:
             pvc_cmd_string = f"{pvc_cmd_string} --ext-db --ext-db-ratio {ext_db_ratio}"

bootstrap-daemon/pvcbootstrapd/lib/installer.py

@@ -66,7 +66,7 @@ def add_preseed(config, cspec_node, host_macaddr, system_drive_target):
 
     # We use the dhcp_address here to allow the listen_address to be 0.0.0.0
     rendered = template.render(
-        debrelease=config.get("repo_release"),
+        debrelease=cspec_node.get("config", {}).get("release"),
         debmirror=config.get("repo_mirror"),
         addpkglist=add_packages,
         filesystem=cspec_node.get("config", {}).get("filesystem"),

bootstrap-daemon/pvcbootstrapd/lib/redfish.py

@@ -196,19 +196,16 @@ class RedfishSession:
         logger.debug(f"POST payload: {payload}")
 
         response = requests.post(url, data=payload, headers=self.headers, verify=False)
-        logger.debug(f"Response: {response.status_code}")
 
-        if response.status_code in [201, 204]:
+        if response.status_code in [200, 201, 204]:
-            return {"response": "ok"}
-        elif response.status_code in [200]:
             try:
                 return response.json()
-            except Exception:
+            except json.decoder.JSONDecodeError as e:
                 return {"json_err": e}
         else:
             try:
                 rinfo = response.json()["error"]["@Message.ExtendedInfo"][0]
-            except Exception:
+            except json.decoder.JSONDecodeError:
                 logger.debug(response)
                 raise
 
@@ -579,7 +576,6 @@ def set_power_state(session, system_root, redfish_vendor, state):
     """
     Set the system power state to the desired state
     """
-    logger.debug(f"Calling set_power_state with {session}, {system_root}, {redfish_vendor}, {state}")
     state_values = {
         "default": {
             "on": "On",
@@ -719,8 +715,8 @@ def redfish_init(config, cspec, data):
     cspec_hostname = cspec_node["node"]["hostname"]
     cspec_fqdn = cspec_node["node"]["fqdn"]
 
-    logger.info("Waiting 30 seconds for system normalization")
+    logger.info("Waiting 60 seconds for system normalization")
-    sleep(30)
+    sleep(60)
 
     notifications.send_webhook(config, "begin", f"Cluster {cspec_cluster}: Beginning Redfish initialization of host {cspec_fqdn}")
 

install-pvcbootstrapd.sh

@@ -103,25 +103,17 @@ if [[ -z ${upstream_mirror} ]]; then
 fi
 echo
 
-echo "Please enter the default Debian release for new clusters (e.g. 'bullseye', 'bookworm'):"
-echo -n "[bookworm] > "
-read debian_release
-if [[ -z ${debian_release} ]]; then
-    debian_release="bookworm"
-fi
-echo
-
 echo "Proceeding with setup!"
 echo
 
 echo "Installing APT dependencies..."
 sudo apt-get update
-sudo apt-get install --yes vlan iptables dnsmasq redis python3 python3-pip python3-requests python3-git python3-ansible-runner python3-filelock python3-flask python3-paramiko python3-flask-restful python3-gevent python3-redis sqlite3 celery pxelinux syslinux-common live-build debootstrap uuid-runtime qemu-user-static apt-cacher-ng
+sudo apt-get install --yes vlan iptables dnsmasq redis python3 python3-pip python3-requests sqlite3 celery pxelinux syslinux-common live-build debootstrap uuid-runtime qemu-user-static apt-cacher-ng
 
 echo "Configuring apt-cacher-ng..."
 sudo systemctl enable --now apt-cacher-ng
 if ! grep -q ${upstream_mirror} /etc/apt-cacher-ng/backends_debian; then
-    echo "http://${upstream_mirror}" | sudo tee -a /etc/apt-cacher-ng/backends_debian &>/dev/null
+    echo "http://${upstream_mirror}" | sudo tee /etc/apt-cacher-ng/backends_debian &>/dev/null
     sudo systemctl restart apt-cacher-ng
 fi
 
@@ -138,7 +130,7 @@ echo "Installing pvcbootstrapd..."
 cp -a bootstrap-daemon ${root_directory}/pvcbootstrapd
 
 echo "Installing PIP dependencies..."
-sudo pip3 install --break-system-packages -r ${root_directory}/pvcbootstrapd/requirements.txt
+sudo pip3 install -r ${root_directory}/pvcbootstrapd/requirements.txt
 
 echo "Determining IP addresses..."
 bootstrap_address="$( awk -F'.' '{ print $1"."$2"."$3".1" }' <<<"${bootstrap_network}" )"
@@ -155,7 +147,6 @@ sed -i "s|BOOTSTRAP_DHCPEND|${bootstrap_dhcpend}|" ${root_directory}/pvcbootstra
 sed -i "s|GIT_REMOTE|${git_remote}|" ${root_directory}/pvcbootstrapd/pvcbootstrapd.yaml
 sed -i "s|GIT_BRANCH|${git_branch}|" ${root_directory}/pvcbootstrapd/pvcbootstrapd.yaml
 sed -i "s|UPSTREAM_MIRROR|${upstream_mirror}|" ${root_directory}/pvcbootstrapd/pvcbootstrapd.yaml
-sed -i "s|DEBIAN_RELEASE|${debian_release}|" ${root_directory}/pvcbootstrapd/pvcbootstrapd.yaml
 
 echo "Creating network configuration for interface ${bootstrap_interface} (is vLAN? ${is_bootstrap_interface_vlan})..."
 if [[ "${is_bootstrap_interface_vlan}" == "yes" ]]; then
@@ -266,12 +257,6 @@ case ${start_flag} in
     ;;
     *)
         echo
-        if [[ "${is_bootstrap_interface_vlan}" == "yes" ]]; then
-            sudo ifup vlan${bootstrap_vlan}
-        else
-            sudo ifup ${bootstrap_interface}
-        fi
-        sudo service apt-cacher-ng restart
         export PVCD_CONFIG_FILE="${root_directory}/pvcbootstrapd/pvcbootstrapd.yaml"
         ${root_directory}/pvcbootstrapd/pvcbootstrapd.py --init-only
     ;;