Update README badge order

As PVC 0.9.101+ no longer support buster or bullseye, remove them as
installer options.

README.md

@@ -1,12 +1,38 @@
-# PVC Live Node Installer
+<p align="center">
+<img alt="Logo banner" src="https://docs.parallelvirtualcluster.org/en/latest/images/pvc_logo_black.png"/>
+<br/><br/>
+<a href="https://www.parallelvirtualcluster.org"><img alt="Website" src="https://img.shields.io/badge/visit-website-blue"/></a>
+<a href="https://github.com/parallelvirtualcluster/pvc/releases"><img alt="Latest Release" src="https://img.shields.io/github/release-pre/parallelvirtualcluster/pvc"/></a>
+<a href="https://docs.parallelvirtualcluster.org/en/latest/?badge=latest"><img alt="Documentation Status" src="https://readthedocs.org/projects/parallelvirtualcluster/badge/?version=latest"/></a>
+<a href="https://github.com/parallelvirtualcluster/pvc"><img alt="License" src="https://img.shields.io/github/license/parallelvirtualcluster/pvc"/></a>
+<a href="https://github.com/psf/black"><img alt="Code style: Black" src="https://img.shields.io/badge/code%20style-black-000000.svg"/></a>
+</p>
 
-**NOTICE FOR GITHUB**: This repository is a read-only mirror of the PVC repositories from my personal GitLab instance. Pull requests submitted here will not be merged. Issues submitted here will however be treated as authoritative.
+## What is PVC?
+
+PVC is a Linux KVM-based hyperconverged infrastructure (HCI) virtualization cluster solution that is fully Free Software, scalable, redundant, self-healing, self-managing, and designed for administrator simplicity. It is an alternative to other HCI solutions such as Ganeti, Harvester, Nutanix, and VMWare, as well as to other common virtualization stacks such as ProxMox and OpenStack.
+
+PVC is a complete HCI solution, built from well-known and well-trusted Free Software tools, to assist an administrator in creating and managing a cluster of servers to run virtual machines, as well as self-managing several important aspects including storage failover, node failure and recovery, virtual machine failure and recovery, and network plumbing. It is designed to act consistently, reliably, and unobtrusively, letting the administrator concentrate on more important things.
+
+PVC is highly scalable. From a minimum (production) node count of 3, up to 12 or more, and supporting many dozens of VMs, PVC scales along with your workload and requirements. Deploy a cluster once and grow it as your needs expand.
+
+As a consequence of its features, PVC makes administrating very high-uptime VMs extremely easy, featuring VM live migration, built-in always-enabled shared storage with transparent multi-node replication, and consistent network plumbing throughout the cluster. Nodes can also be seamlessly removed from or added to service, with zero VM downtime, to facilitate maintenance, upgrades, or other work.
+
+PVC also features an optional, fully customizable VM provisioning framework, designed to automate and simplify VM deployments using custom provisioning profiles, scripts, and CloudInit userdata API support.
+
+Installation of PVC is accomplished by two main components: a [Node installer ISO](https://github.com/parallelvirtualcluster/pvc-installer) which creates on-demand installer ISOs, and an [Ansible role framework](https://github.com/parallelvirtualcluster/pvc-ansible) to configure, bootstrap, and administrate the nodes. Installation can also be fully automated with a companion [cluster bootstrapping system](https://github.com/parallelvirtualcluster/pvc-bootstrap). Once up, the cluster is managed via an HTTP REST API, accessible via a Python Click CLI client ~~or WebUI~~ (eventually).
+
+Just give it physical servers, and it will run your VMs without you having to think about it, all in just an hour or two of setup time.
+
+More information about PVC, its motivations, the hardware requirements, and setting up and managing a cluster [can be found over at our docs page](https://docs.parallelvirtualcluster.org).
+
+# PVC Live Node Installer
 
 This repository contains the generator and configurations for the PVC Live Node Installer ISO. This ISO provides a quick and convenient way to install a PVC base system to a physical server, ready to then be provisioned using the [PVC Ansible](https://github.com/parallelvirtualcluster/pvc-ansible) configuration framework. Part of the [Parallel Virtual Cluster system](https://github.com/parallelvirtualcluster/pvc).
 
-## Using the PVC Installer
+# Using the PVC Installer
 
-### Preparing
+## Preparing
 
 1. Run `./buildiso.sh` from the root of the repository. This will pull down the Debian LiveCD image, extract it, debootstrap a fresh install environment, copy in the configurations, generate a squashfs, then finally generate an ISO file.
 
@@ -16,28 +42,12 @@ Note that artifacts of the build (the LiveCD ISO, debootstrap directory, and squ
 
 3. Boot the server from the ISO, ideally in UEFI mode.
 
-### Booting
+## Booting
 
 The built ISO can be booted in either BIOS (traditional ISOLinux) or UEFI (Grub2) modes. It is strongly recommended to use the latter if the system supports it for maximum flexibility.
 
-### Installing
+## Installing
 
 The installer script will ask several questions to configure the bare minimum system needed for [`pvc-ansible`](https://github.com/parallelvirtualcluster/pvc-ansible) to configure the node.
 
 Follow the prompts carefully; if you make a mistake, you can ^C to cancel the installer, then re-run via `/install.sh` from the resulting root shell.
-
-## License
-
-Copyright (C) 2018-2021  Joshua M. Boniface <joshua@boniface.me>
-
-This repository, and all contained files, is free software: you can
-redistribute it and/or modify it under the terms of the GNU General
-Public License as published by the Free Software Foundation, version 3.
-
-This program is distributed in the hope that it will be useful,
-but WITHOUT ANY WARRANTY; without even the implied warranty of
-MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License
-along with this program.  If not, see <https://www.gnu.org/licenses/>.

buildiso.sh

@@ -189,10 +189,12 @@ mkdir -p config/includes.chroot/etc/initramfs-tools
 cp ../../templates/modules config/includes.chroot/etc/initramfs-tools/modules || fail "Failed to copy critical template file"
 echo "done."
 
-# Install install.sh script
+# Install install.sh and detect.py scripts
 echo -n "Copying PVC node installer script template... "
 cp ../../templates/install.sh config/includes.chroot/install.sh || fail "Failed to copy critical template file"
 chmod +x config/includes.chroot/install.sh
+cp ../../templates/detect.py config/includes.chroot/detect.py || fail "Failed to copy critical template file"
+chmod +x config/includes.chroot/detect.py
 echo "done."
 
 # Customize install.sh script

detect.py

@@ -0,0 +1 @@
+templates/detect.py

templates/detect.py

@@ -0,0 +1,234 @@
+#!/usr/bin/env python3
+
+import subprocess
+import sys
+from json import loads as loads
+from re import match as re_match
+from re import search as re_search
+from re import sub as re_sub
+from shlex import split as shlex_split
+
+#
+# Run a local OS command via shell
+#
+def run_os_command(command_string, background=False, environment=None, timeout=None):
+    if not isinstance(command_string, list):
+        command = shlex_split(command_string)
+    else:
+        command = command_string
+
+    if background:
+
+        def runcmd():
+            try:
+                subprocess.run(
+                    command,
+                    env=environment,
+                    timeout=timeout,
+                    stdout=subprocess.PIPE,
+                    stderr=subprocess.PIPE,
+                )
+            except subprocess.TimeoutExpired:
+                pass
+
+        thread = Thread(target=runcmd, args=())
+        thread.start()
+        return 0, None, None
+    else:
+        try:
+            command_output = subprocess.run(
+                command,
+                env=environment,
+                timeout=timeout,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+            )
+            retcode = command_output.returncode
+        except subprocess.TimeoutExpired:
+            retcode = 128
+        except Exception:
+            retcode = 255
+
+        try:
+            stdout = command_output.stdout.decode("ascii")
+        except Exception:
+            stdout = ""
+        try:
+            stderr = command_output.stderr.decode("ascii")
+        except Exception:
+            stderr = ""
+        return retcode, stdout, stderr
+
+def get_detect_device_lsscsi(detect_string):
+    """
+    Parses a "detect:" string into a normalized block device path using lsscsi.
+
+    A detect string is formatted "detect:<NAME>:<SIZE>:<ID>", where
+    NAME is some unique identifier in lsscsi, SIZE is a human-readable
+    size value to within +/- 3% of the real size of the device, and
+    ID is the Nth (0-indexed) matching entry of that NAME and SIZE.
+    """
+    _, name, size, idd = detect_string.split(":")
+    if _ != "detect":
+        return None
+
+    retcode, stdout, stderr = run_os_command("lsscsi -s")
+    if retcode:
+        print(f"Failed to run lsscsi: {stderr}")
+        return None
+
+    # Get valid lines
+    lsscsi_lines_raw = stdout.split("\n")
+    lsscsi_lines = list()
+    for line in lsscsi_lines_raw:
+        if not line:
+            continue
+        split_line = line.split()
+        if split_line[1] != "disk":
+            continue
+        lsscsi_lines.append(line)
+
+    # Handle size determination (+/- 3%)
+    lsscsi_sizes = set()
+    for line in lsscsi_lines:
+        lsscsi_sizes.add(split_line[-1])
+    for l_size in lsscsi_sizes:
+        b_size = float(re_sub(r"\D.", "", size))
+        t_size = float(re_sub(r"\D.", "", l_size))
+
+        plusthreepct = t_size * 1.03
+        minusthreepct = t_size * 0.97
+
+        if b_size > minusthreepct and b_size < plusthreepct:
+            size = l_size
+            break
+
+    blockdev = None
+    matches = list()
+    for idx, line in enumerate(lsscsi_lines):
+        # Skip non-disk entries
+        if line.split()[1] != "disk":
+            continue
+        # Skip if name is not contained in the line (case-insensitive)
+        if name.lower() not in line.lower():
+            continue
+        # Skip if the size does not match
+        if size != line.split()[-1]:
+            continue
+        # Get our blockdev and append to the list
+        matches.append(line.split()[-2])
+
+    blockdev = None
+    # Find the blockdev at index {idd}
+    for idx, _blockdev in enumerate(matches):
+        if int(idx) == int(idd):
+            blockdev = _blockdev
+            break
+
+    return blockdev
+
+def get_detect_device_nvme(detect_string):
+    """
+    Parses a "detect:" string into a normalized block device path using nvme.
+
+    A detect string is formatted "detect:<NAME>:<SIZE>:<ID>", where
+    NAME is some unique identifier in lsscsi, SIZE is a human-readable
+    size value to within +/- 3% of the real size of the device, and
+    ID is the Nth (0-indexed) matching entry of that NAME and SIZE.
+    """
+
+    unit_map = {
+        'kB': 1000,
+        'MB': 1000*1000,
+        'GB': 1000*1000*1000,
+        'TB': 1000*1000*1000*1000,
+        'PB': 1000*1000*1000*1000*1000,
+    }
+
+    _, name, _size, idd = detect_string.split(":")
+    if _ != "detect":
+        return None
+
+    size_re = re_search(r'([\d.]+)([kKMGTP]B)', _size)
+    size_val = float(size_re.group(1))
+    size_unit = size_re.group(2)
+    size_bytes = int(size_val * unit_map[size_unit])
+
+    retcode, stdout, stderr = run_os_command("nvme list --output-format json")
+    if retcode:
+        print(f"Failed to run nvme: {stderr}")
+        return None
+
+    # Parse the output with json
+    nvme_data = loads(stdout).get('Devices', list())
+
+    # Handle size determination (+/- 3%)
+    size = None
+    nvme_sizes = set()
+    for entry in nvme_data:
+        nvme_sizes.add(entry['PhysicalSize'])
+    for l_size in nvme_sizes:
+        plusthreepct = size_bytes * 1.03
+        minusthreepct = size_bytes * 0.97
+
+        if l_size > minusthreepct and l_size < plusthreepct:
+            size = l_size
+            break
+    if size is None:
+        return None
+
+    blockdev = None
+    matches = list()
+    for entry in nvme_data:
+        # Skip if name is not contained in the line (case-insensitive)
+        if name.lower() not in entry['ModelNumber'].lower():
+            continue
+        # Skip if the size does not match
+        if size != entry['PhysicalSize']:
+            continue
+        # Get our blockdev and append to the list
+        matches.append(entry['DevicePath'])
+
+    blockdev = None
+    # Find the blockdev at index {idd}
+    for idx, _blockdev in enumerate(matches):
+        if int(idx) == int(idd):
+            blockdev = _blockdev
+            break
+
+    return blockdev
+
+
+def get_detect_device(detect_string):
+    """
+    Parses a "detect:" string into a normalized block device path.
+
+    First tries to parse using "lsscsi" (get_detect_device_lsscsi). If this returns an invalid
+    block device name, then try to parse using "nvme" (get_detect_device_nvme). This works around
+    issues with more recent devices (e.g. the Dell R6615 series) not properly reporting block
+    device paths for NVMe devices with "lsscsi".
+    """
+
+    device = get_detect_device_lsscsi(detect_string)
+    if device is None or not re_match(r'^/dev', device):
+        device = get_detect_device_nvme(detect_string)
+
+    if device is not None and re_match(r'^/dev', device):
+        return device
+    else:
+        return None
+
+
+try:
+    detect_string = sys.argv[1]
+except IndexError:
+    print("Please specify a detect: string")
+    exit(1)
+
+blockdev = get_detect_device(detect_string)
+if blockdev is not None:
+    print(blockdev)
+    exit(0)
+else:
+    exit(1)
+

templates/install.sh

@@ -72,7 +72,7 @@ target_deploy_user="XXDEPLOYUSERXX"
 supported_filesystems="ext4 xfs"
 default_filesystem="ext4"
 
-supported_debrelease="buster bullseye bookworm"
+supported_debrelease="bookworm"
 default_debrelease="bookworm"
 default_debmirror="http://ftp.debian.org/debian"
 
@@ -233,56 +233,11 @@ seed_config() {
             target_disk="$( realpath ${target_disk} )"
         ;;
         detect:*)
-            # Read the detect string into separate variables
-            # A detect string is formated thusly:
-            #   detect:<Controller-or-Model-Name>:<Capacity-in-human-units><0-indexed-ID>
-            # For example:
-            #   detect:INTEL:800GB:1
-            #   detect:DELLBOSS:240GB:0
-            #   detect:PERC H330 Mini:200GB:0
-            echo "Attempting to find disk for detect string '${o_target_disk}'"
-            IFS=: read detect b_name b_size b_id <<<"${target_disk}"
-            # Get the lsscsi output (exclude NVMe)
-            lsscsi_data_all="$( lsscsi -s -N )"
-            # Get the available sizes, and match to within +/- 3%
-            lsscsi_sizes=( $( awk '{ print $NF }' <<<"${lsscsi_data_all}" | sort | uniq ) )
-            # For each size...
-            for size in ${lsscsi_sizes[@]}; do
-                # Get whether we match +3% and -3% sizes to handle human -> real deltas
-                # The break below is pretty safe. I can think of no two classes of disks
-                # where the difference is within 3% of each other. Even the common
-                # 120GB -> 128GB and 240GB -> 256GB size deltas are well outside of 3%,
-                # so this should be safe in all cases.
-                # We use Python for this due to BASH's problematic handling of floating-
-                # point numbers.
-                is_match="$(
-                    python <<EOF
-from re import sub
-try:
-    b_size = float(sub(r'\D.','','${b_size}'))
-    t_size = float(sub(r'\D.','','${size}'))
-except ValueError:
-    exit(0)
-plusthreepct = t_size * 1.03
-minusthreepct = t_size * 0.97
-if b_size > minusthreepct and b_size < plusthreepct:
-    print("match")
-EOF
-                )"
-                # If we do, this size is our actual block size, not what was specified
-                if [[ -n ${is_match} ]]; then
-                    b_size=${size}
-                    break
-                fi
-            done
-            # Search for the b_name first
-            lsscsi_data_name="$( grep --color=none -Fiw "${b_name}" <<<"${lsscsi_data_all}" )"
-            # Search for the b_blocks second
-            lsscsi_data_name_size="$( grep --color=none -Fiw "${b_size}" <<<"${lsscsi_data_name}" )"
-            # Read the /dev/X results into an array
-            lsscsi_filtered=( $( awk '{ print $(NF-1) }' <<<"${lsscsi_data_name_size}" ) )
-            # Get the b_id-th entry
-            target_disk="${lsscsi_filtered[${b_id}]}"
+            # Use the detect.py parser to get the target disk from the detect string
+            target_disk="$( /detect.py ${target_disk} )"
+        ;;
+        *)
+            target_disk=""
         ;;
     esac
 

templates/installer.list.chroot

@@ -1 +1 @@
-live-task-standard live-tools live-boot live-boot-initramfs-tools psmisc mdadm lvm2 parted gdisk dosfstools debootstrap sipcalc vim ca-certificates vlan tftp-hpa curl ipmitool lsscsi hdparm
+live-task-standard live-tools live-boot live-boot-initramfs-tools psmisc mdadm lvm2 parted gdisk dosfstools debootstrap sipcalc vim ca-certificates vlan tftp-hpa curl ipmitool nvme-cli jq lsscsi hdparm