pvc/docs/architecture/provisioner.md

# PVC Provisioner API architecture

The PVC Provisioner API is a standalone client application for PVC. It interfaces directly with the Zookeeper database to manage state, and with the Patroni PostgreSQL database to store configuration details.

The Provisioner is built using Flask and is packaged in the Debian package `pvc-client-provisioner`. The Provisioner depends on the common client functions of the `pvc-client-common` package as does the CLI client.

Details of the Provisioner API interface can be found in [the manual](/manuals/provisioner).

## Purpose

The purpose of the Provisioner API is to provide a convenient way for administrators to automate the creation of new virtual machines on the PVC cluster.

The Provisioner allows the administrator to create "templates", a unified set of configurations, which VMs can then use. These templates configure the VM resources (memory, disk, metadata), VM networks, and VM disks separately, allowing the administrator to specify very granular and dynamic configurations for new virtual machines.

Upon triggering a new VM creation, the provisioner also has facilities to create new virtual machines in three main ways:

1. Via cloning an existing RBD disk image, then performing optional post-clone actions on the volume(s).
2. Via booting an installer ISO image, stored as an RBD disk image.
3. Via custom provisioning scripts provided by the administrator.

The first option allows administrators to quickly create new virtual machines based on an existing image, either uploaded by the administrator or created from an existing virtual machine.

The second option allows administrators to install arbitrary operating systems via ISO images, which are uploaded by the administrator. Usually, auto-configuring/kickstarted ISOs are ideal for this purpose.

The third method provides extreme flexibility in setting up Unix-based virtual machines, as standard, arbitrary Python scripts can be provided by the administrator, allowing the system to automatically install and configure the VM exactly to the specifications they want. Furthermore, PVC includes integrated support for `cloud-init` inside VMs, for maximum flexibility in post-install configurations.

## System Templates

The PVC Provisioner has three categories of templates to specify the resources allocated to the virtual machine. They are: System Templates, Network Templates, and Disk Templates.

### System Templates

System templates specify the basic resources of the virtual machine: vCPUs, memory, and configuration metadata (e.g. serial/VNC/Spice consoles, migration methods, additional devices, etc.). PVC VMs use the Libvirt XML configuration format, so these templates specify the required values in the created VM configuration file. When querying details, the API will return JSON representations of the configuration, which are used here for examples.

vCPU and memory configurations are specified explicitly. For instance, a template might be called `small_webserver` and specify 2 `vcpus` and 2GB (always specified in MB) of `memory`:

```
"small_webserver": {
    "vcpus": 2,
    "memory": 2048
}
```

Additional, non-default configuration values can also be specified. For instance, one can specify the `console_type` and additional values for this:

```
"serial_server": {
    "vcpus": 1,
    "memory": 1024,
    "console_type": "serial",
    "serial_device": "auto",
    "serial_logfile": "/var/log/libvirt/VMNAME.xml"
}
```

The serial logfile can also be "auto" - this enables the PVC `vm log` functionality. The literal string `VMNAME` in this value will be replaced with the virtual machine name.

Configuration for a VNC console is similar:

```
"vnc_server": {
    "vcpus": 4,
    "memory": 4096,
    "console_type": "vnc",
    "vnc_port": "auto",
    "vnc_listen": "0.0.0.0"
}
```

Additional PVC metadata can be configured in these templates as well. For example:

```
"limited_server": {
    "vcpus": 1,
    "memory": 1024,
    "pvc_node_limit": "pvchv1,pvchv2",
    "pvc_node_selector": "vms",
    "pvc_node_autostart": "True"
}
```

### Network Templates

Network template specify which PVC networks the virtual machine is active in, as well as the method used to calculate MAC addresses for VM interfaces. Networks are specified by their VNI ID or description within PVC.

For example, a system with a single interface and autogenerated MAC address:

```
"single_net_auto": {
    "networks": {
        "client-net-01"
    },
    "macaddr": "auto"
}
```

In some cases, it may be useful for the administrator to specify a static MAC address pattern for a set of VMs, for instance if they must get consistent DHCP reservations between rebuilds. The `macaddr` field can contain templated MAC address values, in the format `AA:AA:AA:XX:XX:YZ`. In this format, `A` represents the OUI portion (usually the KVM default of `52:54:00`), `X` represents a static prefix specified by the administrator, `Y` represents the VM number/ID, which is autofilled by PVC based on the VM name (or set to 0 for numberless VM names), and `Z` represents the incremental interface ID within the VM. Therefore, to configure a static MAC address, the template could be:

```
"double_net_templated_mac": {
    "networks": {
        "5927",
        "5928"
    },
    "macaddr": "52:54:00:00:01:YZ"
}
```

Note the literal `Y` and `Z` characters in the value. This will expand to the following MAC addresses for a VM called `web3`, which would have VM number/ID `3`:

* Network `5927`: `52:54:00:00:01:30`
* Network `5928`: `52:54:00:00:01:31`

Similarly, a VM called `accounting`, which would have the implied VM number/ID `0`, would expand to:

* Network `5927`: `52:54:00:00:01:00`
* Network `5928`: `52:54:00:00:01:01`

Note that these automated values do not overflow; therefore, PVC does not support templated MAC addresses for >9 numbered VMs (e.g. web1-web9) within a single template, or for >10 networks within each VM. For such cases, static MAC addresses are far less useful anyways and the administrator must consider this. Also note that assigning the same static MAC template to overlapping numbered VMs (e.g. web1-web3 and mail1-mail3) will result in MAC address conflicts within a given client network and must be avoided.

### Disk Templates

Disk templates specify the disk layout, including filesystem and mountpoint for scripted deployments, for the VM. Disks are specified by their virtual disk name within the VM, and sizes are always specified in GB. For a basic, unmanaged VM with a single disk, the template may be as simple as:

```
"single_disk": {
    "vda": {
        "size": 20
    }
}
```

For a scripted VM, two additional values should be specified: the filesystem, which must be a valid filesystem usable by the VM, and the mountpoint:

```
"scripted_single_disk": {
    "vda": {
        "size": 20,
        "filesystem": "ext4",
        "mountpoint": "/"
    }
}
```

Note that these values are technically optional: if unspecified, PVC will not create a filesystem on the device nor attempt to mount it during the scripted configuration steps. This allows administrators to attach unallocated block devices to scripted VMs as well as the main filesystem(s) that the OS will be installed on.

More complicated disk templates are also possible by specifying incrementing `vdX` devices in the VM, for example:

```
scripted_multi_disk_srv": {
    "vda": {
        "size": 4,
        "filesystem": "ext4",
        "mountpoint": "/"
    },
    "vdb": {
        "size": 8,
        "filesystem": "ext4",
        "mountpoint": "/var"
    },
    "vdc": {
        "size": 40,
        "filesystem": "xfs",
        "mountpoint": "/srv"
    }
}
```

## System Definitions

At the next level above configuraton templates, system definitions provide a way to group templates together to provide standard definitions for classes of virtual machines. This definition can then be specified, or autodetected, instead of manually specifying the 3 resource templates on VM creation, as well as specify additional provisioning metadata including the install method and provisioning script template, if applicable.

It is generally a good idea to make use of system definitions, rather than manually specifying all values at install time, in order to reduce the possibility of administrative mistakes in provisioning new VMs. They are however optional: all the required configuration information may be specified explicitly by the administrator when creating a new VM, instead of using a definition.

The `autostart` option specifies to PVC whether the VM should be automatically started after the provisioning sequence completes. It defaults to "True", and this can be avoided by setting this value to "False", requiring the administrator to manually start the VM using PVC commands afterwards.

For example, here are several VM definitions using some of the example system templates above:

```
"webX": {
    "templates": {
        "system": "small_webserver",
        "network": "double_net_templated_mac",
        "disk": "scripted_single_disk"
    },
    "provisioner": {
        "method": "script",
        "script": {
            "name": "basic-pvc-debian",
            "arguments": {
                "keyword_argument": "my_value",
                "another_argument": "some_value"
            }
        }
    },
    "autostart": "False"
}
```

```
"windows-10": {
    "templates": {
        "system": "vnc_server",
        "network": "single_net_auto",
        "disk": "single_disk"
    },
    "provisioner": {
        "method": "iso",
        "iso": "installers/windows-10-installer-201910"
    }
}
```

```
"cloned_mail": {
    "templates": {
        "system": "limited_server",
        "network": "single_net_auto",
        "disk": "scripted_multi_disk_srv"
    }
    "provisioner": {
        "method": "clone",
        "clone": {
            "source_disks": {
                "vda": "templates/mailX_root",
                "vdb": "templates/mailX_var"
            },
        }
    }
}
```

### Scripted installs

Scripted installs specify the `script` `method` in their `provisioner` metadata section. The second value, named `script`, specifies the provisioner script name which must exist, as well as any additional arguments the administrator may wish to pass to the script functions. Provisioner scripts are explained in detail in a subsequent section.

### ISO installs

ISO installs specify the `iso` `method` in their `provisioner` metadata section. The second value, named `iso`, specifies the RBD image containing the ISO which must exist having been previously uploaded by the administrator. The VM is booted immediately after basic configuration, and control is passed to the ISO to perform any installation steps; no other configuration occurrs from the PVC side.

### Clone installs

Clone installs specify the`clone` `method` in their `provisioner` metadata section. The second value, named `clone`, specifies the target virtual devices and their corresponding source RBD images, as well as the provisioner script to run after cloning.

Within the `clone` section, the `source_disks` section specifies a list of disks to clone as well as the target device. These target devices must align with disks from the Disk template, to map the source volumes to the new volumes for the VM. For example, if the Disk template specifies `vda` as a disk with `mountpoint` `/` (the `size` and `filesystem` will be ignored), and the `source_disks` value for `vda` maps to the RBD image `templates/root`, the provisioner will clone the RBD image `templates/root` to a new volume for the VM named, for example, `vms/VMNAME_vda`. If there are additional disks specified in the Disk template that are not specified in the `source_disks` list, they will be created as normal.

PVC performs no actions to a clone deployment aside from creating the additional disks mentioned above, if applicable. All configuration of the clone is the responsibility of the administrator. The cloud-init support from the `script` install method can be useful in this case to create a "golden image" that will then use cloud-init to configure itself on first boot.

## Provisioning scripts

The PVC provisioner provides a scripting framework in order to automate VM installation. This is generally the most useful with UNIX-like systems which can be installed over the network via shell scripts. For instance, the script might install a Debian VM using `debootstrap`.

Provisioner scripts are written in Python 3 and are called in a standardized way during the provisioning sequence. A single function called `install` is called during the provisioning sequence, performing OS installation, after which the system is booted.

The flow of the provisioning sequence is as follows:

1. The provisioner creates the required disks.
1. The provisioner creates a temporary directory on the local system (often the primary hypervisor, but the provisioner may be run in a dedicated virtual machine).
1. The provisioner maps the VM's RBD volumes on the local system.
1. The provisioner mounts the RBD volumes at their `mountpoint` under the temporary directory, along with several temporary virtual filesystems bind-mounted from the local system.
1. The provisioner calls the `install` function of the provisioner script and waits for it to finish execution.
1. The provisioner creates any cloud-init configuration files specified.
1. The provisioner unmounts the RBD volumes and temporary virtual filesystems (cleanup).
1. The provisioner unmaps the RBD volumes from the local system (cleanup).
1. The provisioner defines the new VM in PVC and, optionally, starts it.

*A WARNING*: It's important to remember that these provisioning scripts will run with the same privileges as the provisioner API daemon (usually root) on the system running the daemon. THIS MAY POSE A SECURITY RISK. However, the intent is that administrators of the cluster are the only ones allowed to specify these scripts, and that they check them thoroughly when adding them to the system as well as limit access to the provisioning API to trusted sources. If neither of these conditions are possible, for instance if arbitrary users must specify custom scripts without administrator oversight, then the PVC provisoner may not be ideal, and administrators are encouraged to implement their own custom provisioning engine.

### `install` function

The `install` function is the main entrypoing for a provisioning script, and is the only part of the script that is explicitly called. The provisioner calls this function after setting up the temporary install directory and mounting the volumes. Thus, this script can then perform any sort of tasks required in the VM to install it, and then finishes.

This function is passed a number of keyword arguments that it can then use during installation, described below, as well as any keyword arguments passed via optional arguments to the script.

###### `vm_name`

The `vm_name` keyword argument contains the full name of the new VM.

###### `vm_id`

The `vm_id` keyword argument contains the VM identifier (the last numeral of the VM name, or `0` for a VM that does not end in a numeral).

###### `temporary_directory`

The `temporary_directory` keyword argument contains the path to the temporary directory on which the new VM's disks are mounted. The function *must* perform any installation steps to/under this directory.

###### `disks`

The `disks` keyword argument contains a Python list of the configured disks, as dictionaries of values as specified in the Disk template. The function *may* use these values as appropriate, for instance to specify an `/etc/fstab`.

###### `networks`

The `networks` keyword argument contains a Python list of the configured networks, as dictionaries of values as specified in the Network template. The function *may* use these values as appropriate, for instance to write an `/etc/network/interfaces` file.

## Cloud-Init

PVC contains full support for cloud-init, a tool to automatically configure VMs on first boot from a defined set of metadata. The PVC provisioner includes a cloud-init metadata server that the administrator can use to provide information to running VMs.

### Configuring Cloud-Init in VMs

The PVC provisioner sequence makes no special considerations for cloud-init; the administrator must handle the installation of the cloud-init packages as well as any tweaks to the cloud.cfg file in the installation script. The provisioner does however listen on the standard EC2 interface at `http://169.254.169.254/latest/` from within the VM to provision user data.

### Configuring user-data

The PVC provisioner supports managing cloud-init user-data from within it. This data will be delivered to VMs based on the configuration options.