parallelvirtualcluster
/
pvc
1
0
Fork 0
Code Issues 15 Pull Requests Projects Releases 106 Wiki Activity
pvc/docs/architecture/daemon.md

3.3 KiB
Raw Blame History

PVC Node Daemon architecture

The PVC Node Daemon is the heart of the PVC system and runs on each node to manage the state of the node and its configured resources. The daemon connects directly to the Zookeeper cluster for coordination and state.

The node daemon is build using Python 3.X and is packaged in the Debian package pvc-daemon.

Configuration of the daemon is documented in the manual, however it is recommended to use the Ansible configuration interface to configure the PVC system for you from scratch.

Overall architecture

The PVC daemon is object-oriented - each cluster resource is represented by an Object, which is then present on each node in the cluster. This allows state changes to be reflected across the entire cluster should their data change.

During startup, the system scans the Zookeeper database and sets up the required objects. The database is then watched in real-time for additional changes to the database information.

Startup sequence

The daemon startup sequence is documented below. The main daemon entry-point is Daemon.py inside the pvcnoded folder, which is called from the pvcnoded.py stub file.

  1. The configuration is read from /etc/pvc/pvcnoded.yaml and the configuration object set up.

  2. Any required filesystem directories, mostly dynamic directories, are created.

  3. The logger is set up. If file logging is enabled, this is the state when the first log messages are written.

  4. Host networking is configured based on the pvcnoded.yaml configuration file. In a normal cluster, this is the point where the node will become reachable on the network as all networking is handled by the PVC node daemon.

  5. Sysctl tweaks are applied to the host system, to enable routing/forwarding between nodes via the host.

  6. The node determines its coordinator state and starts the required daemons if applicable. In a normal cluster, this is the point where the dependent services such as Zookeeper, FRR, and Ceph become available. After this step, the daemon waits 5 seconds before proceeding to give these daemons a chance to start up.

  7. The daemon connects to the Zookeeper cluster and starts its listener. If the Zookeeper cluster is unavailable, it will wait some time before abandoning the attempt and starting again from step 1.

  8. Termination handling/cleanup is configured.

  9. The node checks if it is already present in the Zookeeper cluster; if not, it will add itself to the database. Initial static options are also updated in the database here. The daemon state transitions from stop to init.

  10. The node checks if Libvirt is accessible.

  11. The node starts up the NFT firewall if applicable and configures the base rule-set.

  12. The node ensures that dnsmasq is stopped (legacy check, might be safe to remove eventually).

  13. The node begins setting up the object representations of resources, in order:

    a. Node entries

    b. Network entries, creating client networks and starting them as required.

    c. Domain (VM) entries, starting up the VMs as required.

    d. Ceph storage entries (OSDs, Pools, Volumes, Snapshots).

  14. The node activates its keepalived timer and begins sending keepalive updates to the cluster. The daemon state transitions from init to run and the system has started fully.