From 67704198e48fb8d67fad23ab6fe1f7e03613ce7c Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Sun, 28 Jul 2019 18:22:33 -0400 Subject: [PATCH] Add basic architecture docs --- docs/architecture/api.md | 7 ++++++ docs/architecture/cli.md | 7 ++++++ docs/architecture/daemon.md | 50 +++++++++++++++++++++++++++++++++++++ 3 files changed, 64 insertions(+) diff --git a/docs/architecture/api.md b/docs/architecture/api.md index e69de29b..8b4e0805 100644 --- a/docs/architecture/api.md +++ b/docs/architecture/api.md @@ -0,0 +1,7 @@ +# PVC API architecture + +The PVC API is a standalone client application for PVC. It interfaces directly with the Zookeeper database to manage state. + +The API is built using Flask and is packaged in the Debian package `pvc-client-api`. The API depends on the common client functions of the `pvc-client-common` package as does the CLI client. + +Details of the API interface can be found in [the manual](/manuals/api). diff --git a/docs/architecture/cli.md b/docs/architecture/cli.md index e69de29b..53d39407 100644 --- a/docs/architecture/cli.md +++ b/docs/architecture/cli.md @@ -0,0 +1,7 @@ +# PVC CLI architecture + +The PVC CLI is a standalone client application for PVC. It interfaces directly with the Zookeeper database to manage state. + +The CLI is build using Click and is packaged in the Debian package `pvc-client-cli`. The CLI depends on the common client functions of the `pvc-client-common` package. + +The CLI is self-documenting, however [the manual](/manuals/cli) details the required configuration. diff --git a/docs/architecture/daemon.md b/docs/architecture/daemon.md index e69de29b..1d6e40a8 100644 --- a/docs/architecture/daemon.md +++ b/docs/architecture/daemon.md @@ -0,0 +1,50 @@ +# PVC Node Daemon architecture + +The PVC Node Daemon is the heart of the PVC system and runs on each node to manage the state of the node and its configured resources. The daemon connects directly to the Zookeeper cluster for coordination and state. + +The node daemon is build using Python 3.X and is packaged in the Debian package `pvc-daemon`. + +Configuration of the daemon is documented in [the manual](/manuals/daemon), however it is recommended to use the [Ansible configuration interface](/manuals/ansible) to configure the PVC system for you from scratch. + +## Overall architecture + +The PVC daemon is object-oriented - each cluster resource is represented by an Object, which is then present on each node in the cluster. This allows state changes to be reflected across the entire cluster should their data change. + +During startup, the system scans the Zookeeper database and sets up the required objects. The database is then watched in real-time for additional changes to the database information. + +## Startup sequence + +The daemon startup sequence is documented below. The main daemon entrypoint is `Daemon.py` inside the `pvcd` folder, which is called from the `pvcd.py` stub file. + +0. The configuration is read from `/etc/pvc/pvcd.yaml` and the configuration object set up. + +0. Any required filesystem directories, mostly dynamic directories, are created. + +0. The logger is set up. If file logging is enabled, this is the state when the first log messages are written. + +0. Host networking is configured based on the `pvcd.yaml` configuration file. In a normal cluster, this is the point where the node will become reachable on the network as all networking is handled by the PVC node daemon. + +0. Sysctl tweaks are applied to the host system, to enable routing/forwarding between nodes via the host. + +0. The node determines its coordinator state and starts the required daemons if applicable. In a normal cluster, this is the point where the dependent services such as Zookeeper, FRR, and Ceph become available. After this step, the daemon waits 5 seconds before proceeding to give these daemons a chance to start up. + +0. The daemon connects to the Zookeeper cluster and starts its listener. If the Zookeeper cluster is unavailable, it will wait some time before abandoning the attempt and starting again from step 1. + +0. Termination handling/cleanup is configured. + +0. The node checks if it is already present in the Zookeeper cluster; if not, it will add itself to the database. Initial static options are also updated in the database here. The daemon state transitions from `stop` to `init`. + +0. The node checks if Libvirt is accessible. + +0. The node starts up the NFT firewall if applicable and configures the base ruleset. + +0. The node ensures that `dnsmasq` is stopped (legacy check, might be safe to remove eventually). + +0. The node begins setting up the object representations of resources, in order: + + a. Node entries + b. Network entries, creating client networks and starting them as required. + c. Domain (VM) entries, starting up the VMs as required. + d. Ceph storage entries (OSDs, Pools, Volumes, Snapshots). + +0. The node activates its keepalived timer and begins sending keepalive updates to the cluster. The daemon state transitions from `init` to `run` and the system has started fully.