Compare commits

..

280 Commits

Author SHA1 Message Date
92feeefd26 Bump version to 0.9.59 2022-11-15 15:50:15 -05:00
38d63d9837 Flip behaviour of memory selectors
It didn't make any sense to me for mem(prov) to be the default selector,
since this has too many caveats versus mem(free). Switch to using
mem(free) as the default (i.e. "mem") and make memprov the alternative.
2022-11-15 15:45:59 -05:00
095bcb2373 Bump version to 0.9.58 2022-11-07 12:27:48 -05:00
91e450f399 Remove extra lower() call where not needed 2022-11-07 12:26:50 -05:00
79eb994a5e Ensure equality of none and None for selector 2022-11-07 11:59:53 -05:00
d65f512897 Bump version to 0.9.57 2022-11-06 01:39:50 -04:00
8af7189dd0 Add module tag for daemon lib 2022-11-04 03:47:18 -04:00
ea7a4b2b85 Make benchmarker function as a module
1. Move the test_matrix, volume name, and size to module-level variables
so they can be accessed externally if this is imported.
2. Separate the volume creation and volume cleanup into functions.
3. Separate the individual benchmark runs into a function.

This should enable easier calling of the various subcomponents
externally, e.g. for external benchmark scripts.
2022-11-03 21:33:32 -04:00
59f97ebbfb Better handle invalid nets in VMs
1. Error out when trying to add a new network to a VM if the network
doesn't exist on the cluster.
2. When showing the VM list, only show invalid networks in red, not the
whole list.
2022-11-01 10:24:24 -04:00
072337f1f0 Remove VXLAN ref where it isn't correct 2022-11-01 09:40:13 -04:00
c3bc55eff8 Bump version to 0.9.56 2022-10-27 14:21:04 -04:00
6c58d52fa1 Add node autoready oneshot unit
This replicates some of the more important functionality of the defunct
pvc-flush.service unit. On presence of a trigger file (i.e.
/etc/pvc/autoready), it will trigger a "node ready" on boot. It does
nothing on shutdown as this must be handled by other mechanisms, though
a similar autoflush could be added as well.
2022-10-27 14:09:14 -04:00
666e02fbfd Ensure None filesystem is valid 2022-10-21 15:13:52 -04:00
46dde055c4 Move /dev umount to cleanup step 2022-10-21 14:47:48 -04:00
ef437c3dbf Adjust help message text 2022-10-21 14:22:15 -04:00
bd2208e8f6 Add ova as valid name in addition to default_ova 2022-10-21 14:13:40 -04:00
62d5ff11df Fix console config and domain argument 2022-10-21 14:04:17 -04:00
0019881cfa Add cloud-init configuration to debootstrap script
Prevents errors trying to find the cloud-init metadata source.
2022-10-21 14:03:34 -04:00
d46133802b Add pfsense example provisioner script 2022-10-21 13:35:48 -04:00
fcadde057e Include /proc in chroot mounts 2022-10-20 15:00:10 -04:00
2608f38d64 Properly handle missing source_volume from OVAs 2022-10-19 13:18:12 -04:00
89f05ced3f Add missing ceph import 2022-10-19 13:10:40 -04:00
729481126c Move conversion to install() step
Seems more clear to me than doing it in prepare()
2022-10-19 13:09:29 -04:00
41eccb9c7d Add missing imports 2022-10-19 13:07:34 -04:00
e550e39a5a Add output messages during OVA prepare 2022-10-19 12:58:11 -04:00
dff156b2b0 Fix bad comparison 2022-10-19 12:46:15 -04:00
1c4fb80d1f Remove reference to automatic upload of OVA script 2022-10-19 03:37:12 -04:00
ec7beb08cc Add missing flag 2022-10-19 03:34:37 -04:00
3a180193ee Fix wording of initial script paragraphs 2022-10-19 03:27:14 -04:00
e26ff8a975 Fix missing f-string marker 2022-10-15 16:26:47 -04:00
6276414702 Use own domain for docs links 2022-10-08 21:12:59 -04:00
a34d64a71b Add rinse example configuration
Provisions Rocky Linux 8 and 9 systems, and potentially older
CentOS/Fedora/Scientific Linux/SuSE systems. Depends on a custom build
of rinse (3.7.1) with Rocky 9 support.
2022-10-07 19:55:56 -04:00
71297e0179 Add host-model to CPU config in VMs 2022-10-07 09:36:22 -04:00
45c9909428 Add output message to debootstrap install 2022-10-07 02:27:20 -04:00
7268592c87 Fix braces in fstring example 2022-10-06 15:57:31 -04:00
726d0a562b Update copyright header year 2022-10-06 11:55:27 -04:00
39e1fc50ed Fix titles 2022-10-06 11:54:36 -04:00
7a3870fc44 Add OVA script support
1. Ensure that system_template and script are not nullable in the DB.
2. Ensure that the CLI and API enforce the above and clean up CLI
arguments for profile add.
3. Ensure that, before uploading OVAs, a 'default_ova' provisioning
script is present.
4. Use the 'default_ova' script for new OVA uploads.
5. Ensure that OVA details are properly added to the vm_data dict in the
provisioner vmbuilder.
2022-10-06 10:48:12 -04:00
bffab7a5a1 Reverse numbering of example scripts 2022-10-06 10:14:37 -04:00
6cbaeb5dc8 Complete OVA provisioning script 2022-10-06 10:14:04 -04:00
58ce133c8d Remove lingering OVA references 2022-10-06 00:13:36 -04:00
43feb33caa Update documentation to reflect script changes 2022-10-06 00:06:02 -04:00
3a5d8c61da Reorganize and add more comments to examples 2022-10-05 23:35:53 -04:00
1e0b502250 Fix remaining bugs in example scripts 2022-10-05 22:37:11 -04:00
fe17d28385 Ensure inner cleanup and end message response 2022-10-05 22:36:42 -04:00
8aaac33056 Fix bad variable reference 2022-10-05 17:43:23 -04:00
cc7952c232 Add additional import for config 2022-10-05 17:41:37 -04:00
16915ed507 Add better exception handling with ctxtmgrs 2022-10-05 17:35:05 -04:00
2c624ceb2c Add additional missing import in examples 2022-10-05 17:29:34 -04:00
da85480488 Improve error messages 2022-10-05 17:26:09 -04:00
47b0704555 Fix bad ref in examples 2022-10-05 17:25:56 -04:00
7c49967586 Add proper imports to examples 2022-10-05 17:22:04 -04:00
e3f96ac87e Better handle cleanups and fix chroot bug 2022-10-05 17:21:30 -04:00
4df70cf086 Implement new provisioner setup 2022-10-05 16:03:05 -04:00
f1df1cfe93 Bump version to 0.9.55 2022-10-04 13:21:40 -04:00
5942aa50fc Avoid raise/handle deadlocks
Can cause log flooding in some edge cases and isn't really needed any
longer. Use a proper conditional followed by an actual error handler.
2022-10-03 14:04:12 -04:00
096bcdfd75 Try a literal eval first
This is a breakage between the older version of Celery (Deb10) and
newer. The hard removal broke Deb10 instances.

So try that first, and on failure, assume newer Celery format.
2022-09-06 10:34:50 -04:00
239c392892 Bump version to 0.9.54 2022-08-23 11:01:05 -04:00
172d0a86e4 Use proper SSLContext and enable TLSv1
It's bad, but sometimes you need to access the API from a very old
software version. So just enable it for now and clean it up later.
2022-08-23 10:58:47 -04:00
d8e57a26c5 Fix bad variable name 2022-08-18 11:37:57 -04:00
9b499b9f48 Bump version to 0.9.53 2022-08-12 17:47:11 -04:00
881550b610 Actually fix VM sorting
Due to the executor the previous attempt did not work.
2022-08-12 17:46:29 -04:00
2a21d48128 Bump version to 0.9.52 2022-08-12 11:09:25 -04:00
8d0f26ff7a Add additional kb_ values to OSD stats
Allows for easier parsing later to get e.g. % values and more details on
the used amounts.
2022-08-11 11:06:36 -04:00
bcabd7d079 Always sort VM list
Same justification as previous commit.
2022-08-09 12:05:40 -04:00
05a316cdd6 Ensure the node list is sorted
Otherwise the node entries could come back in an arbitrary order; since
this is an ordered list of dictionaries that might not be expected by
the API consumers, so ensure it's always sorted.
2022-08-09 12:03:49 -04:00
4b36753f27 Add reference to bootstrap in index 2022-08-03 20:22:16 -04:00
171f6ac9ed Add missing cluster_req for vm modify 2022-08-02 10:02:26 -04:00
645b525ad7 Bump version to 0.9.51 2022-07-25 23:25:41 -04:00
ec559aec0d Remove pvc-flush service
This service caused more headaches than it was worth, so remove it.

The original goal was to cleanly flush nodes on shutdown and unflush
them on startup, but this is tightly controlled by Ansible playbooks at
this point, and this is something best left to the Administrator and
their particular situation anyways.
2022-07-25 23:21:34 -04:00
71ffd5a191 Add confirmation to disable command 2022-07-21 16:43:37 -04:00
2739c27299 Remove faulty literal_eval 2022-07-18 13:35:15 -04:00
56129a3636 Fix bad changelog entries 2022-07-06 16:57:55 -04:00
932b3c55a3 Bump version to 0.9.50 2022-07-06 16:01:14 -04:00
92e2ff7449 Fix bug with space-containing detect strings 2022-07-06 15:58:57 -04:00
d8d3feee22 Add selector help and adjust flag name
1. Add documentation on the node selector flags. In the API, reference
the daemon configuration manual which now includes details in this
section; in the CLI, provide the help in "pvc vm define" in detail and
then reference that command's help in the other commands that use this
field.

2. Ensure the naming is consistent in the CLI, using the flag name
"--node-selector" everywhere (was "--selector" for "pvc vm" commands and
"--node-selector" for "pvc provisioner" commands).
2022-06-10 02:42:06 -04:00
b1357cafdb Add memfree to selector and use proper defaults 2022-06-10 02:03:12 -04:00
f8cdcb30ba Add migration selector via free memory
Closes #152
2022-05-18 03:47:16 -04:00
51ad2058ed Bump version to 0.9.49 2022-05-06 15:49:39 -04:00
c401a1f655 Use consistent language for primary mode
I didn't call it "router" anywhere else, but the state in the list is
called "coordinator" so, call it "coordinator mode".
2022-05-06 15:40:52 -04:00
7a40c7a55b Add support for replacing/refreshing OSDs
Adds commands to both replace an OSD disk, and refresh (reimport) an
existing OSD disk on a new node. This handles the cases where an OSD
disk should be replaced (either due to upgrades or failures) or where a
node is rebuilt in-place and an existing OSD must be re-imported to it.

This should avoid the need to do a full remove/add sequence for either
case.

Also cleans up some aspects of OSD removal that are identical between
methods (e.g. using safe-to-destroy and sleeping after stopping) and
fixes a bug if an OSD does not truly exist when the daemon starts up.
2022-05-06 15:32:06 -04:00
8027a6efdc Improve handling of rounded values 2022-05-02 15:29:30 -04:00
3801fcc07b Fix bug with initial JSON for stats 2022-05-02 13:28:19 -04:00
c741900baf Refactor OSD removal to use new ZK data
With the OSD LVM information stored in Zookeeper, we can use this to
determine the actual block device to zap rather than relying on runtime
determination and guestimation.
2022-05-02 12:52:22 -04:00
464f0e0356 Store additional OSD information in ZK
Ensures that information like the FSIDs and the OSD LVM volume are
stored in Zookeeper at creation time and updated at daemon start time
(to ensure the data is populated at least once, or if the /dev/sdX
path changes).

This will allow safer operation of OSD removals and the potential
implementation of re-activation after node replacements.
2022-05-02 12:11:39 -04:00
cea8832f90 Ensure initial OSD stats is populated
Values are all invalid but this ensures the client won't error out when
trying to show an OSD that has never checked in yet.
2022-04-29 16:50:30 -04:00
5807351405 Bump version to 0.9.48 2022-04-29 15:03:52 -04:00
d6ca74376a Fix bugs with forced removal 2022-04-29 14:03:07 -04:00
413100a147 Ensure unresponsive OSDs still display in list
It is still useful to see such dead OSDs even if they've never checked
in or have not checked in for quite some time.
2022-04-29 12:11:52 -04:00
4d698be34b Add OSD removal force option
Ensures a removal can continue even in situations where some step(s)
might fail, for instance removing an obsolete OSD from a replaced node.
2022-04-29 11:16:33 -04:00
53aed0a735 Use a singular configured cluster by default
If there is...
  1. No '--cluster' passed, and
  2. No 'local' cluster, and
  3. There is exactly one cluster configured
...then use that cluster by default in the CLI.
2022-01-13 18:36:20 -05:00
ea709f573f Bump version to 0.9.47 2021-12-28 22:03:08 -05:00
1142454934 Add pool PGs count modification
Allows an administrator to adjust the PG count of a given pool. This can
be used to increase the PGs (for example after adding more OSDs) or
decrease it (to remove OSDs, reduce CPU load, etc.).
2021-12-28 21:53:29 -05:00
bbfad340a1 Add PGs count to pool list 2021-12-28 21:12:02 -05:00
c73939e1c5 Fix issue if pool stats have not updated yet 2021-12-28 21:03:10 -05:00
25fe45dd28 Add device class tiers to Ceph pools
Allows specifying a particular device class ("tier") for a given pool,
for instance SSD-only or NVMe-only. This is implemented with Crush
rules on the Ceph side, and via an additional new key in the pool
Zookeeper schema which is defaulted to "default".
2021-12-28 20:58:15 -05:00
58d57d7037 Bump version to 0.9.46 2021-12-28 15:02:14 -05:00
00d2c67c41 Allow single-node clusters to restart and timeout
Prevents a daemon from waiting forever to terminate if it is primary,
and avoids this entirely if there is only a single node in the cluster.
2021-12-28 03:06:03 -05:00
67131de4f6 Fix bug when removing OSDs
Ensure the OSD is down as well as out or purge might fail.
2021-12-28 03:05:34 -05:00
abc23ebb18 Handle detect strings as arguments for blockdevs
Allows specifying blockdevs in the OSD and OSD-DB addition commands as
detect strings rather than actual block device paths. This provides
greater flexibility for automation with pvcbootstrapd (which originates
the concept of detect strings) and in general usage as well.
2021-12-28 02:53:02 -05:00
9f122e916f Allow bypassing confirm message for benchmarks 2021-12-23 21:00:42 -05:00
3ce4d90693 Add auditing to local syslog from PVC client
This ensures that any client command is logged by the local system.
Helps ensure Accounting for users of the CLI. Currently logs the full
command executed along with the $USER environment variable contents.
2021-12-10 16:17:33 -05:00
6ccd19e636 Standardize fuzzy matching and use fullmatch
Solves two problems:

1. How match fuzziness was used was very inconsistent; make them all the
same, i.e. "if is_fuzzy and limit, apply .* to both sides".

2. Use re.fullmatch instead of re.match to ensure exact matching of the
regex to the value. Without fuzziness, this would sometimes cause
inconsistent behavior, for instance if a limit was non-fuzzy "vm",
expecting to match the actual "vm", but also matching "vm1" too.
2021-12-06 16:35:29 -05:00
d8689e6eaa Remove "and started" from message text
This is not necessarily the case.
2021-11-29 16:42:26 -05:00
bc49b5eca2 Fix bug with cloned image sizes 2021-11-29 14:56:50 -05:00
8470dfaa29 Fix bugs with legacy benchmark format 2021-11-26 11:42:35 -05:00
f164d898c1 Bump version to 0.9.45 2021-11-25 09:34:20 -05:00
195f31501c Ensure echo always has an argument 2021-11-25 09:33:26 -05:00
a8899a1d66 Fix ordering of pvcnoded unit
We want to be after network.target and want network-online.target
2021-11-18 16:56:49 -05:00
817dffcf30 Bump version to 0.9.44 2021-11-11 16:20:38 -05:00
eda2a57a73 Add Munin plugin for Ceph utilization 2021-11-08 15:21:09 -05:00
135d28e60b Add 0.05s to connection timeout
This is recommended by the Python Requests documentation:

> It’s a good practice to set connect timeouts to slightly larger than a
  multiple of 3, which is the default TCP packet retransmission window.
2021-11-08 03:11:41 -05:00
e7d7378bae Use separate connect and data timeouts
This allows us to keep a very low connect timeout of 3 seconds, but also
ensure that long commands (e.g. --wait or VM disable) can take as long
as the API requires to complete.

Avoids having to explicitly set very long single-instance timeouts for
other functions which would block forever on an unreachable API.
2021-11-08 03:10:09 -05:00
799c3e8d5d Fix quote in sed for unstable deb build 2021-11-08 02:54:27 -05:00
d0ec24f690 Add sudo to deploy-package task 2021-11-08 02:41:10 -05:00
6e9fcd38a3 Bump version to 0.9.43 2021-11-08 02:29:17 -05:00
f51f9fc4c8 Fix sed commands after Black formatting change 2021-11-08 02:29:05 -05:00
a6dcffc737 Remove references to Ansible manual 2021-11-08 00:29:47 -05:00
364c190106 Remove Ansible and Testing manuals
The Ansible manual can't keep up with the other repo, so it should live
there instead (eventually, after significant rewrites).

The Testing page is obsoleted by the "test-cluster" script.
2021-11-08 00:25:27 -05:00
ea19af6494 Allow American spelling for compatibility 2021-11-08 00:09:59 -05:00
7069d3237c Shorten help messages slightly to fit 2021-11-08 00:07:21 -05:00
619c3f7ff5 Add forced colour support
Allows preserving colour within e.g. watch, where Click would normally
determine that it is "not a terminal". This is done via the wrapper echo
which filters via the local config.
2021-11-08 00:04:20 -05:00
8a75bb3011 Add funding configuration 2021-11-06 18:05:17 -04:00
a817c3e678 Add start delineators to command output 2021-11-06 13:35:30 -04:00
0cc3f2deab Revamp formatting and linting on commit
Remove the prepare script, and run the two stages manually. Better
handle Black reformatting by doing a check (for the errcode) then
reformat and abort commit to review.
2021-11-06 13:34:33 -04:00
21b4bbe51a Apply more granular timeout formatting
We don't need to wait forever if state changes aren't waiting or disable
(which does a shutdown before returning).
2021-11-06 13:34:03 -04:00
87ec31c023 Up timeout when setting VM state
Ensures the API won't time out immediately especially during a
wait-flagged or disable action.
2021-11-06 04:15:10 -04:00
0d857d5ab8 Use positive check rather than negative
Ensure the VM is start before doing shutdown/stop, rather than being
stopped. Prevents overwrite of existing disable state and other
weirdness.
2021-11-06 04:08:33 -04:00
006f40f195 Add disable forcing to CLI
References #148
2021-11-06 04:02:50 -04:00
5f193a6134 Perform automatic shutdown/stop on VM disable
Instead of requiring the VM to already be stopped, instead allow disable
state changes to perform a shutdown first. Also add a force option which
will do a hard stop instead of a shutdown.

References #148
2021-11-06 03:57:24 -04:00
78faa90139 Reformat recent changes with Black 2021-11-06 03:27:07 -04:00
23b1501f40 Fix linting error F541 f-string placeholders 2021-11-06 03:26:03 -04:00
66bfad3109 Fix linting errors F522/F523 unused args 2021-11-06 03:24:50 -04:00
eee5c25d6f Rename build-deb.sh to build-stable-deb.sh
Unifies the naming with the other build-unstable-deb.sh script.
2021-11-06 03:18:58 -04:00
ff4fc18a60 Remove obsolete gitlab-ci config 2021-11-06 03:18:22 -04:00
ac885b855a Ensure all helper scripts pushd/popd
Make sure all of these move to the root of the repository first, then
return to where they were afterwards, using pushd/popd. This allows them
to be executed from anywhere in the repo.
2021-11-06 03:17:47 -04:00
b9c30baf80 Unify formatting and linting
Ensures optimal formatting in addition to linting during manual deploys
and during pre-commit actions.
2021-11-06 03:10:17 -04:00
9b12cc0236 Add newline to start of lint 2021-11-06 03:04:14 -04:00
c41664d2da Reformat code with Black code formatter
Unify the code style along PEP and Black principles using the tool.
2021-11-06 03:02:43 -04:00
3779bc960e Add safe mode to Black 2021-11-06 02:59:54 -04:00
5c620262e9 Move Flake configuration into dedicated file
Avoid passing arguments in the script.
2021-11-06 02:55:37 -04:00
6b88fbd1e3 Clean up linter after Black add (pass two) 2021-11-06 02:51:14 -04:00
a50c8e6a4d Exclude Alembic migrations from Black
These files are autogenerated with their own formats, so we don't want
to override that.
2021-11-06 02:46:06 -04:00
7d6e4353f1 Clean up linter after Black add (pass one) 2021-11-06 02:44:24 -04:00
bf30b31db6 Add black formatter to project root 2021-11-06 02:44:05 -04:00
70bd601dc1 Add Basic Builder configuration
Configuration for my new CI system under Gitea.
2021-10-31 00:09:55 -04:00
2e7b9b28b3 Add some delay and additional tries to fencing 2021-10-27 16:24:17 -04:00
12eef58d42 Fix ordering to show correct message 2021-10-27 13:37:52 -04:00
f2e6892fd2 Support adding the same network to a VM again
This is a supported configuration for some edge cases and should be
allowed.
2021-10-27 13:33:27 -04:00
91fb9e1241 Reorder linting on build-and-deploy 2021-10-27 13:25:14 -04:00
d87bea4159 More gracefully handle restart + live
Instead of erroring, just use the implication that restarting a VM does
not want a live modification, and proceed from there. Update the help
text to match.
2021-10-27 13:23:39 -04:00
3a6f442856 Support removing VM interfaces by MAC
Provides a way to handle multiple interfaces in the same network
gracefully, while making the previous behaviour explicit.
2021-10-27 13:20:05 -04:00
dfca998adf Fix bad test in postinst 2021-10-19 00:27:12 -04:00
55f397a347 Fix bad location of config sets 2021-10-12 17:23:04 -04:00
dfebb2d3e5 Also validate on failures 2021-10-12 17:11:03 -04:00
e88147db4a Bump version to 0.9.42 2021-10-12 15:25:42 -04:00
b8204d89ac Go back to passing if exception
Validation already happened and the set happens again later.
2021-10-12 14:21:52 -04:00
fe73dfbdc9 Use current live value for bridge_mtu
This will ensure that upgrading without the bridge_mtu config key set
will keep things as they are.
2021-10-12 12:24:03 -04:00
8f906c1f81 Use power off in fence instead of reset
Use a power off (and then make the power on a requirement) during a node
fence. Removes some potential ambiguity in the power state, since we
will know for certain if it is off.
2021-10-12 11:04:27 -04:00
2d9fb9688d Validate network MTU after initial read 2021-10-12 10:53:17 -04:00
fb84685c2a Make cluster example images clickable 2021-10-12 03:15:04 -04:00
032ba44d9c Mention fencing only in run state 2021-10-12 03:05:01 -04:00
b7761877e7 Adjust more wording and fix typos 2021-10-12 03:00:21 -04:00
1fe07640b3 Adjust some wording 2021-10-12 02:54:16 -04:00
b8d843ebe4 Remove codeql setup
I don't use this for anything useful, so disable it since a run takes
ages.
2021-10-12 02:51:19 -04:00
95d983ddff Fix formatting of subsection 2021-10-12 02:49:40 -04:00
4c5da1b6a8 Add reference to Ansible manual 2021-10-12 02:48:47 -04:00
be6b1e02e3 Fix spelling errors 2021-10-12 02:47:31 -04:00
ec2a72ed4b Fix link to cluster architecture docs 2021-10-12 02:41:22 -04:00
b06e327add Adjust getting started docs
Update the docs with the current information on setting up a cluster,
including simplifying the Ansible configuration to use the new
create-local-repo.sh script, and simplifying some other sections.
2021-10-12 02:39:25 -04:00
d1f32d2b9c Default to removing build artifacts in b-a-d.sh 2021-10-11 16:41:00 -04:00
3f78ca1cc9 Add explicit 3 second timeout to requests 2021-10-11 16:31:18 -04:00
e866335918 Add version function support to CLI 2021-10-11 15:34:41 -04:00
221494ed1b Add new configs for Ansible 2021-10-11 14:44:18 -04:00
f13cc04b89 Bump version to 0.9.41 2021-10-09 19:39:21 -04:00
4ed537ee3b Add bridge_mtu config to docs 2021-10-09 19:28:50 -04:00
95e01f38d5 Adjust log type of object setup message 2021-10-09 19:23:12 -04:00
3122d73bf5 Avoid duplicate runs of MTU set
It wasn't the validator duplicating, but the update duplicating, so
avoid that happening properly this time.
2021-10-09 19:21:47 -04:00
7ed8ef179c Revert "Avoid duplicate runs of MTU validator"
This reverts commit 56021c443a.
2021-10-09 19:11:42 -04:00
caead02b2a Set all log messages to information state
None of these were "success" messages and thus shouldn't have been ok
state.
2021-10-09 19:09:38 -04:00
87bc5f93e6 Avoid duplicate runs of MTU validator 2021-10-09 19:07:41 -04:00
203893559e Use correct isinstance instead of type 2021-10-09 19:03:31 -04:00
2c51bb0705 Move MTU validation to function
Prevents code duplication and ensures validation runs when an MTU is
updated, not just on network creation.
2021-10-09 19:01:45 -04:00
46d3daf686 Add logger message when setting MTU 2021-10-09 18:56:18 -04:00
e9d05aa24e Ensure vx_mtu is always an int() 2021-10-09 18:52:50 -04:00
d2c18d7b46 Fix bad header length in network list 2021-10-09 18:50:32 -04:00
6ce28c43af Add MTU value checking and log messages
Ensures that if a specified MTU is more than the maximum it is set to
the maximum instead, and adds warning messages for both situations.
2021-10-09 18:48:56 -04:00
87cda72ca9 Fix invalid schema key
Addresses #144
2021-10-09 18:42:33 -04:00
8f71a6d2f6 Add MTU support to network add/modify commands
Addresses #144
2021-10-09 18:06:21 -04:00
c45f8f5bd5 Have VXNetworkInstance set MTU if unset
Makes this explicit in Zookeeper if a network is unset, post-migration
(schema version 6).

Addresses #144
2021-10-09 17:52:57 -04:00
24de0f4189 Add MTU to network creation/modification
Addresses #144
2021-10-09 17:51:32 -04:00
3690a2c1e0 Fix migration bugs and invalid vx_mtu
Addresses #144
2021-10-09 17:35:10 -04:00
50d8aa0586 Add handlers for client network MTUs
Refactors some of the code in VXNetworkInterface to handle MTUs in a
more streamlined fashion. Also fixes a bug whereby bridge client
networks were being explicitly given the cluster dev MTU which might not
be correct. Now adds support for this option explicitly in the configs,
and defaults to 1500 for safety (the standard Ethernet MTU).

Addresses #144
2021-10-09 17:02:27 -04:00
db6e65712d Make n-1 values clearer 2021-10-07 18:11:15 -04:00
cf8e16543c Correct levels in TOC 2021-10-07 18:08:28 -04:00
1a4fcdcc2d Correct spelling errors 2021-10-07 18:07:06 -04:00
9a71db0800 Add documentation sections on IPMI and fencing 2021-10-07 18:05:47 -04:00
6ee4c55071 Correct flawed conditional in verify_ipmi 2021-10-07 15:11:19 -04:00
c27359c4bf Bump version to 0.9.40 2021-10-07 14:42:04 -04:00
46078932c3 Correct bad stop_keepalive_timer call 2021-10-07 14:41:12 -04:00
c89699bc6f Remove redundant wording from header 2021-10-07 12:20:04 -04:00
1b9507e4f5 Replace headers with links in CHANGELOG.md 2021-10-07 12:17:44 -04:00
3db7ac48f4 Add missing period to changelog sentence 2021-10-07 12:10:35 -04:00
1830ec6465 Move changelog into dedicated file
The changelog was getting far too long for the README/docs index to
support, so move it into CHANGELOG.md and link to it instead.
2021-10-07 12:09:26 -04:00
bdb9db8375 Bump version to 0.9.39 2021-10-07 11:52:38 -04:00
c61d7bc313 Add linting to build-and-deploy
Ensures that bad code isn't deployed during testing.
2021-10-07 11:51:05 -04:00
c0f7ba0125 Add limit negation to VM list
When using the "state", "node", or "tag" arguments to a VM list, add
support for a "negate" flag to look for all VMs *not in* the state,
node, or tag state.
2021-10-07 11:50:52 -04:00
761032b321 Add note about fencing at remote sites 2021-10-04 19:58:08 -04:00
3566e13e79 Correct TOC in architecture page 2021-10-04 01:54:22 -04:00
6b324029cf Correct spelling errors 2021-10-04 01:51:58 -04:00
13eeabf44b Double image sizes for example clusters 2021-10-04 01:47:35 -04:00
d86768d3d0 Adjust toc_depth for RTD theme 2021-10-04 01:45:05 -04:00
a167757600 Revamp about and architecture docs
Makes these a little simpler to follow and provides some more up-to-date
information based on recent tests and developments.
2021-10-04 01:42:08 -04:00
a95d9680ac Adjust bump-version changelog heading level 2021-10-04 01:41:48 -04:00
63962f10ba Move changelog headers down one more level 2021-10-04 01:41:22 -04:00
a7a681d92a Adjust indent of index/README versions 2021-10-04 00:33:24 -04:00
da9248cfa2 Bump version to 0.9.38 2021-10-03 22:32:41 -04:00
aa035a61a7 Correct latency units and format name 2021-10-03 17:06:34 -04:00
7c8ba56561 Revamp test result display
Instead of showing CLAT percentiles, which are very hard to interpret
and understand, instead use the main latency buckets.
2021-10-03 15:49:01 -04:00
bba73980de Revamp postinst for the API daemon
Ensures that the worker is always restarted and make the NOTE
conditional more specific.
2021-10-03 15:15:26 -04:00
32b3af697c Tweak fio tests for benchmarks
1. Remove ramp_time as this was giving very strange results.

2. Up the runtime to 75 seconds to compensate.

3. Print the fio command to the console to validate.
2021-10-03 15:06:18 -04:00
7c122ac921 Add benchmark format to list 2021-10-03 15:05:58 -04:00
0dbf139706 Adjust ETA for benchmarks 2021-10-02 04:51:01 -04:00
c909beaf6d Add format parsing for format 1 storage benchmarks 2021-10-02 04:46:44 -04:00
2da49297d2 Add version 2 benchmark list formatting 2021-10-02 02:47:17 -04:00
0ff9a6b8c4 Handle benchmark running state properly 2021-10-02 01:54:51 -04:00
28377178d2 Fix missing argument in database insert 2021-10-02 01:49:47 -04:00
e06b114c48 Update to storage benchmark format 1
1. Runs `fio` with the `--format=json` option and removes all terse
format parsing from the results.

2. Adds a 15-second ramp time to minimize wonky ramp-up results.

3. Sets group_reporting, which isn't necessary with only a single job,
but is here for consistency.
2021-10-02 01:41:08 -04:00
0058f19d88 Fix handling of array of information
With a benchmark info we only ever want test one, so pass only that to
the formatter. Simplifies the format function.
2021-10-02 01:28:39 -04:00
056cf3740d Avoid versioning benchmark lists
This wouldn't work since each individual test is versioned. Instead add
a placeholder for later once additional format(s) are defined.
2021-10-02 01:25:18 -04:00
58f174b87b Add format option to benchmark info
Allows specifying of raw json or json-pretty formats in addition to the
"pretty" formatted option.
2021-10-02 01:13:50 -04:00
37b98fd54f Add benchmark format function support
Allows choosing different list and info functions based on the benchmark
version found. Currently only implements "legacy" version 0 with more to
be added.
2021-10-02 01:07:25 -04:00
f83a345bfe Add test format versioning to storage benchmarks
Adds a test_format database column and a value in the API return for the
test format version, starting at 0 for the existing format as of 0.9.37.

References #143
2021-10-02 00:55:27 -04:00
ce06e4d81b Load benchmark results as JSON
Load the JSON at the API side instead of client side, because that's
what the API doc says it is and it just makes more sense.
2021-09-30 23:40:24 -04:00
23977b04fc Bump version to 0.9.37 2021-09-30 02:08:14 -04:00
bb1cca522f Revamp benchmark tests
1. Move to a time-based (60s) benchmark to avoid these taking an absurd
amount of time to show the same information.

2. Eliminate the 256k random benchmarks, since they don't really add
anything.

3. Add in a 4k single-queue benchmark as this might provide valuable
insight into latency.

4. Adjust the output to reflect the above changes.

While this does change the benchmarking, this should not invalidate any
existing benchmarks since most of the test suit is unchanged (especially
the most important 4M sequential and 4K random tests). It simply removes
an unused entry and adds a more helpful one. The time-based change
should not significantly affect the results either, just reduces the
total runtime for long-tests and increase the runtime for quick tests to
provide a better picture.
2021-09-29 20:51:30 -04:00
9a4dce4e4c Add primary node to benchmark job name
Ensures tracking of the current primary node the job was run on, since
this may be relevant for performance reasons.
2021-09-28 09:58:22 -04:00
f6f6f07488 Add timeouts to queue gets and adjust
Ensure that all keepalive timeouts are set (prevent the queue.get()
actions from blocking forever) and set the thread timeouts to line up as
well. Everything here is thus limited to keepalive_interval seconds
(default 5s) to keep it uniform.
2021-09-27 16:10:27 -04:00
142c999ce8 Re-add success log output during migration 2021-09-27 11:50:55 -04:00
1de069298c Fix missing character in log message 2021-09-27 00:49:43 -04:00
55221b3d97 Simplify VM migration down to 3 steps
Remove two superfluous synchronization steps which are not needed here,
since the exclusive lock handles that situation anyways.

Still does not fix the weird flush->unflush lock timeout bug, but is
better worked-around now due to the cancelling of the other wait freeing
this up and continuing.
2021-09-27 00:03:20 -04:00
0d72798814 Work around synchronization lock issues
Make the block on stage C only wait for 900 seconds (15 minutes) to
prevent indefinite blocking.

The issue comes if a VM is being received, and the current unflush is
cancelled for a flush. When this happens, this lock acquisition seems to
block for no obvious reason, and no other changes seem to affect it.
This is certainly some sort of locking bug within Kazoo but I can't
diagnose it as-is. Leave a TODO to look into this again in the future.
2021-09-26 23:26:21 -04:00
3638efc77e Improve log messages during VM migration 2021-09-26 23:15:38 -04:00
c2c888d684 Use event to non-block wait and fix inf wait 2021-09-26 22:55:39 -04:00
febef2e406 Track status of VM state thread 2021-09-26 22:55:21 -04:00
2a4f38e933 Simplify locking process for VM migration
Rather than using a cumbersome and overly complex ping-pong of read and
write locks, instead move to a much simpler process using exclusive
locks.

Describing the process in ASCII or narrative is cumbersome, but the
process ping-pongs via a set of exclusive locks and wait timers, so that
the two sides are able to synchronize via blocking the exclusive lock.
The end result is a much more streamlined migration (takes about half
the time all things considered) which should be less error-prone.
2021-09-26 22:08:07 -04:00
3b805cdc34 Fix failure to connect to libvirt in keepalive
This should be caught and abort the thread rather than failing and
holding up keepalives.
2021-09-26 20:42:01 -04:00
06f0f7ed91 Fix several bugs in fence handling
1. Output from ipmitool was not being stripped, and stray newlines were
throwing off the comparisons. Fixes this.

2. Several stages were lacking meaningful messages. Adds these in so the
output is more clear about what is going on.

3. Reduce the sleep time after a fence to just 1x the
keepalive_interval, rather than 2x, because this seemed like excessively
long even for slow IPMI interfaces, especially since we're checking the
power state now anyways.

4. Set the node daemon state to an explicit 'fenced' state after a
successful fence to indicate to users that the node was indeed fenced
successfully and not still 'dead'.
2021-09-26 20:07:30 -04:00
fd040ab45a Ensure pvc-flush is after network-online 2021-09-26 17:40:42 -04:00
e23e2dd9bf Fix typo in log message 2021-09-26 03:35:30 -04:00
ee4266f8ca Tweak CLI helptext around OSD actions
Adds some more detail about OSD commands and their values.
2021-09-26 01:29:23 -04:00
0f02c5eaef Fix typo in sgdisk command options 2021-09-26 00:59:05 -04:00
075abec5fe Use re.search instead of re.match
Required since we're not matching the start of the string.
2021-09-26 00:55:29 -04:00
3a1cbf8d01 Raise basic exceptions in CephInstance
Avoids no exception to reraise errors on failures.
2021-09-26 00:50:10 -04:00
a438a4155a Fix OSD creation for partition paths and fix gdisk
The previous implementation did not work with /dev/nvme devices or any
/dev/disk/by-* devices due to some logical failures in the partition
naming scheme, so fix these, and be explicit about what is supported in
the PVC CLI command output.

The 'echo | gdisk' implementation of partition creation also did not
work due to limitations of subprocess.run; instead, use sgdisk which
allows these commands to be written out explicitly and is included in
the same package as gdisk.
2021-09-26 00:12:28 -04:00
65df807b09 Add support for configurable OSD DB ratios
The default of 0.05 (5%) is likely ideal in the initial implementation,
but allow this to be set explicitly for maximum flexibility in
space-constrained or performance-critical use-cases.
2021-09-24 01:06:39 -04:00
d0f3e9e285 Bump version to 0.9.36 2021-09-23 14:01:38 -04:00
adc8a5a3bc Add separate OSD DB device support
Adds in three parts:

1. Create an API endpoint to create OSD DB volume groups on a device.
Passed through to the node via the same command pipeline as
creating/removing OSDs, and creates a volume group with a fixed name
(osd-db).

2. Adds API support for specifying whether or not to use this DB volume
group when creating a new OSD via the "ext_db" flag. Naming and sizing
is fixed for simplicity and based on Ceph recommendations (5% of OSD
size). The Zookeeper schema tracks the block device to use during
removal.

3. Adds CLI support for the new and modified API endpoints, as well as
displaying the block device and DB block device in the OSD list.

While I debated supporting adding a DB device to an existing OSD, in
practice this ended up being a very complex operation involving stopping
the OSD and setting some options, so this is not supported; this can be
specified during OSD creation only.

Closes #142
2021-09-23 13:59:49 -04:00
df277edf1c Move console watcher stop try up
Could cause an exception if d_domain is not defined yet.
2021-09-22 16:02:04 -04:00
772807deb3 Bump version to 0.9.35 2021-09-13 02:20:46 -04:00
58db537093 Add memory and vCPU checks to VM define/modify
Ensures that a VM won't:

(a) Have provisioned more RAM than there is available on a given node.
Due to memory overprovisioning, this is simply a "is the VM memory count
more than the node count", and doesn't factor in free or used memory on
a node, total cluster usage, etc. So if a node has 64GB total RAM, the
VM limit is 64GB. It is up to an administrator to ensure sanity *below*
that value.

(b) Have provisioned more vCPUs than there are CPU cores on the node,
minus 2 to account for hypervisor/storage processes. Will ensure there
is no severe CPU contention caused by a single VM having more vCPUs than
there are actual execution threads available.

Closes #139
2021-09-13 01:51:21 -04:00
e71a6c90bf Add pool size check when resizing volumes
Closes #140
2021-09-12 19:54:51 -04:00
a8e9a56924 Increase build-and-deploy sleep 2021-09-12 19:50:58 -04:00
f3fb492633 Handle VM disk/network stats gathering exceptions 2021-09-12 19:41:07 -04:00
e962743e51 Add VM device hot attach/detach support
Adds a new API endpoint to support hot attach/detach of devices, and the
corresponding client-side logic to use this endpoint when doing VM
network/storage add/remove actions.

The live attach is now the default behaviour for these types of
additions and removals, and can be disabled if needed.

Closes #141
2021-09-12 19:33:00 -04:00
46f1d761f6 Adjust lint script for newer linter 2021-09-12 15:40:38 -04:00
be954c1625 Don't crash cleanup if no this_node 2021-08-29 03:52:18 -04:00
fb46f5f9e9 Change default node object state to flushed 2021-08-29 03:34:08 -04:00
694b8e85a0 Bump version to 0.9.34 2021-08-24 16:15:25 -04:00
eb321497ee Properly handle =-separated fsargs 2021-08-24 11:40:22 -04:00
5b81e59481 Convert argument elements of task status to types 2021-08-23 14:28:12 -04:00
a4c0e0befd Fix typo in output message 2021-08-23 00:39:19 -04:00
a18cef5f25 Bump version to 0.9.33 2021-08-21 03:28:48 -04:00
f6c5aa9992 Avoid failing if no provisioner tasks 2021-08-21 03:25:16 -04:00
ffa3dd5edb Ensure pycache files are removed on deb creation 2021-08-21 03:19:18 -04:00
afb0359c20 Refactor pvcnoded to reduce Daemon.py size
This branch commit refactors the pvcnoded component to better adhere to
good programming practices. The previous Daemon.py was a massive file
which contained almost 2000 lines of direct, root-level code which was
directly imported. Not only was this poor practice, but this resulted
in a nigh-unmaintainable file which was hard even for me to understand.

This refactoring splits a large section of the code from Daemon.py into
separate small modules and functions in the `util/` directory. This will
hopefully make most of the functionality easy to find and modify without
having to dig through a single large file.

Further the existing subcomponents have been moved to the `objects/`
directory which clearly separates them.

Finally, the Daemon.py code has mostly been moved into a function,
`entrypoint()`, which is then called from the `pvcnoded.py` stub.

An additional item is that most format strings have been replaced by
f-strings to make use of the Python 3.6 features in Daemon.py and the
utility files.
2021-08-21 03:14:22 -04:00
afdf254297 Bump version to 0.9.32 2021-08-19 12:37:58 -04:00
42e776fac1 Properly handle exceptions getting VM stats 2021-08-19 12:36:31 -04:00
dae67a1b7b Fix image dimensions and size 2021-08-18 19:51:55 -04:00
b86f8c1e09 Add screenshots to docs 2021-08-18 19:49:53 -04:00
21 changed files with 102 additions and 852 deletions

View File

@ -1 +1 @@
0.9.61
0.9.59

View File

@ -1,15 +1,5 @@
## PVC Changelog
###### [v0.9.61](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.61)
* [provisioner] Fixes a bug in network comparison
* [api] Fixes a bug being unable to rename disabled VMs
###### [v0.9.60](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.60)
* [Provisioner] Cleans up several remaining bugs in the example scripts; they should all be valid now
* [Provisioner] Adjust default libvirt schema to disable RBD caching for a 2x+ performance boost
###### [v0.9.59](https://github.com/parallelvirtualcluster/pvc/releases/tag/v0.9.59)
* [API] Flips the mem(prov) and mem(free) selectors making mem(free) the default for "mem" and "memprov" explicit

View File

@ -398,7 +398,7 @@ class VMBuilderScript(VMBuilder):
if volume.get("source_volume") is not None:
continue
if volume.get("filesystem") is None or volume.get("filesystem") == "swap":
if volume.get("filesystem") is None:
continue
mapped_dst_volume = f"/dev/rbd/{dst_volume}"
@ -473,7 +473,7 @@ class VMBuilderScript(VMBuilder):
]
# We need to know our root disk for later GRUB-ing
root_volume = None
root_disk = None
for volume in volumes:
if volume["mountpoint"] == "/":
root_volume = volume
@ -725,7 +725,6 @@ GRUB_DISABLE_LINUX_UUID=false
if (
volume.get("source_volume") is None
and volume.get("filesystem") is not None
and volume.get("filesystem") != "swap"
):
# Unmount filesystem
retcode, stdout, stderr = pvc_common.run_os_command(

View File

@ -20,7 +20,7 @@
###############################################################################
# This script provides an example of a PVC provisioner script. It will create a
# standard VM config and install a RHEL 8+ or similar OS using rinse.
# standard VM config and install a RHEL-like OS using rinse.
# This script can thus be used as an example or reference implementation of a
# PVC provisioner script and expanded upon as required.
@ -398,7 +398,7 @@ class VMBuilderScript(VMBuilder):
if volume.get("source_volume") is not None:
continue
if volume.get("filesystem") is None or volume.get("filesystem") == "swap":
if volume.get("filesystem") is None:
continue
mapped_dst_volume = f"/dev/rbd/{dst_volume}"
@ -487,7 +487,7 @@ class VMBuilderScript(VMBuilder):
post_packages = ["cloud-init"]
# We need to know our root disk for later GRUB-ing
root_volume = None
root_disk = None
for volume in volumes:
if volume["mountpoint"] == "/":
root_volume = volume
@ -571,6 +571,21 @@ class VMBuilderScript(VMBuilder):
with open(hostname_file, "w") as fh:
fh.write("{}".format(vm_name))
# Fix the cloud-init.target since it's broken by default
cloudinit_target_file = "{}/etc/systemd/system/cloud-init.target".format(
temporary_directory
)
with open(cloudinit_target_file, "w") as fh:
# We lose our indent on these raw blocks to preserve the apperance of the files
# inside the VM itself
data = """[Install]
WantedBy=multi-user.target
[Unit]
Description=Cloud-init target
After=multi-user.target
"""
fh.write(data)
# Due to device ordering within the Libvirt XML configuration, the first Ethernet interface
# will always be on PCI bus ID 2, hence the name "ens2".
# Write a DHCP stanza for ens2
@ -667,6 +682,11 @@ GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=
# Set the timezone to UTC
os.system("ln -sf ../usr/share/zoneinfo/UTC /etc/localtime")
# Unmount the bound devfs and sysfs
os.system("umount {}/dev".format(temporary_directory))
os.system("umount {}/sys".format(temporary_directory))
os.system("umount {}/proc".format(temporary_directory))
def cleanup(self):
"""
cleanup(): Perform any cleanup required due to prepare()/install()
@ -680,7 +700,6 @@ GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=
"""
# Run any imports first
import os
from pvcapid.vmbuilder import open_zk
from pvcapid.Daemon import config
import daemon_lib.common as pvc_common
@ -689,11 +708,6 @@ GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=
# Set the tempdir we used in the prepare() and install() steps
temp_dir = "/tmp/target"
# Unmount the bound devfs and sysfs
os.system(f"umount {temp_dir}/dev")
os.system(f"umount {temp_dir}/sys")
os.system(f"umount {temp_dir}/proc")
# Use this construct for reversing the list, as the normal reverse() messes with the list
for volume in list(reversed(self.vm_data["volumes"])):
dst_volume_name = f"{self.vm_name}_{volume['disk_id']}"
@ -704,7 +718,6 @@ GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=
if (
volume.get("source_volume") is None
and volume.get("filesystem") is not None
and volume.get("filesystem") != "swap"
):
# Unmount filesystem
retcode, stdout, stderr = pvc_common.run_os_command(
@ -715,14 +728,14 @@ GRUB_SERIAL_COMMAND="serial --speed=115200 --unit=0 --word=8 --parity=no --stop=
f"Failed to unmount '{mapped_dst_volume}' on '{mount_path}': {stderr}"
)
# Unmap volume
with open_zk(config) as zkhandler:
success, message = pvc_ceph.unmap_volume(
zkhandler,
volume["pool"],
dst_volume_name,
)
if not success:
raise ProvisioningError(
f"Failed to unmap '{mapped_dst_volume}': {stderr}"
)
# Unmap volume
with open_zk(config) as zkhandler:
success, message = pvc_ceph.unmap_volume(
zkhandler,
volume["pool"],
dst_volume_name,
)
if not success:
raise ProvisioningError(
f"Failed to unmap '{mapped_dst_volume}': {stderr}"
)

View File

@ -27,7 +27,7 @@ from ssl import SSLContext, TLSVersion
from distutils.util import strtobool as dustrtobool
# Daemon version
version = "0.9.61"
version = "0.9.59"
# API version
API_VERSION = 1.0

View File

@ -100,7 +100,7 @@ devices_scsi_controller = """ <controller type='scsi' index='0' model='virtio
# * vm_name
# * disk_id
devices_disk_header = """ <disk type='network' device='disk'>
<driver name='qemu' discard='unmap' cache='none'/>
<driver name='qemu' discard='unmap'/>
<target dev='{disk_id}' bus='scsi'/>
<auth username='libvirt'>
<secret type='ceph' uuid='{ceph_storage_secret}'/>

View File

@ -580,7 +580,7 @@ def delete_template_network_element(name, vni):
networks, code = list_template_network_vnis(name)
found_vni = False
for network in networks:
if network["vni"] == vni:
if network["vni"] == int(vni):
found_vni = True
if not found_vni:
retmsg = {

View File

@ -2,7 +2,7 @@ from setuptools import setup
setup(
name="pvc",
version="0.9.61",
version="0.9.59",
packages=["pvc", "pvc.cli_lib"],
install_requires=[
"Click",

View File

@ -1 +0,0 @@
{"version": "9", "root": "", "base": {"root": "", "schema": "/schema", "schema.version": "/schema/version", "config": "/config", "config.maintenance": "/config/maintenance", "config.primary_node": "/config/primary_node", "config.primary_node.sync_lock": "/config/primary_node/sync_lock", "config.upstream_ip": "/config/upstream_ip", "config.migration_target_selector": "/config/migration_target_selector", "cmd": "/cmd", "cmd.node": "/cmd/nodes", "cmd.domain": "/cmd/domains", "cmd.ceph": "/cmd/ceph", "logs": "/logs", "node": "/nodes", "domain": "/domains", "network": "/networks", "storage": "/ceph", "storage.util": "/ceph/util", "osd": "/ceph/osds", "pool": "/ceph/pools", "volume": "/ceph/volumes", "snapshot": "/ceph/snapshots"}, "logs": {"node": "", "messages": "/messages"}, "node": {"name": "", "keepalive": "/keepalive", "mode": "/daemonmode", "data.active_schema": "/activeschema", "data.latest_schema": "/latestschema", "data.static": "/staticdata", "data.pvc_version": "/pvcversion", "running_domains": "/runningdomains", "count.provisioned_domains": "/domainscount", "count.networks": "/networkscount", "state.daemon": "/daemonstate", "state.router": "/routerstate", "state.domain": "/domainstate", "cpu.load": "/cpuload", "vcpu.allocated": "/vcpualloc", "memory.total": "/memtotal", "memory.used": "/memused", "memory.free": "/memfree", "memory.allocated": "/memalloc", "memory.provisioned": "/memprov", "ipmi.hostname": "/ipmihostname", "ipmi.username": "/ipmiusername", "ipmi.password": "/ipmipassword", "sriov": "/sriov", "sriov.pf": "/sriov/pf", "sriov.vf": "/sriov/vf", "monitoring.plugins": "/monitoring_plugins", "monitoring.data": "/monitoring_data"}, "monitoring_plugin": {"name": "", "last_run": "/last_run", "health_delta": "/health_delta", "message": "/message", "data": "/data", "runtime": "/runtime"}, "sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, "sriov_vf": {"phy": "", "pf": "/pf", "mtu": "/mtu", "mac": "/mac", "phy_mac": "/phy_mac", "config": "/config", "config.vlan_id": "/config/vlan_id", "config.vlan_qos": "/config/vlan_qos", "config.tx_rate_min": "/config/tx_rate_min", "config.tx_rate_max": "/config/tx_rate_max", "config.spoof_check": "/config/spoof_check", "config.link_state": "/config/link_state", "config.trust": "/config/trust", "config.query_rss": "/config/query_rss", "pci": "/pci", "pci.domain": "/pci/domain", "pci.bus": "/pci/bus", "pci.slot": "/pci/slot", "pci.function": "/pci/function", "used": "/used", "used_by": "/used_by"}, "domain": {"name": "", "xml": "/xml", "state": "/state", "profile": "/profile", "stats": "/stats", "node": "/node", "last_node": "/lastnode", "failed_reason": "/failedreason", "storage.volumes": "/rbdlist", "console.log": "/consolelog", "console.vnc": "/vnc", "meta.autostart": "/node_autostart", "meta.migrate_method": "/migration_method", "meta.node_selector": "/node_selector", "meta.node_limit": "/node_limit", "meta.tags": "/tags", "migrate.sync_lock": "/migrate_sync_lock"}, "tag": {"name": "", "type": "/type", "protected": "/protected"}, "network": {"vni": "", "type": "/nettype", "mtu": "/mtu", "rule": "/firewall_rules", "rule.in": "/firewall_rules/in", "rule.out": "/firewall_rules/out", "nameservers": "/name_servers", "domain": "/domain", "reservation": "/dhcp4_reservations", "lease": "/dhcp4_leases", "ip4.gateway": "/ip4_gateway", "ip4.network": "/ip4_network", "ip4.dhcp": "/dhcp4_flag", "ip4.dhcp_start": "/dhcp4_start", "ip4.dhcp_end": "/dhcp4_end", "ip6.gateway": "/ip6_gateway", "ip6.network": "/ip6_network", "ip6.dhcp": "/dhcp6_flag"}, "reservation": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname"}, "lease": {"mac": "", "ip": "/ipaddr", "hostname": "/hostname", "expiry": "/expiry", "client_id": "/clientid"}, "rule": {"description": "", "rule": "/rule", "order": "/order"}, "osd": {"id": "", "node": "/node", "device": "/device", "db_device": "/db_device", "fsid": "/fsid", "ofsid": "/fsid/osd", "cfsid": "/fsid/cluster", "lvm": "/lvm", "vg": "/lvm/vg", "lv": "/lvm/lv", "stats": "/stats"}, "pool": {"name": "", "pgs": "/pgs", "tier": "/tier", "stats": "/stats"}, "volume": {"name": "", "stats": "/stats"}, "snapshot": {"name": "", "stats": "/stats"}}

View File

@ -644,7 +644,7 @@ def rename_vm(zkhandler, domain, new_domain):
# Verify that the VM is in a stopped state; renaming is not supported otherwise
state = zkhandler.read(("domain.state", dom_uuid))
if state not in ["stop", "disable"]:
if state != "stop":
return (
False,
'ERROR: VM "{}" is not in stopped state; VMs cannot be renamed while running.'.format(

View File

@ -540,7 +540,7 @@ class ZKHandler(object):
#
class ZKSchema(object):
# Current version
_version = 9
_version = 8
# Root for doing nested keys
_schema_root = ""
@ -608,17 +608,6 @@ class ZKSchema(object):
"sriov": "/sriov",
"sriov.pf": "/sriov/pf",
"sriov.vf": "/sriov/vf",
"monitoring.plugins": "/monitoring_plugins",
"monitoring.data": "/monitoring_data",
},
# The schema of an individual monitoring plugin data entry (/nodes/{node_name}/monitoring_data/{plugin})
"monitoring_plugin": {
"name": "", # The root key
"last_run": "/last_run",
"health_delta": "/health_delta",
"message": "/message",
"data": "/data",
"runtime": "/runtime",
},
# The schema of an individual SR-IOV PF entry (/nodes/{node_name}/sriov/pf/{pf})
"sriov_pf": {"phy": "", "mtu": "/mtu", "vfcount": "/vfcount"}, # The root key
@ -885,10 +874,9 @@ class ZKSchema(object):
if not zkhandler.zk_conn.exists(nkipath):
result = False
# One might expect child keys under node (specifically, sriov.pf, sriov.vf,
# monitoring.data) to be managed here as well, but those are created
# automatically every time pvcnoded started and thus never need to be validated
# or applied.
# One might expect child keys under node (specifically, sriov.pf and sriov.vf) to be
# managed here as well, but those are created automatically every time pvcnoded starts
# and thus never need to be validated or applied.
# These two have several children layers that must be parsed through
for elem in ["volume"]:

14
debian/changelog vendored
View File

@ -1,17 +1,3 @@
pvc (0.9.61-0) unstable; urgency=high
* [provisioner] Fixes a bug in network comparison
* [api] Fixes a bug being unable to rename disabled VMs
-- Joshua M. Boniface <joshua@boniface.me> Wed, 08 Feb 2023 10:08:05 -0500
pvc (0.9.60-0) unstable; urgency=high
* [Provisioner] Cleans up several remaining bugs in the example scripts; they should all be valid now
* [Provisioner] Adjust default libvirt schema to disable RBD caching for a 2x+ performance boost
-- Joshua M. Boniface <joshua@boniface.me> Tue, 06 Dec 2022 15:42:55 -0500
pvc (0.9.59-0) unstable; urgency=high
* [API] Flips the mem(prov) and mem(free) selectors making mem(free) the default for "mem" and "memprov" explicit

View File

@ -5,4 +5,3 @@ node-daemon/pvcnoded.service lib/systemd/system
node-daemon/pvc.target lib/systemd/system
node-daemon/pvcautoready.service lib/systemd/system
node-daemon/monitoring usr/share/pvc
node-daemon/plugins usr/share/pvc

View File

@ -1,126 +0,0 @@
#!/usr/bin/env python3
# ceph.py - PVC Monitoring example plugin for ceph status
# Part of the Parallel Virtual Cluster (PVC) system
#
# Copyright (C) 2018-2022 Joshua M. Boniface <joshua@boniface.me>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
###############################################################################
# This script provides an example of a PVC monitoring plugin script. It will create
# a simple plugin to check the Ceph cluster health for anomalies, and return a health
# delta reflective of the overall Ceph status (HEALTH_WARN = 10, HEALTH_ERR = 50).
# This script can thus be used as an example or reference implementation of a
# PVC monitoring pluginscript and expanded upon as required.
# A monitoring plugin script must implement the class "MonitoringPluginScript" which
# extends "MonitoringPlugin", providing the 3 functions indicated. Detailed explanation
# of the role of each function is provided in context of the example; see the other
# examples for more potential uses.
# WARNING:
#
# This script will run in the context of the node daemon keepalives as root.
# DO NOT install untrusted, unvetted plugins under any circumstances.
# This import is always required here, as MonitoringPlugin is used by the
# MonitoringPluginScript class
from pvcnoded.objects.MonitoringInstance import MonitoringPlugin
# A monitoring plugin script must always expose its nice name, which must be identical to
# the file name
PLUGIN_NAME = "ceph"
# The MonitoringPluginScript class must be named as such, and extend MonitoringPlugin.
class MonitoringPluginScript(MonitoringPlugin):
def setup(self):
"""
setup(): Perform special setup steps during node daemon startup
This step is optional and should be used sparingly.
"""
pass
def run(self):
"""
run(): Perform the check actions and return a PluginResult object
"""
# Run any imports first
from rados import Rados
from json import loads, dumps
# Connect to the Ceph cluster
try:
ceph_conn = Rados(
conffile=self.config["ceph_config_file"],
conf=dict(keyring=self.config["ceph_admin_keyring"]),
)
ceph_conn.connect(timeout=1)
except Exception as e:
self.log(f"Failed to connect to Ceph cluster: {e}", state="e")
return self.plugin_result
# Get the Ceph cluster health
try:
health_status = loads(
ceph_conn.mon_command(dumps({"prefix": "health", "format": "json"}), b"", timeout=1)[1]
)
ceph_health = health_status["status"]
except Exception as e:
self.log(f"Failed to get health data from Ceph cluster: {e}", state="e")
return self.plugin_result
finally:
ceph_conn.shutdown()
# Get a list of error entries in the health status output
error_entries = health_status["checks"].keys()
# Set the health delta based on the errors presented
if ceph_health == "HEALTH_ERR":
health_delta = 50
message = f"Ceph cluster in ERROR state: {', '.join(error_entries)}"
elif ceph_health == "HEALTH_WARN":
health_delta = 10
message = f"Ceph cluster in WARNING state: {', '.join(error_entries)}"
else:
health_delta = 0
message = "Ceph cluster in OK state"
# Set the health delta in our local PluginResult object
self.plugin_result.set_health_delta(health_delta)
# Set the message in our local PluginResult object
self.plugin_result.set_message(message)
# Set the detailed data in our local PluginResult object
self.plugin_result.set_data(dumps(health_status))
# Return our local PluginResult object
return self.plugin_result
def cleanup(self):
"""
cleanup(): Perform special cleanup steps during node daemon termination
This step is optional and should be used sparingly.
"""
pass

View File

@ -1,158 +0,0 @@
#!/usr/bin/env python3
# dpkg.py - PVC Monitoring example plugin for dpkg status
# Part of the Parallel Virtual Cluster (PVC) system
#
# Copyright (C) 2018-2022 Joshua M. Boniface <joshua@boniface.me>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
###############################################################################
# This script provides an example of a PVC monitoring plugin script. It will create
# a simple plugin to check the system dpkg status is as expected, with no invalid
# packages or obsolete configuration files, and will return a 1 health delta for each
# flaw in invalid packages, upgradable packages, and obsolete config files.
# This script can thus be used as an example or reference implementation of a
# PVC monitoring pluginscript and expanded upon as required.
# A monitoring plugin script must implement the class "MonitoringPluginScript" which
# extends "MonitoringPlugin", providing the 3 functions indicated. Detailed explanation
# of the role of each function is provided in context of the example; see the other
# examples for more potential uses.
# WARNING:
#
# This script will run in the context of the node daemon keepalives as root.
# DO NOT install untrusted, unvetted plugins under any circumstances.
# This import is always required here, as MonitoringPlugin is used by the
# MonitoringPluginScript class
from pvcnoded.objects.MonitoringInstance import MonitoringPlugin
# A monitoring plugin script must always expose its nice name, which must be identical to
# the file name
PLUGIN_NAME = "dpkg"
# The MonitoringPluginScript class must be named as such, and extend MonitoringPlugin.
class MonitoringPluginScript(MonitoringPlugin):
def setup(self):
"""
setup(): Perform special setup steps during node daemon startup
This step is optional and should be used sparingly.
"""
pass
def run(self):
"""
run(): Perform the check actions and return a PluginResult object
"""
# Run any imports first
from re import match
from json import dumps
import daemon_lib.common as pvc_common
# Get Debian version
with open('/etc/debian_version', 'r') as fh:
debian_version = fh.read().strip()
# Get a list of dpkg packages for analysis
retcode, stdout, stderr = pvc_common.run_os_command("/usr/bin/dpkg --list")
# Get a list of installed packages and states
packages = list()
for dpkg_line in stdout.split('\n'):
if match('^[a-z][a-z] ', dpkg_line):
line_split = dpkg_line.split()
package_state = line_split[0]
package_name = line_split[1]
packages.append((package_name, package_state))
count_ok = 0
count_inconsistent = 0
list_inconsistent = list()
for package in packages:
if package[1] == "ii":
count_ok += 1
else:
count_inconsistent += 1
list_inconsistent.append(package[0])
# Get upgradable packages
retcode, stdout, stderr = pvc_common.run_os_command("/usr/bin/apt list --upgradable")
list_upgradable = list()
for apt_line in stdout.split('\n'):
if match('^[a-z][a-z] ', apt_line):
line_split = apt_line.split('/')
package_name = line_split[0]
list_upgradable.append(package_name)
count_upgradable = len(list_upgradable)
# Get obsolete config files (dpkg-* or ucf-* under /etc)
retcode, stdout, stderr = pvc_common.run_os_command("/usr/bin/find /etc -type f -a \( -name '*.dpkg-*' -o -name '*.ucf-*' \)")
obsolete_conffiles = list()
for conffile_line in stdout.split('\n'):
if conffile_line:
obsolete_conffiles.append(conffile_line)
count_obsolete_conffiles = len(obsolete_conffiles)
# Set health_delta based on the results
health_delta = 0
if count_inconsistent > 0:
health_delta += 1
if count_upgradable > 0:
health_delta += 1
if count_obsolete_conffiles > 0:
health_delta += 1
# Set the health delta in our local PluginResult object
self.plugin_result.set_health_delta(health_delta)
# Craft the message
message = f"Debian {debian_version}; Obsolete conffiles: {count_obsolete_conffiles}; Packages valid: {count_ok}, inconsistent: {count_inconsistent}, upgradable: {count_upgradable}"
# Set the message in our local PluginResult object
self.plugin_result.set_message(message)
# Set the detailed data in our local PluginResult object
detailed_data = {
"debian_version": debian_version,
"obsolete_conffiles": obsolete_conffiles,
"inconsistent_packages": list_inconsistent,
"upgradable_packages": list_upgradable,
}
self.plugin_result.set_data(dumps(detailed_data))
# Return our local PluginResult object
return self.plugin_result
def cleanup(self):
"""
cleanup(): Perform special cleanup steps during node daemon termination
This step is optional and should be used sparingly.
"""
pass

View File

@ -1,105 +0,0 @@
#!/usr/bin/env python3
# load.py - PVC Monitoring example plugin for load
# Part of the Parallel Virtual Cluster (PVC) system
#
# Copyright (C) 2018-2022 Joshua M. Boniface <joshua@boniface.me>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
###############################################################################
# This script provides an example of a PVC monitoring plugin script. It will create
# a simple plugin to check the system load against the total number of CPU cores,
# and return a 10 health delta (100 -> 90) if the load average is > 1/2 that number.
# This script can thus be used as an example or reference implementation of a
# PVC monitoring pluginscript and expanded upon as required.
# A monitoring plugin script must implement the class "MonitoringPluginScript" which
# extends "MonitoringPlugin", providing the 3 functions indicated. Detailed explanation
# of the role of each function is provided in context of the example; see the other
# examples for more potential uses.
# WARNING:
#
# This script will run in the context of the node daemon keepalives as root.
# DO NOT install untrusted, unvetted plugins under any circumstances.
# This import is always required here, as MonitoringPlugin is used by the
# MonitoringPluginScript class
from pvcnoded.objects.MonitoringInstance import MonitoringPlugin
# A monitoring plugin script must always expose its nice name, which must be identical to
# the file name
PLUGIN_NAME = "load"
# The MonitoringPluginScript class must be named as such, and extend MonitoringPlugin.
class MonitoringPluginScript(MonitoringPlugin):
def setup(self):
"""
setup(): Perform special setup steps during node daemon startup
This step is optional and should be used sparingly.
"""
pass
def run(self):
"""
run(): Perform the check actions and return a PluginResult object
"""
# Run any imports first
from os import getloadavg
from psutil import cpu_count
# Get the current 1-minute system load average
load_average = getloadavg()[0]
# Get the number of CPU cores
cpu_cores = cpu_count()
# Check that the load average is greater or equal to the cpu count
if load_average > float(cpu_cores):
# Set the health delta to 10 (subtract 10 from the total of 100)
health_delta = 10
# Craft a message that can be used by the clients
message = f"Current load is {load_average} out of {cpu_cores} CPU cores"
else:
# Set the health delta to 0 (no change)
health_delta = 0
# Craft a message that can be used by the clients
message = f"Current load is {load_average} out of {cpu_cores} CPU cores"
# Set the health delta in our local PluginResult object
self.plugin_result.set_health_delta(health_delta)
# Set the message in our local PluginResult object
self.plugin_result.set_message(message)
# Return our local PluginResult object
return self.plugin_result
def cleanup(self):
"""
cleanup(): Perform special cleanup steps during node daemon termination
This step is optional and should be used sparingly.
"""
pass

View File

@ -128,8 +128,6 @@ pvc:
configuration:
# directories: PVC system directories
directories:
# plugin_directory: Directory containing node monitoring plugins
plugin_directory: "/usr/share/pvc/plugins"
# dynamic_directory: Temporary in-memory directory for active configurations
dynamic_directory: "/run/pvc"
# log_directory: Logging directory

View File

@ -27,7 +27,6 @@ import pvcnoded.util.services
import pvcnoded.util.libvirt
import pvcnoded.util.zookeeper
import pvcnoded.objects.MonitoringInstance as MonitoringInstance
import pvcnoded.objects.DNSAggregatorInstance as DNSAggregatorInstance
import pvcnoded.objects.MetadataAPIInstance as MetadataAPIInstance
import pvcnoded.objects.VMInstance as VMInstance
@ -49,7 +48,7 @@ import re
import json
# Daemon version
version = "0.9.61"
version = "0.9.59"
##########################################################
@ -59,7 +58,6 @@ version = "0.9.61"
def entrypoint():
keepalive_timer = None
monitoring_instance = None
# Get our configuration
config = pvcnoded.util.config.get_configuration()
@ -206,7 +204,7 @@ def entrypoint():
# Define a cleanup function
def cleanup(failure=False):
nonlocal logger, zkhandler, keepalive_timer, d_domain, monitoring_instance
nonlocal logger, zkhandler, keepalive_timer, d_domain
logger.out("Terminating pvcnoded and cleaning up", state="s")
@ -255,13 +253,6 @@ def entrypoint():
except Exception:
pass
# Clean up any monitoring plugins that have cleanup
try:
logger.out("Performing monitoring plugin cleanup", state="s")
monitoring_instance.run_cleanups()
except Exception:
pass
# Set stop state in Zookeeper
zkhandler.write([(("node.state.daemon", config["node_hostname"]), "stop")])
@ -1024,14 +1015,9 @@ def entrypoint():
state="i",
)
# Set up the node monitoring instance
monitoring_instance = MonitoringInstance.MonitoringInstance(
zkhandler, config, logger, this_node
)
# Start keepalived thread
keepalive_timer = pvcnoded.util.keepalive.start_keepalive_timer(
logger, config, zkhandler, this_node, monitoring_instance
logger, config, zkhandler, this_node
)
# Tick loop; does nothing since everything is async

View File

@ -1,357 +0,0 @@
#!/usr/bin/env python3
# PluginInstance.py - Class implementing a PVC monitoring instance
# Part of the Parallel Virtual Cluster (PVC) system
#
# Copyright (C) 2018-2022 Joshua M. Boniface <joshua@boniface.me>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, version 3.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
###############################################################################
import concurrent.futures
import time
import importlib.util
from os import walk
from datetime import datetime
class PluginResult(object):
def __init__(self, zkhandler, config, logger, this_node, plugin_name):
self.zkhandler = zkhandler
self.config = config
self.logger = logger
self.this_node = this_node
self.plugin_name = plugin_name
self.current_time = int(time.time())
self.health_delta = 0
self.message = None
self.data = None
self.runtime = "0.00"
def set_health_delta(self, new_delta):
self.health_delta = new_delta
def set_message(self, new_message):
self.message = new_message
def set_data(self, new_data):
self.data = new_data
def set_runtime(self, new_runtime):
self.runtime = new_runtime
def to_zookeeper(self):
self.zkhandler.write(
[
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.name",
self.plugin_name,
),
self.plugin_name,
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.last_run",
self.plugin_name,
),
self.current_time,
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.health_delta",
self.plugin_name,
),
self.health_delta,
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.message",
self.plugin_name,
),
self.message,
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.data",
self.plugin_name,
),
self.data,
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.runtime",
self.plugin_name,
),
self.runtime,
),
]
)
class MonitoringPlugin(object):
def __init__(self, zkhandler, config, logger, this_node, plugin_name):
self.zkhandler = zkhandler
self.config = config
self.logger = logger
self.this_node = this_node
self.plugin_name = plugin_name
self.plugin_result = PluginResult(
self.zkhandler,
self.config,
self.logger,
self.this_node,
self.plugin_name,
)
#
# Helper functions; exposed to child MonitoringPluginScript instances
#
def log(self, message, state="d"):
"""
Log a message to the PVC logger instance using the plugin name as a prefix
Takes "state" values as defined by the PVC logger instance, defaulting to debug:
"d": debug
"i": informational
"t": tick/keepalive
"w": warning
"e": error
"""
if state == "d" and not self.config["debug"]:
return
self.logger.out(message, state=state, prefix=self.plugin_name)
#
# Primary class functions; implemented by the individual plugins
#
def setup(self):
"""
setup(): Perform setup of the plugin; run once during daemon startup
OPTIONAL
"""
pass
def run(self):
"""
run(): Run the plugin, returning a PluginResult object
"""
return self.plugin_result
def cleanup(self):
"""
cleanup(): Clean up after the plugin; run once during daemon shutdown
OPTIONAL
"""
pass
class MonitoringInstance(object):
def __init__(self, zkhandler, config, logger, this_node):
self.zkhandler = zkhandler
self.config = config
self.logger = logger
self.this_node = this_node
# Get a list of plugins from the plugin_directory
plugin_files = next(walk(self.config["plugin_directory"]), (None, None, []))[
2
] # [] if no file
self.all_plugins = list()
self.all_plugin_names = list()
# Load each plugin file into the all_plugins list
for plugin_file in sorted(plugin_files):
try:
self.logger.out(
f"Loading monitoring plugin from {self.config['plugin_directory']}/{plugin_file}",
state="i",
)
loader = importlib.machinery.SourceFileLoader(
"plugin_script", f"{self.config['plugin_directory']}/{plugin_file}"
)
spec = importlib.util.spec_from_loader(loader.name, loader)
plugin_script = importlib.util.module_from_spec(spec)
spec.loader.exec_module(plugin_script)
plugin = plugin_script.MonitoringPluginScript(
self.zkhandler,
self.config,
self.logger,
self.this_node,
plugin_script.PLUGIN_NAME,
)
self.all_plugins.append(plugin)
self.all_plugin_names.append(plugin.plugin_name)
# Create plugin key
self.zkhandler.write(
[
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.name",
plugin.plugin_name,
),
plugin.plugin_name,
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.last_run",
plugin.plugin_name,
),
"0",
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.health_delta",
plugin.plugin_name,
),
"0",
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.message",
plugin.plugin_name,
),
"Initializing",
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.data",
plugin.plugin_name,
),
None,
),
(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin.runtime",
plugin.plugin_name,
),
"0.00",
),
]
)
self.logger.out(
f"Successfully loaded monitoring plugin '{plugin.plugin_name}'",
state="o",
)
except Exception as e:
self.logger.out(
f"Failed to load monitoring plugin: {e}",
state="w",
)
self.zkhandler.write(
[
(
("node.monitoring.plugins", self.this_node.name),
self.all_plugin_names,
),
]
)
# Clean up any old plugin data for which a plugin file no longer exists
for plugin_key in self.zkhandler.children(
("node.monitoring.data", self.this_node.name)
):
if plugin_key not in self.all_plugin_names:
self.zkhandler.delete(
(
"node.monitoring.data",
self.this_node.name,
"monitoring_plugin",
plugin_key,
)
)
def run_plugin(self, plugin):
time_start = datetime.now()
result = plugin.run()
time_end = datetime.now()
time_delta = time_end - time_start
runtime = "{:0.02f}".format(time_delta.total_seconds())
result.set_runtime(runtime)
self.logger.out(
result.message, state="t", prefix=f"{plugin.plugin_name} ({runtime}s)"
)
result.to_zookeeper()
return result
def run_plugins(self):
total_health = 100
self.logger.out("Running monitoring plugins:", state="t")
plugin_results = list()
with concurrent.futures.ThreadPoolExecutor(max_workers=99) as executor:
to_future_plugin_results = {
executor.submit(self.run_plugin, plugin): plugin
for plugin in self.all_plugins
}
for future in concurrent.futures.as_completed(to_future_plugin_results):
plugin_results.append(future.result())
for result in plugin_results:
if result is not None:
total_health -= result.health_delta
if total_health > 90:
health_colour = self.logger.fmt_green
elif total_health > 50:
health_colour = self.logger.fmt_yellow
else:
health_colour = self.logger.fmt_red
self.logger.out(
f"System health: {health_colour}{total_health}/100{self.logger.fmt_end}",
state="t",
)
def run_cleanup(self, plugin):
return plugin.cleanup()
def run_cleanups(self):
with concurrent.futures.ThreadPoolExecutor(max_workers=99) as executor:
to_future_plugin_results = {
executor.submit(self.run_cleanup, plugin): plugin
for plugin in self.all_plugins
}
for future in concurrent.futures.as_completed(to_future_plugin_results):
# This doesn't do anything, just lets us wait for them all to complete
pass

View File

@ -180,9 +180,6 @@ def get_configuration():
raise MalformedConfigurationError(e)
config_directories = {
"plugin_directory": o_directories.get(
"plugin_directory", "/usr/share/pvc/plugins"
),
"dynamic_directory": o_directories.get("dynamic_directory", None),
"log_directory": o_directories.get("log_directory", None),
"console_log_directory": o_directories.get("console_log_directory", None),

View File

@ -51,7 +51,7 @@ libvirt_vm_states = {
}
def start_keepalive_timer(logger, config, zkhandler, this_node, monitoring_instance):
def start_keepalive_timer(logger, config, zkhandler, this_node):
keepalive_interval = config["keepalive_interval"]
logger.out(
f"Starting keepalive timer ({keepalive_interval} second interval)", state="s"
@ -59,7 +59,7 @@ def start_keepalive_timer(logger, config, zkhandler, this_node, monitoring_insta
keepalive_timer = BackgroundScheduler()
keepalive_timer.add_job(
node_keepalive,
args=(logger, config, zkhandler, this_node, monitoring_instance),
args=(logger, config, zkhandler, this_node),
trigger="interval",
seconds=keepalive_interval,
)
@ -97,6 +97,29 @@ def collect_ceph_stats(logger, config, zkhandler, this_node, queue):
logger.out("Failed to open connection to Ceph cluster: {}".format(e), state="e")
return
if debug:
logger.out("Getting health stats from monitor", state="d", prefix="ceph-thread")
# Get Ceph cluster health for local status output
command = {"prefix": "health", "format": "json"}
try:
health_status = json.loads(
ceph_conn.mon_command(json.dumps(command), b"", timeout=1)[1]
)
ceph_health = health_status["status"]
except Exception as e:
logger.out("Failed to obtain Ceph health data: {}".format(e), state="e")
ceph_health = "HEALTH_UNKN"
if ceph_health in ["HEALTH_OK"]:
ceph_health_colour = logger.fmt_green
elif ceph_health in ["HEALTH_UNKN"]:
ceph_health_colour = logger.fmt_cyan
elif ceph_health in ["HEALTH_WARN"]:
ceph_health_colour = logger.fmt_yellow
else:
ceph_health_colour = logger.fmt_red
# Primary-only functions
if this_node.router_state == "primary":
if debug:
@ -385,6 +408,8 @@ def collect_ceph_stats(logger, config, zkhandler, this_node, queue):
ceph_conn.shutdown()
queue.put(ceph_health_colour)
queue.put(ceph_health)
queue.put(osds_this_node)
if debug:
@ -623,7 +648,7 @@ def collect_vm_stats(logger, config, zkhandler, this_node, queue):
# Keepalive update function
def node_keepalive(logger, config, zkhandler, this_node, monitoring_instance):
def node_keepalive(logger, config, zkhandler, this_node):
debug = config["debug"]
if debug:
logger.out("Keepalive starting", state="d", prefix="main-thread")
@ -752,14 +777,16 @@ def node_keepalive(logger, config, zkhandler, this_node, monitoring_instance):
if config["enable_storage"]:
try:
osds_this_node = ceph_thread_queue.get(
timeout=(config["keepalive_interval"] - 1)
ceph_health_colour = ceph_thread_queue.get(
timeout=config["keepalive_interval"]
)
ceph_health = ceph_thread_queue.get(timeout=config["keepalive_interval"])
osds_this_node = ceph_thread_queue.get(timeout=config["keepalive_interval"])
except Exception:
logger.out("Ceph stats queue get exceeded timeout, continuing", state="w")
ceph_health_colour = logger.fmt_cyan
ceph_health = "UNKNOWN"
osds_this_node = "?"
else:
osds_this_node = "0"
# Set our information in zookeeper
keepalive_time = int(time.time())
@ -812,8 +839,8 @@ def node_keepalive(logger, config, zkhandler, this_node, monitoring_instance):
if config["log_keepalive_cluster_details"]:
logger.out(
"{bold}Maintenance:{nofmt} {maint} "
"{bold}Node VMs:{nofmt} {domcount} "
"{bold}Node OSDs:{nofmt} {osdcount} "
"{bold}Active VMs:{nofmt} {domcount} "
"{bold}Networks:{nofmt} {netcount} "
"{bold}Load:{nofmt} {load} "
"{bold}Memory [MiB]: VMs:{nofmt} {allocmem} "
"{bold}Used:{nofmt} {usedmem} "
@ -822,7 +849,7 @@ def node_keepalive(logger, config, zkhandler, this_node, monitoring_instance):
nofmt=logger.fmt_end,
maint=this_node.maintenance,
domcount=this_node.domains_count,
osdcount=osds_this_node,
netcount=len(zkhandler.children("base.network")),
load=this_node.cpuload,
freemem=this_node.memfree,
usedmem=this_node.memused,
@ -830,6 +857,22 @@ def node_keepalive(logger, config, zkhandler, this_node, monitoring_instance):
),
state="t",
)
if config["enable_storage"] and config["log_keepalive_storage_details"]:
logger.out(
"{bold}Ceph cluster status:{nofmt} {health_colour}{health}{nofmt} "
"{bold}Total OSDs:{nofmt} {total_osds} "
"{bold}Node OSDs:{nofmt} {node_osds} "
"{bold}Pools:{nofmt} {total_pools} ".format(
bold=logger.fmt_bold,
health_colour=ceph_health_colour,
nofmt=logger.fmt_end,
health=ceph_health,
total_osds=len(zkhandler.children("base.osd")),
node_osds=osds_this_node,
total_pools=len(zkhandler.children("base.pool")),
),
state="t",
)
# Look for dead nodes and fence them
if not this_node.maintenance:
@ -875,7 +918,5 @@ def node_keepalive(logger, config, zkhandler, this_node, monitoring_instance):
[(("node.state.daemon", node_name), "dead")]
)
monitoring_instance.run_plugins()
if debug:
logger.out("Keepalive finished", state="d", prefix="main-thread")