Commit Graph

356 Commits

Author SHA1 Message Date
Joshua Boniface d42bb74dc9 Use explicit acquire/release instead of with
The with blocks did not seem to work as expected. Go back to exclusive
locks as well since these are more consistent.
2020-10-21 11:38:23 -04:00
Joshua Boniface 42c5f84ba7 Do multiple lock attempts 2020-10-21 11:21:37 -04:00
Joshua Boniface 88556f4a33 Convert from exclusive to write lock 2020-10-21 11:12:36 -04:00
Joshua Boniface 3839040092 Add exclusive lock function 2020-10-21 10:46:41 -04:00
Joshua Boniface 84ade53fae Add locks for VM state changes
Use exclusive locks during API events which change VM state. This is
fairly critical to avoid potential duplicate updates. Only implemented
for these specifically required functions to avoid major performance
hits elsewhere.
2020-10-21 10:40:00 -04:00
Joshua Boniface 72f47f216a Revert "Add locking in common zkhander"
This reverts commit 53c0d2b4f6.

This resulted in a massive performance hit and some inconsistent
behaviour. Revert for now an re-investigate later.
2020-10-21 03:49:13 -04:00
Joshua Boniface 53c0d2b4f6 Add locking in common zkhander
Ensures that every changed made here is locked, thus preventing
duplicate updates, etc.
2020-10-21 03:17:18 -04:00
Joshua Boniface e7ab1bfddd Add cluster overprovision determination
Adds a check of (n-1) memory overprovisioning. (n-1) is considered to be
the configuration that excludes the "largest" node. The cluster will
report degraded when in this state.
2020-10-18 14:57:22 -04:00
Joshua Boniface f749633f7c Use provisioned memory for mem migration selector
Use the new "provisioned" memory field, instead of the "allocated"
memory field, to determine the optimal node when using the "mem"
migration selector. This will take into account non-running VMs in the
calculation as well as running VMs.
2020-10-18 14:17:15 -04:00
Joshua Boniface a4b80be5ed Add provisioned memory to node info
Adds a separate field to the node memory, "provisioned", which totals
the amount of memory provisioned to all VMs on the node, regardless of
state, and in contrast to "allocated" which only counts running VMs.

Allows for the detection of potential overprovisioned states when
factoring in non-running VMs.

Includes the supporting code to get this data, since the original
implementation of VM memory selection was dependent on the VM being
running and getting this from libvirt. Now, if the VM is not active, it
gets this from the domain XML instead.
2020-10-18 14:17:15 -04:00
Joshua Boniface e4891831ce Better handle missing elements from net config
Prevents situations with an un-editable, invalid config being stuck.
2020-08-21 10:27:45 -04:00
Joshua Boniface 1967034493 Use get() for all remaining VM XML gets
Prevents KeyErrors and such.
2020-08-21 10:10:13 -04:00
Joshua Boniface 15e78aa9f0 Add status information in cluster status
Provide textual explanations for the degraded status, including
specific node/VM/OSD issues as well as detailed Ceph health. "Single
pane of glass" mentality.
2020-08-17 12:25:23 -04:00
Joshua Boniface 2b4d980685 Display Ceph health in PVC status as well
Makes this output a little more realistic and allows proper monitoring
of the Ceph cluster status (separate from the PVC status which is
tracking only OSD up/in state).
2020-08-13 15:10:57 -04:00
Joshua Boniface 5b5b7d2276 Improve the conditional so it will always work 2020-08-11 23:08:40 -04:00
Joshua Boniface 0468eeb531 Support live resizing of running disk volumes
This wasn't happening automatically, nor does it happen with qemu-img
commands, so we have to manually trigger a libvirt blockResize against
the volume. This setup is a little roundabout but seems to work fine.
2020-08-11 21:46:12 -04:00
Joshua Boniface 142743b2c0 Fix erroneous comma 2020-08-05 10:34:30 -04:00
Joshua Boniface 8fc5299d38 Avoid failing if CPU features are missing 2020-07-08 12:32:42 -04:00
Joshua Boniface 37a58d35e8 Implement limiting of node output
Closes #98
2020-06-25 11:51:53 -04:00
Joshua Boniface deaf138e45 Add stats to VM information 2020-06-07 00:42:11 -04:00
Joshua Boniface 654a3cb7fa Improve debug output and use ceph df util data 2020-06-06 22:52:49 -04:00
Joshua Boniface fba39cb739 Fix broken sorting for pools and volumes 2020-06-06 21:28:54 -04:00
Joshua Boniface ce60836c34 Allow enforcement of live migration
Provides a CLI and API argument to force live migration, which triggers
a new VM state "migrate-live". The node daemon VMInstance during migrate
will read this flag from the state and, if enforced, will not trigger a
shutdown migration.

Closes #95
2020-06-06 12:00:44 -04:00
Joshua Boniface f61d443773 Allow move of migrated VM to current node
Will make the migrate permanent instead of throwing an error.

Fixes #96
2020-06-06 11:25:10 -04:00
Joshua Boniface 0fab7072ac Sort all Ceph lists by numeric ID 2020-05-29 13:31:18 -04:00
Joshua Boniface 2d507f8b42 Ensure rbdlist is updated when modifying VM config 2020-05-12 11:08:47 -04:00
Joshua Boniface 3cf90c46ad Correct bad handling of static reservations 2020-05-09 10:20:06 -04:00
Joshua Boniface 7b2180b626 Get both reservations in leases by default 2020-05-09 10:05:55 -04:00
Joshua Boniface 44efd66f2c Fix error renaming keys
This function was not implemented and thus failed; implements it.
2020-03-30 21:38:18 -04:00
Joshua Boniface 8678dedfea Revert "Implement wait for node coordinator transition"
This reverts commit 0aefafa7f7.

This does not work since the API goes away during the transition.

References #72
2020-02-19 10:50:21 -05:00
Joshua Boniface 0aefafa7f7 Implement wait for node coordinator transition
References #72
2020-02-19 10:50:04 -05:00
Joshua Boniface 6db4df51c0 Remove obsolete follow_console_log function 2020-02-19 10:19:49 -05:00
Joshua Boniface 5ddf72855b Clean up obsolete is_cli flags 2020-02-19 10:18:41 -05:00
Joshua Boniface 0e05ce8b07 Use correct wording of "shut down" 2020-02-19 10:04:58 -05:00
Joshua Boniface 07577a52a9 Implement wait support for various VM commands
Implements wait support for VM restart, shutdown, move, migrate, and
unmigrate commands, similar to node flush/node unflush.

Includes some additional refactoring of the move command to make its
operation identical to migrate, only without recording the previous
node.

References #72
2020-02-19 09:45:31 -05:00
Joshua Boniface 45040a5635 Make wait flag optional on flush functions
References #72
2020-02-19 09:44:38 -05:00
Joshua Boniface 7c99618752 Correct handling of bare bytes values 2020-02-17 12:32:20 -05:00
Joshua Boniface 670596ed8e Add dead node states to status 2020-02-15 18:51:02 -05:00
Joshua Boniface 95c59c2b39 Support non-extension fromhuman for normalization 2020-02-11 20:31:56 -05:00
Joshua Boniface e419855911 Support converting types during upload
Allow the user to specify other, non-raw files and upload them,
performing a conversion with qemu-img convert and a temporary block
device as a shim (since qemu-img can't use FIFOs).

Also ensures that the target volume exists before proceeding.

Addresses #68
2020-02-09 20:29:12 -05:00
Joshua Boniface 49e5ce1176 Support uploading disk images to volumes in API
Addresses #68
2020-02-09 13:45:04 -05:00
Joshua Boniface 92df125a77 Add missing library imports in common functions 2020-02-08 23:43:49 -05:00
Joshua Boniface 7ace5b5056 Remove /ceph/cmd pipe for (most) Ceph commands
Addresses #80
2020-02-08 23:40:02 -05:00
Joshua Boniface eeb8879f73 Move run_os_command to common functions
References #80
2020-02-08 23:33:15 -05:00
Joshua Boniface 4505b239eb Rename API and common Debian packages
Closes #79
2020-02-08 18:50:38 -05:00
Joshua Boniface 199b4bd40e Remove obsolete daemon-common section 2018-10-14 16:30:07 -04:00
Joshua Boniface f2431f179e Correct multiple issues with DHCP and add firewall control logic 2018-10-08 23:53:41 -04:00
Joshua Boniface 168435a827 Correct signals problem 2018-10-03 23:45:19 -04:00
Joshua Boniface 3e5e50e438 Handle daemons for dnsmasq 2018-10-03 23:17:59 -04:00
Joshua Boniface 505a951a7d Add delete function to zkhandler 2018-09-30 02:52:22 -04:00
Joshua Boniface fa1e3591cb Use much nicer updated zkhander from client library 2018-09-29 01:31:09 -04:00
Joshua Boniface 0f9637cb69 Make the IP failover work including threaded background os commands 2018-09-24 04:08:35 -04:00
Joshua Boniface b3259a3dcc Fix up some bugs and split out common daemon functions 2018-09-23 21:19:56 -04:00
Joshua Boniface 513de96626 Major refactor to separate out and standardize libraries 2018-09-20 03:43:34 -04:00
Joshua Boniface ac2ae4b3e3 Tweak some more things in the Daemon to reflect split 2018-09-12 12:43:18 -04:00
Joshua Boniface b6c7153401 Bump version and split common libraries into separate package 2018-09-12 12:11:59 -04:00