parallelvirtualcluster
/
pvc
1
0
Fork 0
Code Issues 11 Pull Requests Projects Releases 109 Wiki Activity
2,512 Commits 1 Branch 111 Tags 27 MiB
Commit Graph

244 Commits

Author SHA1 Message Date
Joshua Boniface 7e42118e6f Adjust lock schema in NodeInstance and VMInstance 
Removes a superfluous lock and puts the sync_lock keys in more usable
places.
 2021-06-09 22:51:00 -04:00
Joshua Boniface 24663a3333 Add missing VM schema entry 2021-06-09 22:12:24 -04:00
Joshua Boniface a9a57533a7 Integrate schema handling within ZKHandler 
Abstracts away the schema management, especially when doing actions, to
prevent duplication in other areas.
 2021-06-09 13:23:57 -04:00
Joshua Boniface 76c37e6628 Tweak some field names slightly and add missing 2021-06-09 09:58:18 -04:00
Joshua Boniface 0a04adf8f9 Allow empty sub_paths 2021-06-09 01:54:29 -04:00
Joshua Boniface f2b55ba937 Fix some bugs with migrations 2021-06-09 00:04:16 -04:00
Joshua Boniface 5540bdc86b Add automatic schema upgrade to nodes 
Performs an automatic schema upgrade when all nodes are updated to the
latest version.

Addresses #129
 2021-06-08 23:35:39 -04:00
Joshua Boniface 3c102b3769 Add per-node schema tracking 
This will allow nodes to start with their own schema versions, and then
be updated simultaneously by the API.

References #129
 2021-06-08 23:35:39 -04:00
Joshua Boniface 602dd7b714 Update version 0 schema and add full validation 
Addresses #129
 2021-06-08 23:35:39 -04:00
Joshua Boniface 126f0742cd Add Zookeeper schema manager to zkhandler 
Adds a new class, ZKSchema, to handle schema management in Zookeeper in
an automated and consistent way. This should solve several issues:

1. Pain in managing changes to ZK keys
2. Pain in handling those changes during live upgrades
3. Simplifying the codebase to remove hardcoded ZK paths

The current master schema for PVC 0.9.19 is committed as version 0.

Addresses #129
 2021-06-08 23:35:39 -04:00
Joshua Boniface f0dc0fc782 Avoid duplicating maintenance state change 
This makes no functional difference, but is technically more correct.
 2021-06-05 01:36:40 -04:00
Joshua Boniface 5d88e92acc Avoid returning errors with duplicate router mode 
Like the previous (new) flush change, these shouldn't be errors, but
simply information "what you want is already done" messages.
 2021-06-05 01:14:31 -04:00
Joshua Boniface 505c109875 Avoid re-flush or re-ready nodes if unnecessary 2021-06-05 01:08:32 -04:00
Joshua Boniface 7dea5d2fac Move logger to common, fix buffering 2021-06-01 18:50:26 -04:00
Joshua Boniface 9764090d6d Merge node common with daemon common 2021-06-01 12:22:11 -04:00
Joshua Boniface f73c433fc7 Remove useless try and import 2021-06-01 12:05:17 -04:00
Joshua Boniface a48bf2d71e More gracefully handle none selectors 
Allow selection of "none" as the node selector, and handle this by
always using the cluster default instead of writing it in.
 2021-06-01 11:13:13 -04:00
Joshua Boniface 33a54cf7f2 Move configuration keys to /config tree 2021-06-01 10:48:55 -04:00
Joshua Boniface d3b5b5236a Remove transactional delete 
This just doesn't work due to the darn limit on recursive deletes in
transactions.
 2021-05-31 19:22:01 -04:00
Joshua Boniface 8625e9bd3e Update Delete to recursive method 2021-05-31 03:14:09 -04:00
Joshua Boniface ed4f84a3ec Add log handling and persistent listener 2021-05-31 03:14:09 -04:00
Joshua Boniface 9be426507a Fix erroneous lock calls 2021-05-30 15:31:17 -04:00
Joshua Boniface 58a5b00aa1 Remove extraneous zkhandler reference 2021-05-30 01:01:40 -04:00
Joshua Boniface b4f2cf879e Rework vm library for new zkhandler 2021-05-29 21:17:19 -04:00
Joshua Boniface 3603b782c0 Rework node library for new zkhandler 2021-05-29 20:56:21 -04:00
Joshua Boniface 62cb72b62f Rework network library for new zkhandler 2021-05-29 20:53:42 -04:00
Joshua Boniface b186a75b4e Rework common library for new zkhandler 2021-05-29 20:35:28 -04:00
Joshua Boniface 6205dba451 Rework cluster library for new zkhandler 2021-05-29 20:32:20 -04:00
Joshua Boniface 688d1a6ae2 Rework ceph library for new zkhandler 2021-05-29 20:29:51 -04:00
Joshua Boniface f82da03a62 Add first wrappers and exceptions 2021-05-28 00:19:39 -04:00
Joshua Boniface fef230ad98 Implement class-based version of zkhander 2021-05-27 22:50:00 -04:00
Joshua Boniface f46c2e7f6a Implement VM rename functionality 
Closes #125
 2021-05-23 17:21:19 -04:00
Joshua Boniface 0bf276fd51 Update copyright year in headers 2021-03-25 17:01:55 -04:00
Joshua Boniface f4ec161aa2 Update file copyright header. 
Remove the option to select a later version of the GPL.
 2021-03-25 16:58:02 -04:00
Joshua Boniface 8c4c1aba38 Revert "Add object-map to default RBD volume features" 
This reverts commit 00e9cca6f0.

This causes RBD mapping to fail on Debian 10. Reverting but keeping in
history for future reference.
 2021-02-28 01:57:30 -05:00
Joshua Boniface 00e9cca6f0 Add object-map to default RBD volume features 
This allows for much quicker image deletes especially for empty images.
 2021-02-17 13:57:28 -05:00
Joshua Boniface ab05e0f3db Add B suffix back to bare bytes values 2021-02-17 11:37:36 -05:00
Joshua Boniface 9291ce6ffc Correct output of fromhuman and add size compare 
Ensures that the bytes_tohuman returns an integer to avoid the hacky
workaround of stripping off the B.

Adds a verification on the size of a new volume, that it is not larger
than the free space of the pool to prevent errors/excessively-large
volumes from being created.

Closes #120
 2021-02-17 11:27:26 -05:00
Joshua Boniface 0e4bece441 Add missing inc/dec of snapshot_count 2021-02-14 17:02:49 -05:00
Joshua Boniface b33c0ab0e2 Final final fix to snapshot ordering 2021-02-14 16:48:42 -05:00
Joshua Boniface 150c61d226 Actually fix sorting issue 2021-02-14 16:41:59 -05:00
Joshua Boniface 26b0a8b5c1 Fix sorting bug with snapshot list 2021-02-14 16:34:43 -05:00
Joshua Boniface 1b6613c280 Add live VNC information to domain output 
Sets in the node daemon, returns via the API, and shows in the CLI,
information about the live VNC listen address and port for VNC-enabled
VMs.

Closes #115
 2020-12-20 16:00:55 -05:00
Joshua Boniface f58e95e4c1 Fix bugs in modifying networks 
1. Use a consistent "is not None" to verify records are changing.
2. Fix bug where IPv6 network had no remove setter (it is now a blank
string, the first thing I would expect).
3. 1 fixes a bug whereby it was impossible to unset DHCPv4 status.
 2020-12-04 04:15:04 -05:00
Joshua Boniface 0f8e5c6536 Add VNI to VM network API list 
Saves some processing on API clients.
 2020-12-01 04:44:33 -05:00
Joshua Boniface 593810e53e Add volume_count to pool API data 2020-12-01 03:40:41 -05:00
Joshua Boniface 6912bd7a2a Add missing return statement for vm_modify 2020-11-08 00:57:13 -05:00
Joshua Boniface 0bf130077c Lint: W605 invalid escape sequence '<char>' 
The previous attempt to correct these with character classes failed.
Instead, use the proper `r'blah'` regex formatting.
 2020-11-07 17:41:09 -05:00
Joshua Boniface 3ed97d209f Revert "Lint: W605 invalid escape sequence '\^'" 
This reverts commit 639937f9c2.
 2020-11-07 17:39:34 -05:00
Joshua Boniface f33398458e Revert "Lint: W605 invalid escape sequence '\$'" 
This reverts commit 76b1cafdcc.
 2020-11-07 17:38:03 -05:00
Joshua Boniface 78eedf0d2e Remove more dead code 2020-11-07 15:20:36 -05:00
Joshua Boniface 6f66b77a00 Lint: E121/E126 continuation line under/over-indented for hanging indent 2020-11-07 15:06:21 -05:00
Joshua Boniface 9dd2a24ce9 Lint: E226 missing whitespace around arithmetic operator 2020-11-07 14:50:57 -05:00
Joshua Boniface 260b39ebf2 Lint: E302 expected 2 blank lines, found X 2020-11-07 14:45:24 -05:00
Joshua Boniface e9643651f7 Lint: F401 'daemon_lib.ansiprint' imported but unused 
Removing further obsolete code from daemon-lib
 2020-11-07 13:50:27 -05:00
Joshua Boniface 449f766a2c Lint: F821 undefined name 'zk_conn' 
Actually removes these entire sections of dead code; the daemon-common
library does no formatting at all since it is no longer used by the CLI.
 2020-11-07 13:45:26 -05:00
Joshua Boniface 99d723dd8b Lint: F821 undefined name 'count' 2020-11-07 13:39:51 -05:00
Joshua Boniface 6cf7f178a6 Lint: F821 undefined name 'time' 2020-11-07 13:38:54 -05:00
Joshua Boniface 145b1531a4 Lint: F821 undefined name 'name' 2020-11-07 13:38:16 -05:00
Joshua Boniface a4823bbb9c Lint: F821 undefined name 'volume' 2020-11-07 13:37:55 -05:00
Joshua Boniface 516e36686c Lint: F821 undefined name 'name' 2020-11-07 13:37:24 -05:00
Joshua Boniface dc1269ffc2 Lint: F821 undefined name 'new_name' 2020-11-07 13:36:56 -05:00
Joshua Boniface 146c969ef7 Lint: F821 undefined name 'logger' 2020-11-07 13:36:28 -05:00
Joshua Boniface f304547ad5 Lint: F821 undefined name 'time' 2020-11-07 13:35:12 -05:00
Joshua Boniface 961ebb4c01 Lint: E305 expected 2 blank lines after class or function definition, found X 2020-11-07 13:17:49 -05:00
Joshua Boniface d2490419c5 Lint: E202 whitespace before ']' 2020-11-07 13:02:54 -05:00
Joshua Boniface d2e5ede399 Lint: E202 whitespace before ')' 2020-11-07 12:58:54 -05:00
Joshua Boniface 3f242cd437 Lint: E202 whitespace before '}' 2020-11-07 12:57:42 -05:00
Joshua Boniface b7daa8e1f6 E201 whitespace after '[' 2020-11-07 12:39:59 -05:00
Joshua Boniface c88965e898 Lint: E201 whitespace after '(' 2020-11-07 12:39:27 -05:00
Joshua Boniface e333f2b935 Lint: E201 whitespace after '{' 2020-11-07 12:38:31 -05:00
Joshua Boniface 6ab261f4cb Lint: E251 unexpected spaces around keyword / parameter equals 2020-11-07 12:11:53 -05:00
Joshua Boniface 5b3ee363b2 Lint: E222 multiple spaces after operator 2020-11-07 12:10:24 -05:00
Joshua Boniface fad27a7f4d Lint: E131 continuation line unaligned for hanging indent 2020-11-06 22:29:49 -05:00
Joshua Boniface 69858788c1 Lint: E227 missing whitespace around bitwise or shift operator 2020-11-06 21:41:10 -05:00
Joshua Boniface 2eef6a1c21 Lint: E265 block comment should start with '# ' 2020-11-06 21:32:17 -05:00
Joshua Boniface 4b47a2424c Lint: E303 too many blank lines (2) 2020-11-06 21:16:52 -05:00
Joshua Boniface cb2defbde9 Lint: W391 blank line at end of file 2020-11-06 21:14:19 -05:00
Joshua Boniface 5da314902f Lint: F841 local variable '<variable>' is assigned to but never used 2020-11-06 21:13:13 -05:00
Joshua Boniface aecb845d6a Lint: E713 test for membership should be 'not in' 2020-11-06 20:37:52 -05:00
Joshua Boniface 72ae149cf1 Lint: E712 comparison to True should be 'if cond is True:' or 'if cond:' 2020-11-06 19:35:19 -05:00
Joshua Boniface fb4aafcea9 Lint: E111 indentation is not a multiple of four 2020-11-06 19:26:22 -05:00
Joshua Boniface 2d8f684fc8 Lint: E722 do not use bare 'except' (2) 2020-11-06 19:24:10 -05:00
Joshua Boniface d9e7b7ec15 Lint: F401 <library> imported but unused 2020-11-06 19:22:49 -05:00
Joshua Boniface ebf254f62d Lint: W293 blank line contains whitespace 2020-11-06 19:11:07 -05:00
Joshua Boniface 76b1cafdcc Lint: W605 invalid escape sequence '\$' 2020-11-06 19:01:22 -05:00
Joshua Boniface 639937f9c2 Lint: W605 invalid escape sequence '\^' 2020-11-06 18:59:30 -05:00
Joshua Boniface ec0b8acf90 Support per-VM migration type selectors 
Allow a VM to specify its migration type as a default choice. The valid
options are "default" (i.e. behave as now), "live" which forces a live
migration only, and "shutdown" which forces a shutdown migration only.
The new option is treated as a VM meta option and is set to default if
not found.
 2020-10-29 12:01:29 -04:00
Joshua Boniface d42bb74dc9 Use explicit acquire/release instead of with 
The with blocks did not seem to work as expected. Go back to exclusive
locks as well since these are more consistent.
 2020-10-21 11:38:23 -04:00
Joshua Boniface 42c5f84ba7 Do multiple lock attempts 2020-10-21 11:21:37 -04:00
Joshua Boniface 88556f4a33 Convert from exclusive to write lock 2020-10-21 11:12:36 -04:00
Joshua Boniface 3839040092 Add exclusive lock function 2020-10-21 10:46:41 -04:00
Joshua Boniface 84ade53fae Add locks for VM state changes 
Use exclusive locks during API events which change VM state. This is
fairly critical to avoid potential duplicate updates. Only implemented
for these specifically required functions to avoid major performance
hits elsewhere.
 2020-10-21 10:40:00 -04:00
Joshua Boniface 72f47f216a Revert "Add locking in common zkhander" 
This reverts commit 53c0d2b4f6.

This resulted in a massive performance hit and some inconsistent
behaviour. Revert for now an re-investigate later.
 2020-10-21 03:49:13 -04:00
Joshua Boniface 53c0d2b4f6 Add locking in common zkhander 
Ensures that every changed made here is locked, thus preventing
duplicate updates, etc.
 2020-10-21 03:17:18 -04:00
Joshua Boniface e7ab1bfddd Add cluster overprovision determination 
Adds a check of (n-1) memory overprovisioning. (n-1) is considered to be
the configuration that excludes the "largest" node. The cluster will
report degraded when in this state.
 2020-10-18 14:57:22 -04:00
Joshua Boniface f749633f7c Use provisioned memory for mem migration selector 
Use the new "provisioned" memory field, instead of the "allocated"
memory field, to determine the optimal node when using the "mem"
migration selector. This will take into account non-running VMs in the
calculation as well as running VMs.
 2020-10-18 14:17:15 -04:00
Joshua Boniface a4b80be5ed Add provisioned memory to node info 
Adds a separate field to the node memory, "provisioned", which totals
the amount of memory provisioned to all VMs on the node, regardless of
state, and in contrast to "allocated" which only counts running VMs.

Allows for the detection of potential overprovisioned states when
factoring in non-running VMs.

Includes the supporting code to get this data, since the original
implementation of VM memory selection was dependent on the VM being
running and getting this from libvirt. Now, if the VM is not active, it
gets this from the domain XML instead.
 2020-10-18 14:17:15 -04:00
Joshua Boniface e4891831ce Better handle missing elements from net config 
Prevents situations with an un-editable, invalid config being stuck.
 2020-08-21 10:27:45 -04:00
Joshua Boniface 1967034493 Use get() for all remaining VM XML gets 
Prevents KeyErrors and such.
 2020-08-21 10:10:13 -04:00
Joshua Boniface 15e78aa9f0 Add status information in cluster status 
Provide textual explanations for the degraded status, including
specific node/VM/OSD issues as well as detailed Ceph health. "Single
pane of glass" mentality.
 2020-08-17 12:25:23 -04:00
Joshua Boniface 2b4d980685 Display Ceph health in PVC status as well 
Makes this output a little more realistic and allows proper monitoring
of the Ceph cluster status (separate from the PVC status which is
tracking only OSD up/in state).
 2020-08-13 15:10:57 -04:00
Joshua Boniface 5b5b7d2276 Improve the conditional so it will always work 2020-08-11 23:08:40 -04:00
Joshua Boniface 0468eeb531 Support live resizing of running disk volumes 
This wasn't happening automatically, nor does it happen with qemu-img
commands, so we have to manually trigger a libvirt blockResize against
the volume. This setup is a little roundabout but seems to work fine.
 2020-08-11 21:46:12 -04:00
Joshua Boniface 142743b2c0 Fix erroneous comma 2020-08-05 10:34:30 -04:00
Joshua Boniface 8fc5299d38 Avoid failing if CPU features are missing 2020-07-08 12:32:42 -04:00
Joshua Boniface 37a58d35e8 Implement limiting of node output 
Closes #98
 2020-06-25 11:51:53 -04:00
Joshua Boniface deaf138e45 Add stats to VM information 2020-06-07 00:42:11 -04:00
Joshua Boniface 654a3cb7fa Improve debug output and use ceph df util data 2020-06-06 22:52:49 -04:00
Joshua Boniface fba39cb739 Fix broken sorting for pools and volumes 2020-06-06 21:28:54 -04:00
Joshua Boniface ce60836c34 Allow enforcement of live migration 
Provides a CLI and API argument to force live migration, which triggers
a new VM state "migrate-live". The node daemon VMInstance during migrate
will read this flag from the state and, if enforced, will not trigger a
shutdown migration.

Closes #95
 2020-06-06 12:00:44 -04:00
Joshua Boniface f61d443773 Allow move of migrated VM to current node 
Will make the migrate permanent instead of throwing an error.

Fixes #96
 2020-06-06 11:25:10 -04:00
Joshua Boniface 0fab7072ac Sort all Ceph lists by numeric ID 2020-05-29 13:31:18 -04:00
Joshua Boniface 2d507f8b42 Ensure rbdlist is updated when modifying VM config 2020-05-12 11:08:47 -04:00
Joshua Boniface 3cf90c46ad Correct bad handling of static reservations 2020-05-09 10:20:06 -04:00
Joshua Boniface 7b2180b626 Get both reservations in leases by default 2020-05-09 10:05:55 -04:00
Joshua Boniface 44efd66f2c Fix error renaming keys 
This function was not implemented and thus failed; implements it.
 2020-03-30 21:38:18 -04:00
Joshua Boniface 8678dedfea Revert "Implement wait for node coordinator transition" 
This reverts commit 0aefafa7f7.

This does not work since the API goes away during the transition.

References #72
 2020-02-19 10:50:21 -05:00
Joshua Boniface 0aefafa7f7 Implement wait for node coordinator transition 
References #72
 2020-02-19 10:50:04 -05:00
Joshua Boniface 6db4df51c0 Remove obsolete follow_console_log function 2020-02-19 10:19:49 -05:00
Joshua Boniface 5ddf72855b Clean up obsolete is_cli flags 2020-02-19 10:18:41 -05:00
Joshua Boniface 0e05ce8b07 Use correct wording of "shut down" 2020-02-19 10:04:58 -05:00
Joshua Boniface 07577a52a9 Implement wait support for various VM commands 
Implements wait support for VM restart, shutdown, move, migrate, and
unmigrate commands, similar to node flush/node unflush.

Includes some additional refactoring of the move command to make its
operation identical to migrate, only without recording the previous
node.

References #72
 2020-02-19 09:45:31 -05:00
Joshua Boniface 45040a5635 Make wait flag optional on flush functions 
References #72
 2020-02-19 09:44:38 -05:00
Joshua Boniface 7c99618752 Correct handling of bare bytes values 2020-02-17 12:32:20 -05:00
Joshua Boniface 670596ed8e Add dead node states to status 2020-02-15 18:51:02 -05:00
Joshua Boniface 95c59c2b39 Support non-extension fromhuman for normalization 2020-02-11 20:31:56 -05:00
Joshua Boniface e419855911 Support converting types during upload 
Allow the user to specify other, non-raw files and upload them,
performing a conversion with qemu-img convert and a temporary block
device as a shim (since qemu-img can't use FIFOs).

Also ensures that the target volume exists before proceeding.

Addresses #68
 2020-02-09 20:29:12 -05:00
Joshua Boniface 49e5ce1176 Support uploading disk images to volumes in API 
Addresses #68
 2020-02-09 13:45:04 -05:00
Joshua Boniface 92df125a77 Add missing library imports in common functions 2020-02-08 23:43:49 -05:00
Joshua Boniface 7ace5b5056 Remove /ceph/cmd pipe for (most) Ceph commands 
Addresses #80
 2020-02-08 23:40:02 -05:00
Joshua Boniface eeb8879f73 Move run_os_command to common functions 
References #80
 2020-02-08 23:33:15 -05:00
Joshua Boniface 4505b239eb Rename API and common Debian packages 
Closes #79
 2020-02-08 18:50:38 -05:00
Joshua Boniface 199b4bd40e Remove obsolete daemon-common section 2018-10-14 16:30:07 -04:00
Joshua Boniface f2431f179e Correct multiple issues with DHCP and add firewall control logic 2018-10-08 23:53:41 -04:00
Joshua Boniface 168435a827 Correct signals problem 2018-10-03 23:45:19 -04:00
Joshua Boniface 3e5e50e438 Handle daemons for dnsmasq 2018-10-03 23:17:59 -04:00
Joshua Boniface 505a951a7d Add delete function to zkhandler 2018-09-30 02:52:22 -04:00
Joshua Boniface fa1e3591cb Use much nicer updated zkhander from client library 2018-09-29 01:31:09 -04:00
Joshua Boniface 0f9637cb69 Make the IP failover work including threaded background os commands 2018-09-24 04:08:35 -04:00
Joshua Boniface b3259a3dcc Fix up some bugs and split out common daemon functions 2018-09-23 21:19:56 -04:00
Joshua Boniface 513de96626 Major refactor to separate out and standardize libraries 2018-09-20 03:43:34 -04:00
Joshua Boniface ac2ae4b3e3 Tweak some more things in the Daemon to reflect split 2018-09-12 12:43:18 -04:00
Joshua Boniface b6c7153401 Bump version and split common libraries into separate package 2018-09-12 12:11:59 -04:00