parallelvirtualcluster
/
pvc
1
0
Fork 0
Code Issues 11 Pull Requests Projects Releases 111 Wiki Activity
pvc/daemon-common/vm.py

734 lines
27 KiB
Python
Raw Normal View History

Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
#!/usr/bin/env python3
# vm.py - PVC client function library, VM fuctions
# Part of the Parallel Virtual Cluster (PVC) system
#
Update copyright header year to 2020
2020-01-08 19:38:02 -05:00
# Copyright (C) 2018-2020 Joshua M. Boniface <joshua@boniface.me>
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <https://www.gnu.org/licenses/>.
#
###############################################################################
import time
import re
import lxml.objectify
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
Rename API and common Debian packages Closes #79
2020-02-08 18:48:59 -05:00
import daemon_lib.zkhandler as zkhandler
import daemon_lib.common as common
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Rename API and common Debian packages Closes #79
2020-02-08 18:48:59 -05:00
import daemon_lib.ceph as ceph
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
#
# Cluster search functions
#
def getClusterDomainList(zk_conn):
# Get a list of UUIDs by listing the children of /domains
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
uuid_list = zkhandler.listchildren(zk_conn, '/domains')
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
name_list = []
# For each UUID, get the corresponding name from the data
for uuid in uuid_list:
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
name_list.append(zkhandler.readdata(zk_conn, '/domains/%s' % uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return uuid_list, name_list
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
def searchClusterByUUID(zk_conn, uuid):
try:
# Get the lists
uuid_list, name_list = getClusterDomainList(zk_conn)
# We're looking for UUID, so find that element ID
index = uuid_list.index(uuid)
# Get the name_list element at that index
name = name_list[index]
except ValueError:
# We didn't find anything
return None
return name
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
def searchClusterByName(zk_conn, name):
try:
# Get the lists
uuid_list, name_list = getClusterDomainList(zk_conn)
# We're looking for name, so find that element ID
index = name_list.index(name)
# Get the uuid_list element at that index
uuid = uuid_list[index]
except ValueError:
# We didn't find anything
return None
return uuid
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
def getDomainUUID(zk_conn, domain):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
if common.validateUUID(domain):
dom_name = searchClusterByUUID(zk_conn, domain)
dom_uuid = searchClusterByName(zk_conn, dom_name)
else:
dom_uuid = searchClusterByName(zk_conn, domain)
dom_name = searchClusterByUUID(zk_conn, dom_uuid)
return dom_uuid
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Add getDomainName to vm functions
2018-09-25 02:26:37 -04:00
def getDomainName(zk_conn, domain):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Add getDomainName to vm functions
2018-09-25 02:26:37 -04:00
if common.validateUUID(domain):
dom_name = searchClusterByUUID(zk_conn, domain)
dom_uuid = searchClusterByName(zk_conn, dom_name)
else:
dom_uuid = searchClusterByName(zk_conn, domain)
dom_name = searchClusterByUUID(zk_conn, dom_uuid)
return dom_name
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
#
# Direct functions
#
Add is_migrated check function
2019-07-25 14:33:50 -04:00
def is_migrated(zk_conn, domain):
# Validate that VM exists in cluster
dom_uuid = getDomainUUID(zk_conn, domain)
if not dom_uuid:
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
last_node = zkhandler.readdata(zk_conn, '/domains/{}/lastnode'.format(dom_uuid))
if last_node:
return True
Fix typo in common
2019-07-25 15:45:45 -04:00
else:
Add is_migrated check function
2019-07-25 14:33:50 -04:00
return False
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Implement lock flush function for clients Uses a similar command pathway as the Ceph commands do, but in a different location (pending move of the Ceph commands to a similar location).
2019-08-07 13:38:49 -04:00
def flush_locks(zk_conn, domain):
# Validate that VM exists in cluster
dom_uuid = getDomainUUID(zk_conn, domain)
if not dom_uuid:
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Verify that the VM is in a stopped state; freeing locks is not safe otherwise
state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
if state != 'stop':
return False, 'ERROR: VM "{}" is not in stopped state; flushing RBD locks on a running VM is dangerous.'.format(domain)
# Tell the cluster to create a new OSD for the host
flush_locks_string = 'flush_locks {}'.format(dom_uuid)
zkhandler.writedata(zk_conn, {'/cmd/domains': flush_locks_string})
# Wait 1/2 second for the cluster to get the message and start working
time.sleep(0.5)
# Acquire a read lock, so we get the return exclusively
lock = zkhandler.readlock(zk_conn, '/cmd/domains')
with lock:
try:
result = zkhandler.readdata(zk_conn, '/cmd/domains').split()[0]
if result == 'success-flush_locks':
message = 'Flushed locks on VM "{}"'.format(domain)
success = True
else:
message = 'ERROR: Failed to flush locks on VM "{}"; check node logs for details.'.format(domain)
success = False
Lint: E722 do not use bare 'except' (2)
2020-11-06 19:24:10 -05:00
except Exception:
Implement lock flush function for clients Uses a similar command pathway as the Ceph commands do, but in a different location (pending move of the Ceph commands to a similar location).
2019-08-07 13:38:49 -04:00
message = 'ERROR: Command ignored by node.'
success = False
# Acquire a write lock to ensure things go smoothly
lock = zkhandler.writelock(zk_conn, '/cmd/domains')
with lock:
time.sleep(0.5)
zkhandler.writedata(zk_conn, {'/cmd/domains': ''})
return success, message
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Support per-VM migration type selectors Allow a VM to specify its migration type as a default choice. The valid options are "default" (i.e. behave as now), "live" which forces a live migration only, and "shutdown" which forces a shutdown migration only. The new option is treated as a VM meta option and is set to default if not found.
2020-10-29 11:31:32 -04:00
def define_vm(zk_conn, config_data, target_node, node_limit, node_selector, node_autostart, migration_method=None, profile=None, initial_state='stop'):
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# Parse the XML data
Store list of RBD disks in ZK Store a basic list of RBD disks in Zookeeper for access by the node subsystem to handle RBD locks. This avoids the need to implement complex parsing logic inside the fencing configuration (or elsewhere). Also handle a malformed XML content properly during VM define.
2019-07-09 10:22:23 -04:00
try:
parsed_xml = lxml.objectify.fromstring(config_data)
Lint: E722 do not use bare 'except' (2)
2020-11-06 19:24:10 -05:00
except Exception:
Store list of RBD disks in ZK Store a basic list of RBD disks in Zookeeper for access by the node subsystem to handle RBD locks. This avoids the need to implement complex parsing logic inside the fencing configuration (or elsewhere). Also handle a malformed XML content properly during VM define.
2019-07-09 10:22:23 -04:00
return False, 'ERROR: Failed to parse XML data.'
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = parsed_xml.uuid.text
dom_name = parsed_xml.name.text
Enforce VM uniqueness when defining Closes #56
2019-12-09 09:56:59 -05:00
# Ensure that the UUID and name are unique
Fix missing argument
2019-12-09 11:31:56 -05:00
if searchClusterByUUID(zk_conn, dom_uuid) or searchClusterByName(zk_conn, dom_name):
Enforce VM uniqueness when defining Closes #56
2019-12-09 09:56:59 -05:00
return False, 'ERROR: Specified VM "{}" or UUID "{}" matches an existing VM on the cluster'.format(dom_name, dom_uuid)
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if not target_node:
Fix minor bugs
2019-10-12 02:03:23 -04:00
target_node = common.findTargetNode(zk_conn, dom_uuid)
Properly verify node inputs
2019-06-24 13:25:24 -04:00
else:
# Verify node is valid
valid_node = common.verifyNode(zk_conn, target_node)
if not valid_node:
Store list of RBD disks in ZK Store a basic list of RBD disks in Zookeeper for access by the node subsystem to handle RBD locks. This avoids the need to implement complex parsing logic inside the fencing configuration (or elsewhere). Also handle a malformed XML content properly during VM define.
2019-07-09 10:22:23 -04:00
return False, 'ERROR: Specified node "{}" is invalid.'.format(target_node)
# Obtain the RBD disk list using the common functions
Add stats to VM information
2020-06-07 00:40:21 -04:00
ddisks = common.getDomainDisks(parsed_xml, {})
Store list of RBD disks in ZK Store a basic list of RBD disks in Zookeeper for access by the node subsystem to handle RBD locks. This avoids the need to implement complex parsing logic inside the fencing configuration (or elsewhere). Also handle a malformed XML content properly during VM define.
2019-07-09 10:22:23 -04:00
rbd_list = []
for disk in ddisks:
if disk['type'] == 'rbd':
rbd_list.append(disk['name'])
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Better handle list elements when defining VM
2019-12-19 12:03:46 -05:00
# Join the limit
Handle the list of node_limits even better
2019-12-19 13:22:38 -05:00
if isinstance(node_limit, list) and node_limit:
Better handle list elements when defining VM
2019-12-19 12:03:46 -05:00
formatted_node_limit = ','.join(node_limit)
else:
Handle the list of node_limits even better
2019-12-19 13:22:38 -05:00
formatted_node_limit = ''
Better handle list elements when defining VM
2019-12-19 12:03:46 -05:00
# Join the RBD list
Handle the list of node_limits even better
2019-12-19 13:22:38 -05:00
if isinstance(rbd_list, list) and rbd_list:
Better handle list elements when defining VM
2019-12-19 12:03:46 -05:00
formatted_rbd_list = ','.join(rbd_list)
else:
Handle the list of node_limits even better
2019-12-19 13:22:38 -05:00
formatted_rbd_list = ''
Better handle list elements when defining VM
2019-12-19 12:03:46 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# Add the new domain to Zookeeper
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
zkhandler.writedata(zk_conn, {
'/domains/{}'.format(dom_uuid): dom_name,
Support overriding initial state of VM Still defaults to "stop" state unless overridden in the calling function.
2020-01-08 17:46:30 -05:00
'/domains/{}/state'.format(dom_uuid): initial_state,
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
'/domains/{}/node'.format(dom_uuid): target_node,
'/domains/{}/lastnode'.format(dom_uuid): '',
Better handle list elements when defining VM
2019-12-19 12:03:46 -05:00
'/domains/{}/node_limit'.format(dom_uuid): formatted_node_limit,
Implement VM metadata and use it Implements the storing of three VM metadata attributes: 1. Node limits - allows specifying a list of hosts on which the VM must run. This limit influences the migration behaviour of VMs. 2. Per-VM node selectors - allows each VM to have its migration autoselection method specified, to automatically allow different methods per VM based on the administrator's preferences. 3. VM autorestart - allows a VM to be automatically restarted from a stopped state, presumably due to a failure to find a target node (either due to limits or otherwise) during a flush/fence recovery, on the next node unflush/ready state of its home hypervisor. Useful mostly in conjunction with limits to ensure that VMs which were shut down due to there being no valid migration targets are started back up when their node becomes ready again. Includes the full client interaction with these metadata options, including printing, as well as defining a new function to modify this metadata. For the CLI it is set/modified either on `vm define` or via the `vm meta` command. For the API it is set/modified either on a POST to the `/vm` endpoint (during VM definition) or on POST to the `/vm/<vm>` endpoint. For the API this replaces the previous reserved word for VM creation from scratch as this will no longer be implemented in-daemon (see #22). Closes #52
2019-10-12 01:17:39 -04:00
'/domains/{}/node_selector'.format(dom_uuid): node_selector,
'/domains/{}/node_autostart'.format(dom_uuid): node_autostart,
Support per-VM migration type selectors Allow a VM to specify its migration type as a default choice. The valid options are "default" (i.e. behave as now), "live" which forces a live migration only, and "shutdown" which forces a shutdown migration only. The new option is treated as a VM meta option and is set to default if not found.
2020-10-29 11:31:32 -04:00
'/domains/{}/migration_method'.format(dom_uuid): migration_method,
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
'/domains/{}/failedreason'.format(dom_uuid): '',
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
'/domains/{}/consolelog'.format(dom_uuid): '',
Better handle list elements when defining VM
2019-12-19 12:03:46 -05:00
'/domains/{}/rbdlist'.format(dom_uuid): formatted_rbd_list,
Add profile from provisioner to data stored in ZK
2019-12-11 16:49:11 -05:00
'/domains/{}/profile'.format(dom_uuid): profile,
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
'/domains/{}/xml'.format(dom_uuid): config_data
})
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
return True, 'Added new VM with Name "{}" and UUID "{}" to database.'.format(dom_name, dom_uuid)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Support per-VM migration type selectors Allow a VM to specify its migration type as a default choice. The valid options are "default" (i.e. behave as now), "live" which forces a live migration only, and "shutdown" which forces a shutdown migration only. The new option is treated as a VM meta option and is set to default if not found.
2020-10-29 11:31:32 -04:00
def modify_vm_metadata(zk_conn, domain, node_limit, node_selector, node_autostart, provisioner_profile, migration_method):
Implement VM metadata and use it Implements the storing of three VM metadata attributes: 1. Node limits - allows specifying a list of hosts on which the VM must run. This limit influences the migration behaviour of VMs. 2. Per-VM node selectors - allows each VM to have its migration autoselection method specified, to automatically allow different methods per VM based on the administrator's preferences. 3. VM autorestart - allows a VM to be automatically restarted from a stopped state, presumably due to a failure to find a target node (either due to limits or otherwise) during a flush/fence recovery, on the next node unflush/ready state of its home hypervisor. Useful mostly in conjunction with limits to ensure that VMs which were shut down due to there being no valid migration targets are started back up when their node becomes ready again. Includes the full client interaction with these metadata options, including printing, as well as defining a new function to modify this metadata. For the CLI it is set/modified either on `vm define` or via the `vm meta` command. For the API it is set/modified either on a POST to the `/vm` endpoint (during VM definition) or on POST to the `/vm/<vm>` endpoint. For the API this replaces the previous reserved word for VM creation from scratch as this will no longer be implemented in-daemon (see #22). Closes #52
2019-10-12 01:17:39 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
if not dom_uuid:
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
if node_limit is not None:
Fix bug setting node limit of CSV
2019-12-26 19:08:26 -05:00
zkhandler.writedata(zk_conn, {
'/domains/{}/node_limit'.format(dom_uuid): node_limit
})
Implement VM metadata and use it Implements the storing of three VM metadata attributes: 1. Node limits - allows specifying a list of hosts on which the VM must run. This limit influences the migration behaviour of VMs. 2. Per-VM node selectors - allows each VM to have its migration autoselection method specified, to automatically allow different methods per VM based on the administrator's preferences. 3. VM autorestart - allows a VM to be automatically restarted from a stopped state, presumably due to a failure to find a target node (either due to limits or otherwise) during a flush/fence recovery, on the next node unflush/ready state of its home hypervisor. Useful mostly in conjunction with limits to ensure that VMs which were shut down due to there being no valid migration targets are started back up when their node becomes ready again. Includes the full client interaction with these metadata options, including printing, as well as defining a new function to modify this metadata. For the CLI it is set/modified either on `vm define` or via the `vm meta` command. For the API it is set/modified either on a POST to the `/vm` endpoint (during VM definition) or on POST to the `/vm/<vm>` endpoint. For the API this replaces the previous reserved word for VM creation from scratch as this will no longer be implemented in-daemon (see #22). Closes #52
2019-10-12 01:17:39 -04:00
if node_selector is not None:
zkhandler.writedata(zk_conn, {
'/domains/{}/node_selector'.format(dom_uuid): node_selector
})
if node_autostart is not None:
zkhandler.writedata(zk_conn, {
'/domains/{}/node_autostart'.format(dom_uuid): node_autostart
})
Allow modification of a VM profile And fix some bugs around this. Fixes #76
2020-01-30 11:45:46 -05:00
if provisioner_profile is not None:
zkhandler.writedata(zk_conn, {
'/domains/{}/profile'.format(dom_uuid): provisioner_profile
})
Support per-VM migration type selectors Allow a VM to specify its migration type as a default choice. The valid options are "default" (i.e. behave as now), "live" which forces a live migration only, and "shutdown" which forces a shutdown migration only. The new option is treated as a VM meta option and is set to default if not found.
2020-10-29 11:31:32 -04:00
if migration_method is not None:
zkhandler.writedata(zk_conn, {
'/domains/{}/migration_method'.format(dom_uuid): migration_method
})
Implement VM metadata and use it Implements the storing of three VM metadata attributes: 1. Node limits - allows specifying a list of hosts on which the VM must run. This limit influences the migration behaviour of VMs. 2. Per-VM node selectors - allows each VM to have its migration autoselection method specified, to automatically allow different methods per VM based on the administrator's preferences. 3. VM autorestart - allows a VM to be automatically restarted from a stopped state, presumably due to a failure to find a target node (either due to limits or otherwise) during a flush/fence recovery, on the next node unflush/ready state of its home hypervisor. Useful mostly in conjunction with limits to ensure that VMs which were shut down due to there being no valid migration targets are started back up when their node becomes ready again. Includes the full client interaction with these metadata options, including printing, as well as defining a new function to modify this metadata. For the CLI it is set/modified either on `vm define` or via the `vm meta` command. For the API it is set/modified either on a POST to the `/vm` endpoint (during VM definition) or on POST to the `/vm/<vm>` endpoint. For the API this replaces the previous reserved word for VM creation from scratch as this will no longer be implemented in-daemon (see #22). Closes #52
2019-10-12 01:17:39 -04:00
return True, 'Successfully modified PVC metadata of VM "{}".'.format(domain)
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Fix some problems with modify command after split
2018-09-25 02:32:08 -04:00
def modify_vm(zk_conn, domain, restart, new_vm_config):
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
Fix some problems with modify command after split
2018-09-25 02:32:08 -04:00
dom_name = getDomainName(zk_conn, domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Ensure rbdlist is updated when modifying VM config
2020-05-12 11:07:58 -04:00
# Parse and valiate the XML
try:
parsed_xml = lxml.objectify.fromstring(new_vm_config)
Lint: E722 do not use bare 'except' (2)
2020-11-06 19:24:10 -05:00
except Exception:
Ensure rbdlist is updated when modifying VM config
2020-05-12 11:07:58 -04:00
return False, 'ERROR: Failed to parse XML data.'
# Obtain the RBD disk list using the common functions
Add stats to VM information
2020-06-07 00:40:21 -04:00
ddisks = common.getDomainDisks(parsed_xml, {})
Ensure rbdlist is updated when modifying VM config
2020-05-12 11:07:58 -04:00
rbd_list = []
for disk in ddisks:
if disk['type'] == 'rbd':
rbd_list.append(disk['name'])
# Join the RBD list
if isinstance(rbd_list, list) and rbd_list:
formatted_rbd_list = ','.join(rbd_list)
else:
formatted_rbd_list = ''
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# Add the modified config to Zookeeper
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
zk_data = {
'/domains/{}'.format(dom_uuid): dom_name,
Ensure rbdlist is updated when modifying VM config
2020-05-12 11:07:58 -04:00
'/domains/{}/rbdlist'.format(dom_uuid): formatted_rbd_list,
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
'/domains/{}/xml'.format(dom_uuid): new_vm_config
}
zkhandler.writedata(zk_conn, zk_data)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Ensure VM restart happens after cfg update Fixes #74
2020-01-30 11:15:07 -05:00
if restart:
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'restart'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Ensure VM restart happens after cfg update Fixes #74
2020-01-30 11:15:07 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return True, ''
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Allow dumping of VMs
2019-03-12 21:09:54 -04:00
def dump_vm(zk_conn, domain):
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Allow dumping of VMs
2019-03-12 21:09:54 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Gram the domain XML and dump it to stdout
vm_xml = zkhandler.readdata(zk_conn, '/domains/{}/xml'.format(dom_uuid))
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
return True, vm_xml
Allow dumping of VMs
2019-03-12 21:09:54 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
def undefine_vm(zk_conn, domain):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Shut down the VM
Fix bad ZK node entry
2019-07-06 01:42:55 -04:00
current_vm_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
if current_vm_state != 'stop':
# Set the domain into stop mode
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'stop'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
# Wait for 2 seconds to allow state to flow to all nodes
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
time.sleep(2)
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Gracefully terminate the class instances
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'delete'})
time.sleep(2)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# Delete the configurations
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
zkhandler.deletekey(zk_conn, '/domains/{}'.format(dom_uuid))
return True, 'Undefined VM "{}" from the cluster.'.format(domain)
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
def remove_vm(zk_conn, domain):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
dom_uuid = getDomainUUID(zk_conn, domain)
if not dom_uuid:
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
disk_list = common.getDomainDiskList(zk_conn, dom_uuid)
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Shut down the VM
current_vm_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
if current_vm_state != 'stop':
# Set the domain into stop mode
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'stop'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
# Wait for 2 seconds to allow state to flow to all nodes
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
time.sleep(2)
# Gracefully terminate the class instances
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'delete'})
time.sleep(2)
# Delete the configurations
zkhandler.deletekey(zk_conn, '/domains/{}'.format(dom_uuid))
Add delay after deleting VM Otherwise there might still be a watcher on the block device when we go to delete it, which causes a failure.
2019-07-10 19:05:36 -04:00
time.sleep(2)
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Remove disks
for disk in disk_list:
# vmpool/vmname_volume
Don't crash if VM has invalid disks Useful if storage=False and removing a VM.
2019-07-31 23:05:00 -04:00
try:
disk_pool, disk_name = disk.split('/')
retcode, message = ceph.remove_volume(zk_conn, disk_pool, disk_name)
except ValueError:
continue
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
return True, 'Removed VM "{}" and disks from the cluster.'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
def start_vm(zk_conn, domain):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Set the VM to start
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'start'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Always use the passed domain in messages Don't ever output an ugly UUID unless that's what was actually sent to the function. Makes output much much nicer, especially on errors.
2019-07-05 16:38:54 -04:00
return True, 'Starting VM "{}".'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
def restart_vm(zk_conn, domain, wait=False):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Get state and verify we're OK to proceed
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
current_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
if current_state != 'start':
Always use the passed domain in messages Don't ever output an ugly UUID unless that's what was actually sent to the function. Makes output much much nicer, especially on errors.
2019-07-05 16:38:54 -04:00
return False, 'ERROR: VM "{}" is not in "start" state!'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
retmsg = 'Restarting VM "{}".'.format(domain)
# Set the VM to restart
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'restart'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
if wait:
while zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid)) == 'restart':
time.sleep(1)
retmsg = 'Restarted VM "{}"'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
return True, retmsg
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
def shutdown_vm(zk_conn, domain, wait=False):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Get state and verify we're OK to proceed
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
current_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
if current_state != 'start':
Always use the passed domain in messages Don't ever output an ugly UUID unless that's what was actually sent to the function. Makes output much much nicer, especially on errors.
2019-07-05 16:38:54 -04:00
return False, 'ERROR: VM "{}" is not in "start" state!'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
retmsg = 'Shutting down VM "{}"'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# Set the VM to shutdown
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'shutdown'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
if wait:
while zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid)) == 'shutdown':
time.sleep(1)
Use correct wording of "shut down"
2020-02-19 10:04:58 -05:00
retmsg = 'Shut down VM "{}"'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
return True, retmsg
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
def stop_vm(zk_conn, domain):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Set the VM to start
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'stop'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Always use the passed domain in messages Don't ever output an ugly UUID unless that's what was actually sent to the function. Makes output much much nicer, especially on errors.
2019-07-05 16:38:54 -04:00
return True, 'Forcibly stopping VM "{}".'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Add VM disable command Closes #53
2019-10-23 23:37:42 -04:00
def disable_vm(zk_conn, domain):
# Validate that VM exists in cluster
dom_uuid = getDomainUUID(zk_conn, domain)
if not dom_uuid:
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Get state and verify we're OK to proceed
current_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
if current_state != 'stop':
return False, 'ERROR: VM "{}" must be stopped before disabling!'.format(domain)
# Set the VM to start
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
Lint: E202 whitespace before '}'
2020-11-07 12:57:42 -05:00
zkhandler.writedata(zk_conn, {'/domains/{}/state'.format(dom_uuid): 'disable'})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Add VM disable command Closes #53
2019-10-23 23:37:42 -04:00
Use consistent tense for VM states Replace "failed" with "fail" and "disabled" with "disable" for consistency with the remaining states.
2019-10-23 23:53:25 -04:00
return True, 'Marked VM "{}" as disable.'.format(domain)
Add VM disable command Closes #53
2019-10-23 23:37:42 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Allow enforcement of live migration Provides a CLI and API argument to force live migration, which triggers a new VM state "migrate-live". The node daemon VMInstance during migrate will read this flag from the state and, if enforced, will not trigger a shutdown migration. Closes #95
2020-06-06 11:49:21 -04:00
def move_vm(zk_conn, domain, target_node, wait=False, force_live=False):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
# Get state and verify we're OK to proceed
current_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
if current_state != 'start':
# If the current state isn't start, preserve it; we're not doing live migration
target_state = current_state
else:
Allow enforcement of live migration Provides a CLI and API argument to force live migration, which triggers a new VM state "migrate-live". The node daemon VMInstance during migrate will read this flag from the state and, if enforced, will not trigger a shutdown migration. Closes #95
2020-06-06 11:49:21 -04:00
if force_live:
target_state = 'migrate-live'
else:
target_state = 'migrate'
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
current_node = zkhandler.readdata(zk_conn, '/domains/{}/node'.format(dom_uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if not target_node:
Use build-in selector and respect limits in client Use the new built-in selector option, as well as respecting node limits, when performing migrate or move actions on a VM via the clients.
2019-10-12 01:45:44 -04:00
target_node = common.findTargetNode(zk_conn, dom_uuid)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
else:
Properly verify node inputs
2019-06-24 13:25:24 -04:00
# Verify node is valid
valid_node = common.verifyNode(zk_conn, target_node)
if not valid_node:
Fix minor bugs
2019-10-12 01:50:15 -04:00
return False, 'ERROR: Specified node "{}" is invalid.'.format(target_node)
Properly verify node inputs
2019-06-24 13:25:24 -04:00
Use build-in selector and respect limits in client Use the new built-in selector option, as well as respecting node limits, when performing migrate or move actions on a VM via the clients.
2019-10-12 01:45:44 -04:00
# Check if node is within the limit
Properly handle empty node limit
2020-01-05 13:42:23 -05:00
node_limit = zkhandler.readdata(zk_conn, '/domains/{}/node_limit'.format(dom_uuid))
if node_limit and target_node not in node_limit.split(','):
Fix minor bugs
2019-10-12 01:50:15 -04:00
return False, 'ERROR: Specified node "{}" is not in the allowed list of nodes for VM "{}".'.format(target_node, domain)
Use build-in selector and respect limits in client Use the new built-in selector option, as well as respecting node limits, when performing migrate or move actions on a VM via the clients.
2019-10-12 01:45:44 -04:00
Properly verify node inputs
2019-06-24 13:25:24 -04:00
# Verify if node is current node
Massive rejigger into single daemon Completely restructure the daemon code to move the 4 discrete daemons into a single daemon that can be run on every hypervisor. Introduce the idea of a static list of "coordinator" nodes which are configured at install time to run Zookeeper and FRR in router mode, and which are allowed to take on client network management duties (gateway, DHCP, DNS, etc.) while also allowing them to run VMs (i.e. no dedicated "router" nodes required).
2018-10-14 02:01:35 -04:00
if target_node == current_node:
Allow move of migrated VM to current node Will make the migrate permanent instead of throwing an error. Fixes #96
2020-06-06 11:21:58 -04:00
last_node = zkhandler.readdata(zk_conn, '/domains/{}/lastnode'.format(dom_uuid))
if last_node:
Fix erroneous comma
2020-08-05 10:34:30 -04:00
zkhandler.writedata(zk_conn, {'/domains/{}/lastnode'.format(dom_uuid): ''})
Allow move of migrated VM to current node Will make the migrate permanent instead of throwing an error. Fixes #96
2020-06-06 11:21:58 -04:00
return True, 'Making temporary migration permanent for VM "{}".'.format(domain)
Always use the passed domain in messages Don't ever output an ugly UUID unless that's what was actually sent to the function. Makes output much much nicer, especially on errors.
2019-07-05 16:38:54 -04:00
return False, 'ERROR: VM "{}" is already running on node "{}".'.format(domain, current_node)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Avoid migration to invalid target_nodes
2019-10-17 12:16:21 -04:00
if not target_node:
return False, 'ERROR: Could not find a valid migration target for VM "{}".'.format(domain)
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
retmsg = 'Permanently migrating VM "{}" to node "{}".'.format(domain, target_node)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
zkhandler.writedata(zk_conn, {
Add locks for VM state changes Use exclusive locks during API events which change VM state. This is fairly critical to avoid potential duplicate updates. Only implemented for these specifically required functions to avoid major performance hits elsewhere.
2020-10-21 10:40:00 -04:00
'/domains/{}/state'.format(dom_uuid): target_state,
'/domains/{}/node'.format(dom_uuid): target_node,
'/domains/{}/lastnode'.format(dom_uuid): ''
})
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock.release()
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
if wait:
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
while zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid)) == target_state:
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
time.sleep(1)
retmsg = 'Permanently migrated VM "{}" to node "{}"'.format(domain, target_node)
return True, retmsg
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Allow enforcement of live migration Provides a CLI and API argument to force live migration, which triggers a new VM state "migrate-live". The node daemon VMInstance during migrate will read this flag from the state and, if enforced, will not trigger a shutdown migration. Closes #95
2020-06-06 11:49:21 -04:00
def migrate_vm(zk_conn, domain, target_node, force_migrate, wait=False, force_live=False):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Get state and verify we're OK to proceed
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
current_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
if current_state != 'start':
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
# If the current state isn't start, preserve it; we're not doing live migration
target_state = current_state
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
else:
Allow enforcement of live migration Provides a CLI and API argument to force live migration, which triggers a new VM state "migrate-live". The node daemon VMInstance during migrate will read this flag from the state and, if enforced, will not trigger a shutdown migration. Closes #95
2020-06-06 11:49:21 -04:00
if force_live:
target_state = 'migrate-live'
else:
target_state = 'migrate'
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
current_node = zkhandler.readdata(zk_conn, '/domains/{}/node'.format(dom_uuid))
last_node = zkhandler.readdata(zk_conn, '/domains/{}/lastnode'.format(dom_uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if last_node and not force_migrate:
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
return False, 'ERROR: VM "{}" has been previously migrated.'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if not target_node:
Use build-in selector and respect limits in client Use the new built-in selector option, as well as respecting node limits, when performing migrate or move actions on a VM via the clients.
2019-10-12 01:45:44 -04:00
target_node = common.findTargetNode(zk_conn, dom_uuid)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
else:
Properly verify node inputs
2019-06-24 13:25:24 -04:00
# Verify node is valid
valid_node = common.verifyNode(zk_conn, target_node)
if not valid_node:
Fix minor bugs
2019-10-12 01:50:15 -04:00
return False, 'ERROR: Specified node "{}" is invalid.'.format(target_node)
Properly verify node inputs
2019-06-24 13:25:24 -04:00
Use build-in selector and respect limits in client Use the new built-in selector option, as well as respecting node limits, when performing migrate or move actions on a VM via the clients.
2019-10-12 01:45:44 -04:00
# Check if node is within the limit
Properly handle empty node limit
2020-01-05 13:42:23 -05:00
node_limit = zkhandler.readdata(zk_conn, '/domains/{}/node_limit'.format(dom_uuid))
if node_limit and target_node not in node_limit.split(','):
Fix minor bugs
2019-10-12 01:50:15 -04:00
return False, 'ERROR: Specified node "{}" is not in the allowed list of nodes for VM "{}".'.format(target_node, domain)
Use build-in selector and respect limits in client Use the new built-in selector option, as well as respecting node limits, when performing migrate or move actions on a VM via the clients.
2019-10-12 01:45:44 -04:00
Properly verify node inputs
2019-06-24 13:25:24 -04:00
# Verify if node is current node
Massive rejigger into single daemon Completely restructure the daemon code to move the 4 discrete daemons into a single daemon that can be run on every hypervisor. Introduce the idea of a static list of "coordinator" nodes which are configured at install time to run Zookeeper and FRR in router mode, and which are allowed to take on client network management duties (gateway, DHCP, DNS, etc.) while also allowing them to run VMs (i.e. no dedicated "router" nodes required).
2018-10-14 02:01:35 -04:00
if target_node == current_node:
Always use the passed domain in messages Don't ever output an ugly UUID unless that's what was actually sent to the function. Makes output much much nicer, especially on errors.
2019-07-05 16:38:54 -04:00
return False, 'ERROR: VM "{}" is already running on node "{}".'.format(domain, current_node)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Avoid migration to invalid target_nodes
2019-10-17 12:16:21 -04:00
if not target_node:
return False, 'ERROR: Could not find a valid migration target for VM "{}".'.format(domain)
Make force_migrate not replace existing node Previous to this, if once force-migrated a VM, the previous_node value would be updated to the current node, which is likely never what an administrator would want. Change this functionality so that the previous node value is not changed, and update the documentation to reflect this.
2019-07-07 15:10:48 -04:00
# Don't overwrite an existing last_node when using force_migrate
if last_node and force_migrate:
current_node = last_node
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
retmsg = 'Migrating VM "{}" to node "{}".'.format(domain, target_node)
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
zkhandler.writedata(zk_conn, {
'/domains/{}/state'.format(dom_uuid): target_state,
'/domains/{}/node'.format(dom_uuid): target_node,
'/domains/{}/lastnode'.format(dom_uuid): current_node
})
lock.release()
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
if wait:
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
while zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid)) == target_state:
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
time.sleep(1)
retmsg = 'Migrated VM "{}" to node "{}"'.format(domain, target_node)
return True, retmsg
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Allow enforcement of live migration Provides a CLI and API argument to force live migration, which triggers a new VM state "migrate-live". The node daemon VMInstance during migrate will read this flag from the state and, if enforced, will not trigger a shutdown migration. Closes #95
2020-06-06 11:49:21 -04:00
def unmigrate_vm(zk_conn, domain, wait=False, force_live=False):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
if not dom_uuid:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Get state and verify we're OK to proceed
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
current_state = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
if current_state != 'start':
Don't set unmigrated VMs to start
2019-03-20 10:19:01 -04:00
# If the current state isn't start, preserve it; we're not doing live migration
target_state = current_state
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
else:
Allow enforcement of live migration Provides a CLI and API argument to force live migration, which triggers a new VM state "migrate-live". The node daemon VMInstance during migrate will read this flag from the state and, if enforced, will not trigger a shutdown migration. Closes #95
2020-06-06 11:49:21 -04:00
if force_live:
target_state = 'migrate-live'
else:
target_state = 'migrate'
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
target_node = zkhandler.readdata(zk_conn, '/domains/{}/lastnode'.format(dom_uuid))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Massive rejigger into single daemon Completely restructure the daemon code to move the 4 discrete daemons into a single daemon that can be run on every hypervisor. Introduce the idea of a static list of "coordinator" nodes which are configured at install time to run Zookeeper and FRR in router mode, and which are allowed to take on client network management duties (gateway, DHCP, DNS, etc.) while also allowing them to run VMs (i.e. no dedicated "router" nodes required).
2018-10-14 02:01:35 -04:00
if target_node == '':
Always use the passed domain in messages Don't ever output an ugly UUID unless that's what was actually sent to the function. Makes output much much nicer, especially on errors.
2019-07-05 16:38:54 -04:00
return False, 'ERROR: VM "{}" has not been previously migrated.'.format(domain)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
retmsg = 'Unmigrating VM "{}" back to node "{}".'.format(domain, target_node)
Use explicit acquire/release instead of with The with blocks did not seem to work as expected. Go back to exclusive locks as well since these are more consistent.
2020-10-21 11:32:40 -04:00
lock = zkhandler.exclusivelock(zk_conn, '/domains/{}/state'.format(dom_uuid))
lock.acquire()
zkhandler.writedata(zk_conn, {
'/domains/{}/state'.format(dom_uuid): target_state,
'/domains/{}/node'.format(dom_uuid): target_node,
'/domains/{}/lastnode'.format(dom_uuid): ''
})
lock.release()
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
if wait:
Clean up obsolete is_cli flags
2020-02-19 10:18:41 -05:00
while zkhandler.readdata(zk_conn, '/domains/{}/state'.format(dom_uuid)) == target_state:
Implement wait support for various VM commands Implements wait support for VM restart, shutdown, move, migrate, and unmigrate commands, similar to node flush/node unflush. Includes some additional refactoring of the move command to make its operation identical to migrate, only without recording the previous node. References #72
2020-02-19 09:45:31 -05:00
time.sleep(1)
retmsg = 'Unmigrated VM "{}" back to node "{}"'.format(domain, target_node)
return True, retmsg
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
def get_console_log(zk_conn, domain, lines=1000):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
if not dom_uuid:
return False, 'ERROR: Could not find VM "{}" in the cluster!'.format(domain)
# Get the data from ZK
console_log = zkhandler.readdata(zk_conn, '/domains/{}/consolelog'.format(dom_uuid))
# Shrink the log buffer to length lines
shrunk_log = console_log.split('\n')[-lines:]
loglines = '\n'.join(shrunk_log)
Support providing console log lines from API
2019-12-25 19:10:12 -05:00
return True, loglines
Implement domain log watching Implements the ability for a client to watch almost-live domain console logs from the hypervisors. It does this using a deque-based "tail -f" mechanism (with a configurable buffer per-VM) that watches the domain console logfile in the (configurable) directory every half-second. It then stores the current buffer in Zookeeper when changed, where a client can then request it, either as a static piece of text in the `less` pager, or via a similar "tail -f" functionality implemented using fixed line splitting and comparison to provide a generally-seamless output. Enabling this feature requires each guest VM to implement a Libvirt serial log and write its (text) console to it, for example using the default logging directory: ``` <serial type='pty'> <log file='/var/log/libvirt/vmname.log' append='off'/> <serial> ``` The append mode can be either on or off; on grows files unbounded, off causes the log (and hence the PVC log data) to be truncated on initial VM startup from offline. The administrator must choose how they best want to handle this until Libvirt implements their own clog-type logging format.
2019-04-11 19:06:06 -04:00
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
def get_info(zk_conn, domain):
Support removing VMs (and disks) from PVC clients Adds full support for removing a VM entirely, including its RBD disks, via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
# Validate that VM exists in cluster
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
dom_uuid = getDomainUUID(zk_conn, domain)
if not dom_uuid:
return False, 'ERROR: No VM named "{}" is present in the cluster.'.format(domain)
# Gather information from XML config and print it
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
domain_information = common.getInformationFromXML(zk_conn, dom_uuid)
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if not domain_information:
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
return False, 'ERROR: Could not get information about VM "{}".'.format(domain)
return True, domain_information
Lint: E302 expected 2 blank lines, found X
2020-11-07 14:45:24 -05:00
Support disabling fuzzy limit matching Used for the APIs when they filter by name but we don't want a fuzzy limit since we specified an exact name.
2019-07-05 16:28:18 -04:00
def get_list(zk_conn, node, state, limit, is_fuzzy=True):
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if node:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# Verify node is valid
Fix variable naming bug
2019-07-05 14:18:18 -04:00
if not common.verifyNode(zk_conn, node):
Add quotes around vars in all return messages
2019-07-08 22:37:26 -04:00
return False, 'Specified node "{}" is invalid.'.format(node)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if state:
Lint: E202 whitespace before ']'
2020-11-07 13:02:54 -05:00
valid_states = ['start', 'restart', 'shutdown', 'stop', 'disable', 'fail', 'migrate', 'unmigrate', 'provision']
Lint: E713 test for membership should be 'not in'
2020-11-06 20:37:52 -05:00
if state not in valid_states:
Support VM list filtering by state
2019-03-20 11:31:54 -04:00
return False, 'VM state "{}" is not valid.'.format(state)
Use zkhandler for everything; add delete operation
2018-10-27 15:27:08 -04:00
full_vm_list = zkhandler.listchildren(zk_conn, '/domains')
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
vm_list = []
Handle regex in a more consistent way
2019-03-12 23:52:59 -04:00
# Set our limit to a sensible regex
Support disabling fuzzy limit matching Used for the APIs when they filter by name but we don't want a fuzzy limit since we specified an exact name.
2019-07-05 16:28:18 -04:00
if limit and is_fuzzy:
Handle regex in a more consistent way
2019-03-12 23:52:59 -04:00
try:
# Implcitly assume fuzzy limits
Lint: W605 invalid escape sequence '\^'
2020-11-06 18:59:30 -05:00
if not re.match('[^].*', limit):
Handle regex in a more consistent way
2019-03-12 23:52:59 -04:00
limit = '.*' + limit
Lint: W605 invalid escape sequence '\$'
2020-11-06 19:01:22 -05:00
if not re.match('.*[$]', limit):
Handle regex in a more consistent way
2019-03-12 23:52:59 -04:00
limit = limit + '.*'
except Exception as e:
return False, 'Regex Error: {}'.format(e)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# If we're limited, remove other nodes' VMs
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
vm_node = {}
vm_state = {}
Improve limit handling
2018-09-25 02:20:32 -04:00
for vm in full_vm_list:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
# Check we don't match the limit
Improve limit handling
2018-09-25 02:20:32 -04:00
name = zkhandler.readdata(zk_conn, '/domains/{}'.format(vm))
Deleted the wrong line
2019-03-12 21:39:17 -04:00
vm_node[vm] = zkhandler.readdata(zk_conn, '/domains/{}/node'.format(vm))
Support VM list filtering by state
2019-03-20 11:31:54 -04:00
vm_state[vm] = zkhandler.readdata(zk_conn, '/domains/{}/state'.format(vm))
Handle regex in a more consistent way
2019-03-12 23:52:59 -04:00
# Handle limiting
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if limit:
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
try:
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if re.match(limit, vm):
if not node and not state:
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
vm_list.append(common.getInformationFromXML(zk_conn, vm))
Improve limit handling
2018-09-25 02:20:32 -04:00
else:
Support VM list filtering by state
2019-03-20 11:31:54 -04:00
if vm_node[vm] == node or vm_state[vm] == state:
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
vm_list.append(common.getInformationFromXML(zk_conn, vm))
Improve limit handling
2018-09-25 02:20:32 -04:00
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if re.match(limit, name):
if not node and not state:
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
vm_list.append(common.getInformationFromXML(zk_conn, vm))
Improve limit handling
2018-09-25 02:20:32 -04:00
else:
Support VM list filtering by state
2019-03-20 11:31:54 -04:00
if vm_node[vm] == node or vm_state[vm] == state:
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
vm_list.append(common.getInformationFromXML(zk_conn, vm))
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
except Exception as e:
Improve limit handling
2018-09-25 02:20:32 -04:00
return False, 'Regex Error: {}'.format(e)
Refactor CLI client to split out actual functions for reuse
2018-09-20 03:25:58 -04:00
else:
Massive rejigger into single daemon Completely restructure the daemon code to move the 4 discrete daemons into a single daemon that can be run on every hypervisor. Introduce the idea of a static list of "coordinator" nodes which are configured at install time to run Zookeeper and FRR in router mode, and which are allowed to take on client network management duties (gateway, DHCP, DNS, etc.) while also allowing them to run VMs (i.e. no dedicated "router" nodes required).
2018-10-14 02:01:35 -04:00
# Check node to avoid unneeded ZK calls
Make conditionals more Pythonic
2019-06-24 13:37:56 -04:00
if not node and not state:
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
vm_list.append(common.getInformationFromXML(zk_conn, vm))
Improve limit handling
2018-09-25 02:20:32 -04:00
else:
Support VM list filtering by state
2019-03-20 11:31:54 -04:00
if vm_node[vm] == node or vm_state[vm] == state:
Reorganize client functions Move some functions to the common file for future usage.
2019-07-09 09:29:47 -04:00
vm_list.append(common.getInformationFromXML(zk_conn, vm))
Update vm.py to allow API use
2019-05-20 22:15:28 -04:00
return True, vm_list