Commit Graph

629 Commits

Author SHA1 Message Date
Joshua Boniface 47c49f7672 Fix click import (again) 2020-01-06 12:55:44 -05:00
Joshua Boniface 27279c9686 Properly import click echo 2020-01-06 12:38:20 -05:00
Joshua Boniface 9ea86cc570 Better handle key values 2020-01-06 11:56:34 -05:00
Joshua Boniface 1113234270 Send DELETE instead of POST 2020-01-06 10:47:01 -05:00
Joshua Boniface 66863a0183 Improve output of cluster 2020-01-06 09:23:35 -05:00
Joshua Boniface 007ee56a46 Fix some bugs and where no-cluster message happens 2020-01-06 09:18:22 -05:00
Joshua Boniface be5c67f696 Add messages when adding/removing clusters to CLI 2020-01-06 09:11:00 -05:00
Joshua Boniface 8364d67c51 Correct bad name 2020-01-05 19:19:19 -05:00
Joshua Boniface a9b8403812 Support viewing source volumes 2020-01-05 19:11:52 -05:00
Joshua Boniface 33709828b1 Handle bad response to logs 2020-01-05 17:07:39 -05:00
Joshua Boniface c39c6e8d86 Mention what to do if no cluster found 2020-01-05 15:00:08 -05:00
Joshua Boniface ef293b7778 Fix syntax error 2020-01-05 13:19:21 -05:00
Joshua Boniface b9fdce6728 Use consistent function and stderr for debug 2020-01-05 12:51:06 -05:00
Joshua Boniface baae9b4982 Print initial cluster message to stderr
Allows it to be easily filtered out
2020-01-05 12:38:29 -05:00
Joshua Boniface 08570bd7b9 Don't click.echo except in main program
Return all the strings we printed directly, so that the caller can print
them normally.
2020-01-05 12:35:00 -05:00
Joshua Boniface 46f3915ab4 Put list handling in the right place 2020-01-05 11:51:03 -05:00
Joshua Boniface 44753c0609 Always handle single-instance results 2020-01-05 02:55:28 -05:00
Joshua Boniface 4fb0d66f6a Use proper variable name 2020-01-05 00:49:50 -05:00
Joshua Boniface 94c60f9daa Mention aborting on no cluster 2020-01-04 15:00:24 -05:00
Joshua Boniface 089b3d7a43 Better output of job status 2020-01-04 14:45:40 -05:00
Joshua Boniface aa2bb7c94c Add create and status functions 2020-01-04 14:31:22 -05:00
Joshua Boniface b88bdc29af Add profile handling 2020-01-04 14:06:36 -05:00
Joshua Boniface a4ad58e64c Add userdata and script options 2020-01-04 13:04:01 -05:00
Joshua Boniface e5d38ec6bd Implement template CLI functions 2020-01-04 11:58:30 -05:00
Joshua Boniface f80b432d3b Handle the empty list better 2020-01-02 12:18:41 -05:00
Joshua Boniface f624cf9e24 Handle no-good-cluster scenarios 2020-01-02 11:19:11 -05:00
Joshua Boniface cb3cc2782a Add initial bits of provisioner CLI 2020-01-02 11:19:08 -05:00
Joshua Boniface d6bd616b31 Fix some bad output 2019-12-30 14:35:44 -05:00
Joshua Boniface 0ced9b2777 Fix incorrect bolding on output 2019-12-30 14:19:17 -05:00
Joshua Boniface 217b163d4b Remove indented empty lines 2019-12-30 13:29:07 -05:00
Joshua Boniface 18e6192178 Add support for multiple clusters in CLI 2019-12-30 13:27:40 -05:00
Joshua Boniface 18c14cbf77 Handle there being no good nodes/VMs 2019-12-30 09:39:04 -05:00
Joshua Boniface 2325f87389 Correct typo in header 2019-12-30 09:07:41 -05:00
Joshua Boniface d2af6f4508 Complete conversion to API client 2019-12-29 20:33:51 -05:00
Joshua Boniface d2f27cc8fe Add network functions to API client 2019-12-29 16:34:32 -05:00
Joshua Boniface d0b6bb4cc3 Convert VM functions to API client 2019-12-29 16:33:50 -05:00
Joshua Boniface a85d8e5c4d Add some QOL improvements 2019-12-29 16:33:50 -05:00
Joshua Boniface e0c7397703 Complete node API client functions 2019-12-29 16:33:50 -05:00
Joshua Boniface 28a0ce772b Node info moved to API 2019-12-29 16:33:50 -05:00
Joshua Boniface 3d178304f1 Convert some node functions to API 2019-12-29 16:33:50 -05:00
Joshua Boniface a9aeb2e836 Implement log viewing/following with API 2019-12-29 16:33:50 -05:00
Joshua Boniface b7eddc6ae9 Add CLI lib 2019-12-29 16:33:50 -05:00
Joshua Boniface 1f6f486b3b Fix broken unwanted change to pvc.py 2019-12-25 13:35:31 -05:00
Joshua Boniface ebcf0cee43 Add doc endpoint 2019-12-23 23:11:55 -05:00
Joshua Boniface 7901ffb5e3 Handle the list of node_limits even better 2019-12-19 13:22:38 -05:00
Joshua Boniface 1924154dce Improve help text 2019-12-08 23:59:17 -05:00
Joshua Boniface 1fb560e996 Add DNS nameservers to networks 2019-12-08 23:55:45 -05:00
Joshua Boniface 32b4ed66e8 Add VM disable command
Closes #53
2019-10-23 23:44:47 -04:00
Joshua Boniface 4bff533571 Add cluster status command 2019-10-22 11:23:12 -04:00
Joshua Boniface 03447d3374 Update copyright string year to include 2019 2019-10-13 12:09:51 -04:00
Joshua Boniface 01c8986381 Update help messages in meta command 2019-10-12 02:08:52 -04:00
Joshua Boniface e5393082b6 Use build-in selector and respect limits in client
Use the new built-in selector option, as well as respecting node limits,
when performing migrate or move actions on a VM via the clients.
2019-10-12 01:45:44 -04:00
Joshua Boniface 8dc0c8f0ac Fix minor bugs 2019-10-12 01:36:50 -04:00
Joshua Boniface 5995353597 Implement VM metadata and use it
Implements the storing of three VM metadata attributes:
1. Node limits - allows specifying a list of hosts on which the VM must
run. This limit influences the migration behaviour of VMs.
2. Per-VM node selectors - allows each VM to have its migration
autoselection method specified, to automatically allow different methods
per VM based on the administrator's preferences.
3. VM autorestart - allows a VM to be automatically restarted from a
stopped state, presumably due to a failure to find a target node (either
due to limits or otherwise) during a flush/fence recovery, on the next
node unflush/ready state of its home hypervisor. Useful mostly in
conjunction with limits to ensure that VMs which were shut down due to
there being no valid migration targets are started back up when their
node becomes ready again.

Includes the full client interaction with these metadata options,
including printing, as well as defining a new function to modify this
metadata. For the CLI it is set/modified either on `vm define` or via the
`vm meta` command. For the API it is set/modified either on a POST to
the `/vm` endpoint (during VM definition) or on POST to the `/vm/<vm>`
endpoint. For the API this replaces the previous reserved word for VM
creation from scratch as this will no longer be implemented in-daemon
(see #22).

Closes #52
2019-10-12 01:17:39 -04:00
Joshua Boniface 94501f400f Remove vm add command
This is likely not going to be used with the planned implementation of
the automatic provisioning daemon, which will be either an API client or
direct Python binding client.
2019-10-11 23:52:20 -04:00
Joshua Boniface 6c9b49d7cd Add clone volume command to CLI 2019-10-10 14:11:13 -04:00
Joshua Boniface f6b40911f8 Revert "Add base CLI for deploy section"
This reverts commit ed43c70f61.
2019-10-10 14:09:40 -04:00
Joshua Boniface 1c44a1a26e Revert "Add base CLI for deploy image section"
This reverts commit f918058e6e.
2019-10-10 14:09:33 -04:00
Joshua Boniface f918058e6e Add base CLI for deploy image section 2019-09-09 10:57:36 -04:00
Joshua Boniface ed43c70f61 Add base CLI for deploy section 2019-08-25 21:18:33 -04:00
Joshua Boniface 3b7c1adf71 Implement configurable replcfg (client-side)
Implements administrator-selectable replication configurations for new
pools in PVC clusters, overriding the default of copies=3,mincopies=2.
2019-08-23 14:12:15 -04:00
Joshua Boniface 5876f4589b Add little easter egg to the init command
Because who doesn't like David Wilcox?
2019-08-20 09:19:56 -04:00
Joshua Boniface 427ef9454a Use dash in flush-locks command 2019-08-07 17:50:25 -04:00
Joshua Boniface d63d9637a5 Add flush_locks command to CLI client 2019-08-07 13:42:01 -04:00
Joshua Boniface 2dbf276e51 Implement snapshot renaming
[3/2] Implements #44
2019-07-28 23:00:35 -04:00
Joshua Boniface 35363671a0 Implement Ceph volume resize and rename
Includes a simple implementation of a zookeeper "rename" facility,
allowing a key and all data to be replaced by a new key with a different
name but containing all the same child elements and data.

[2/2] Implements #44
2019-07-26 15:13:21 -04:00
Joshua Boniface d5f263bdd6 Implement Ceph volume resize and rename in clients
[1/2] Implements #44
2019-07-26 15:13:21 -04:00
Joshua Boniface 9ccf3390ab Add storage prefix to Ceph commands
Add the "storage" prefix to all Ceph-based commands in both the CLI and
the API. This partially abstracts the storage subsystem from the Ceph
tool specifically, should future storage subsystems be added or changed.
The name "ceph" is still used due to the non-abstracted components of
the Ceph management, e.g. referencing Ceph-specific concepts like OSDs
or pools.
2019-07-10 15:14:17 -04:00
Joshua Boniface 7a8aee9fe7 Remove flush locking functionality
This just seemed like more trouble that it was worth. Flush locks were
originally intended as a way to counteract the weird issues around
flushing that were mostly fixed by the code refactoring, so this will
help test if those issues are truly gone. If not, will look into a
cleaner solution that doesn't result in unchangeable states.
2019-07-09 23:59:17 -04:00
Joshua Boniface 31b4dd9aed Show radosdf output in clients 2019-07-08 10:59:49 -04:00
Joshua Boniface f75216d946 Make force_migrate not replace existing node
Previous to this, if once force-migrated a VM, the previous_node value
would be updated to the current node, which is likely never what an
administrator would want. Change this functionality so that the previous
node value is not changed, and update the documentation to reflect this.
2019-07-07 15:10:48 -04:00
Joshua Boniface 804f72d905 Remove obsolete init attempt 2019-07-06 19:03:55 -04:00
Joshua Boniface 6ee3c91a63 Use None instead of all in ceph.py
Make it like other optional args (like direction in net ACLs) and use
None instead of 'all' when specifying any option
2019-07-05 13:59:29 -04:00
Joshua Boniface d755a80fe8 Fix bugs with refactoring 2019-07-05 00:44:40 -04:00
Joshua Boniface 84a905b7f0 Refactor ceph client for API 2019-07-05 00:30:03 -04:00
Joshua Boniface bcab11ee49 Refactor network.py for API usage 2019-07-04 23:01:22 -04:00
Joshua Boniface d28b1ba133 Support removing VMs (and disks) from PVC clients
Adds full support for removing a VM entirely, including its RBD disks,
via the PVC client(s). Avoids needing an undefine + manual removal.
2019-06-27 11:19:48 -04:00
Joshua Boniface d5e5ed4b69 Add hostname into confirmation 2019-06-21 15:54:54 -04:00
Joshua Boniface ef272b0b7d Add removal confirmations and zap disk before add 2019-06-21 15:52:28 -04:00
Joshua Boniface 26a4b3b757 Fix logic error in init 2019-06-21 15:38:32 -04:00
Joshua Boniface 519203d7fd Fix bad flag value 2019-06-21 15:09:15 -04:00
Joshua Boniface 93d2e0da6f Improve handling of init 2019-06-21 14:16:32 -04:00
Joshua Boniface 867ad1fc1b Support human-readable biconversion and in volumes 2019-06-21 09:23:52 -04:00
Joshua Boniface 33bf3ee3cf Add confirmation to cluster init 2019-06-21 09:02:58 -04:00
Joshua Boniface 3dd56c55c0 Fix a few more bugs 2019-06-19 15:32:32 -04:00
Joshua Boniface 822cf53a49 Nicer list handling for snapshots 2019-06-19 15:22:44 -04:00
Joshua Boniface 5d4ecafb03 Support listing of snapshots 2019-06-19 15:15:17 -04:00
Joshua Boniface 4ca0fe8f6f Handle per-pool lists better 2019-06-19 14:11:03 -04:00
Joshua Boniface a940d03959 Fix some bugs and add RBD volume stats 2019-06-19 10:25:22 -04:00
Joshua Boniface a4ab3075ab Correct some bugs around new code 2019-06-19 00:23:25 -04:00
Joshua Boniface 01959cb9e3 Implementation of RBD volumes and snapshots
Adds the ability to manage RBD volumes (add/remove) and RBD
snapshots (add/remove). (Working) list functions to come.
2019-06-19 00:12:44 -04:00
Joshua Boniface 537ad5de43 Make ceph pool removal confirmation verbose 2019-06-18 21:51:17 -04:00
Joshua Boniface 264c2d4748 Fix broken prompting for pool removal 2019-06-18 21:33:39 -04:00
Joshua Boniface 69462d2c7b Ensure myhostname is short
PVC now uses shortnames for node names, so ensure this is reflected
in the default choices for some node-level commands.
2019-05-23 22:27:34 -04:00
Joshua Boniface 4bfbbaa7d9 Remove commented needless call 2019-05-21 23:08:28 -04:00
Joshua Boniface 73443ecbaf Update vm.py to allow API use 2019-05-20 22:15:28 -04:00
Joshua Boniface 91ea96b772 Actually send wait for unflush 2019-05-11 00:16:38 -04:00
Joshua Boniface c19902d952 Implement flush locking for nodes
Implements a locking mechanism to prevent clobbering of node
flushes. When a flush begins, a global cluster lock is placed
which is freed once the flush completes. While the lock is in place,
other flush events queue waiting for the lock to free before
proceeding.

Modifies the CLI output flow when the `--wait` option is specified.
First, if a lock exists when running the command, the message is
tweaked to indicate this, and the client will wait first for the
lock to free, and then for the flush as normal. Second, the wait
depends on the active lock rather than the domain_status for
consistency purposes.

Closes #32
2019-05-10 23:52:24 -04:00
Joshua Boniface 045ad131af Update node.py to allow API use
Addresses #33
2019-05-10 23:27:34 -04:00
Joshua Boniface b6ecd36588 Implement domain log watching
Implements the ability for a client to watch almost-live domain
console logs from the hypervisors. It does this using a deque-based
"tail -f" mechanism (with a configurable buffer per-VM) that watches
the domain console logfile in the (configurable) directory every
half-second. It then stores the current buffer in Zookeeper when
changed, where a client can then request it, either as a static piece
of text in the `less` pager, or via a similar "tail -f" functionality
implemented using fixed line splitting and comparison to provide a
generally-seamless output.

Enabling this feature requires each guest VM to implement a Libvirt
serial log and write its (text) console to it, for example using the
default logging directory:

```
<serial type='pty'>
    <log file='/var/log/libvirt/vmname.log' append='off'/>
<serial>
```

The append mode can be either on or off; on grows files unbounded,
off causes the log (and hence the PVC log data) to be truncated on
initial VM startup from offline. The administrator must choose how
they best want to handle this until Libvirt implements their own
clog-type logging format.
2019-05-10 23:26:59 -04:00
Joshua Boniface 5ad2dda6d4 Only print file name 2019-05-10 23:26:59 -04:00
Joshua Boniface 9809b798cc Correct bug in non-editor modify 2019-05-10 23:26:59 -04:00
Joshua Boniface ba4a44f5e7 Allow the zk_conn to be none properly 2019-05-10 23:26:59 -04:00
Joshua Boniface 2e4c0c8eb9 Correct header file name 2019-03-20 12:06:15 -04:00
Joshua Boniface 55ce75aa9f Remove junk file 2019-03-20 12:05:35 -04:00
Joshua Boniface 9f0d7c5bf8 Support VM list filtering by state 2019-03-20 11:31:54 -04:00
Joshua Boniface 3ec09686d2 Don't load provisioner 2019-03-17 12:56:52 -04:00
Joshua Boniface 946442ae38 Add support for bridge-only VNIs 2019-03-15 13:54:11 -04:00
Joshua Boniface 9a3f0b5737 Use consistent -t/--target naming 2019-03-12 23:17:31 -04:00
Joshua Boniface 498259fc29 Add correct yaml open 2019-03-12 23:11:10 -04:00
Joshua Boniface 5be1cdc40a Support YAML in the client and update configfile 2019-03-12 22:55:29 -04:00
Joshua Boniface c4a97f04a7 Support raw VM listing 2019-03-12 21:30:01 -04:00
Joshua Boniface 081d855a03 Allow dumping of VMs 2019-03-12 21:09:54 -04:00
Joshua Boniface 24144ea0ba Go back to simple init for now 2019-03-11 01:49:11 -04:00
Joshua Boniface fc890cc606 Initial work on bootstrap setup 2019-03-10 20:40:03 -04:00
Joshua Boniface 4c1e1b4622 Make everything work with dual-stack 2018-11-14 00:26:52 -05:00
Joshua Boniface 6fe06c31bb Support more IPv6 stuff including list and info updates 2018-11-13 01:45:18 -05:00
Joshua Boniface 9fcce4b09a Support setting a CRUSH weight on new OSDs 2018-11-01 23:03:27 -04:00
Joshua Boniface 2ea8a14ba4 Support OSD out/in and commands 2018-11-01 22:08:11 -04:00
Joshua Boniface 99fcb21e3b Support adding and removing Ceph pools 2018-10-31 23:38:17 -04:00
Joshua Boniface 9a271bda0a Add OSD list function 2018-10-30 09:17:32 -04:00
Joshua Boniface bfbe9188ce Finish setup of Ceph OSD addition and basic management 2018-10-29 17:51:25 -04:00
Joshua Boniface 7f18c99172 Add OSD add and OSD remove commands 2018-10-28 22:15:25 -04:00
Joshua Boniface 103ae95fb9 Add Ceph commands and status handling 2018-10-27 18:11:58 -04:00
Joshua Boniface 7542296e8c Handle the zookeeper conn better; add ceph key for later 2018-10-25 11:54:05 -04:00
Joshua Boniface a3f7419124 Handle colouring of network lists when a network is invalid 2018-10-20 15:27:07 -04:00
Joshua Boniface 187a572c13 Make a whole bunch of things work 2018-10-17 20:05:22 -04:00
Joshua Boniface c726865b89 Support writing ACLs 2018-10-17 00:23:27 -04:00
Joshua Boniface f198f62563 Massive rejigger into single daemon
Completely restructure the daemon code to move the 4 discrete daemons
into a single daemon that can be run on every hypervisor. Introduce the
idea of a static list of "coordinator" nodes which are configured at
install time to run Zookeeper and FRR in router mode, and which are
allowed to take on client network management duties (gateway, DHCP, DNS,
etc.) while also allowing them to run VMs (i.e. no dedicated "router"
nodes required).
2018-10-14 02:40:54 -04:00