parallelvirtualcluster
/
pvc
1
0
Fork 0
Code Issues 11 Pull Requests Projects Releases 111 Wiki Activity
3,232 Commits 1 Branch 113 Tags 28 MiB
Commit Graph

628 Commits

Author SHA1 Message Date
Joshua Boniface aa035a61a7 Correct latency units and format name 2021-10-03 17:06:34 -04:00
Joshua Boniface 7c8ba56561 Revamp test result display 
Instead of showing CLAT percentiles, which are very hard to interpret
and understand, instead use the main latency buckets.
 2021-10-03 15:49:01 -04:00
Joshua Boniface 7c122ac921 Add benchmark format to list 2021-10-03 15:05:58 -04:00
Joshua Boniface 0dbf139706 Adjust ETA for benchmarks 2021-10-02 04:51:01 -04:00
Joshua Boniface c909beaf6d Add format parsing for format 1 storage benchmarks 2021-10-02 04:46:44 -04:00
Joshua Boniface 2da49297d2 Add version 2 benchmark list formatting 2021-10-02 02:47:17 -04:00
Joshua Boniface 0058f19d88 Fix handling of array of information 
With a benchmark info we only ever want test one, so pass only that to
the formatter. Simplifies the format function.
 2021-10-02 01:28:39 -04:00
Joshua Boniface 056cf3740d Avoid versioning benchmark lists 
This wouldn't work since each individual test is versioned. Instead add
a placeholder for later once additional format(s) are defined.
 2021-10-02 01:25:18 -04:00
Joshua Boniface 58f174b87b Add format option to benchmark info 
Allows specifying of raw json or json-pretty formats in addition to the
"pretty" formatted option.
 2021-10-02 01:13:50 -04:00
Joshua Boniface 37b98fd54f Add benchmark format function support 
Allows choosing different list and info functions based on the benchmark
version found. Currently only implements "legacy" version 0 with more to
be added.
 2021-10-02 01:07:25 -04:00
Joshua Boniface ce06e4d81b Load benchmark results as JSON 
Load the JSON at the API side instead of client side, because that's
what the API doc says it is and it just makes more sense.
 2021-09-30 23:40:24 -04:00
Joshua Boniface 23977b04fc Bump version to 0.9.37 2021-09-30 02:08:14 -04:00
Joshua Boniface bb1cca522f Revamp benchmark tests 
1. Move to a time-based (60s) benchmark to avoid these taking an absurd
amount of time to show the same information.

2. Eliminate the 256k random benchmarks, since they don't really add
anything.

3. Add in a 4k single-queue benchmark as this might provide valuable
insight into latency.

4. Adjust the output to reflect the above changes.

While this does change the benchmarking, this should not invalidate any
existing benchmarks since most of the test suit is unchanged (especially
the most important 4M sequential and 4K random tests). It simply removes
an unused entry and adds a more helpful one. The time-based change
should not significantly affect the results either, just reduces the
total runtime for long-tests and increase the runtime for quick tests to
provide a better picture.
 2021-09-29 20:51:30 -04:00
Joshua Boniface ee4266f8ca Tweak CLI helptext around OSD actions 
Adds some more detail about OSD commands and their values.
 2021-09-26 01:29:23 -04:00
Joshua Boniface a438a4155a Fix OSD creation for partition paths and fix gdisk 
The previous implementation did not work with /dev/nvme devices or any
/dev/disk/by-* devices due to some logical failures in the partition
naming scheme, so fix these, and be explicit about what is supported in
the PVC CLI command output.

The 'echo | gdisk' implementation of partition creation also did not
work due to limitations of subprocess.run; instead, use sgdisk which
allows these commands to be written out explicitly and is included in
the same package as gdisk.
 2021-09-26 00:12:28 -04:00
Joshua Boniface 65df807b09 Add support for configurable OSD DB ratios 
The default of 0.05 (5%) is likely ideal in the initial implementation,
but allow this to be set explicitly for maximum flexibility in
space-constrained or performance-critical use-cases.
 2021-09-24 01:06:39 -04:00
Joshua Boniface d0f3e9e285 Bump version to 0.9.36 2021-09-23 14:01:38 -04:00
Joshua Boniface adc8a5a3bc Add separate OSD DB device support 
Adds in three parts:

1. Create an API endpoint to create OSD DB volume groups on a device.
Passed through to the node via the same command pipeline as
creating/removing OSDs, and creates a volume group with a fixed name
(osd-db).

2. Adds API support for specifying whether or not to use this DB volume
group when creating a new OSD via the "ext_db" flag. Naming and sizing
is fixed for simplicity and based on Ceph recommendations (5% of OSD
size). The Zookeeper schema tracks the block device to use during
removal.

3. Adds CLI support for the new and modified API endpoints, as well as
displaying the block device and DB block device in the OSD list.

While I debated supporting adding a DB device to an existing OSD, in
practice this ended up being a very complex operation involving stopping
the OSD and setting some options, so this is not supported; this can be
specified during OSD creation only.

Closes #142
 2021-09-23 13:59:49 -04:00
Joshua Boniface 772807deb3 Bump version to 0.9.35 2021-09-13 02:20:46 -04:00
Joshua Boniface e962743e51 Add VM device hot attach/detach support 
Adds a new API endpoint to support hot attach/detach of devices, and the
corresponding client-side logic to use this endpoint when doing VM
network/storage add/remove actions.

The live attach is now the default behaviour for these types of
additions and removals, and can be disabled if needed.

Closes #141
 2021-09-12 19:33:00 -04:00
Joshua Boniface 694b8e85a0 Bump version to 0.9.34 2021-08-24 16:15:25 -04:00
Joshua Boniface 5b81e59481 Convert argument elements of task status to types 2021-08-23 14:28:12 -04:00
Joshua Boniface a18cef5f25 Bump version to 0.9.33 2021-08-21 03:28:48 -04:00
Joshua Boniface f6c5aa9992 Avoid failing if no provisioner tasks 2021-08-21 03:25:16 -04:00
Joshua Boniface afdf254297 Bump version to 0.9.32 2021-08-19 12:37:58 -04:00
Joshua Boniface 13e309b450 Fix colours of network status elements 2021-08-18 19:41:53 -04:00
Joshua Boniface 7ecc6a2635 Bump version to 0.9.31 2021-07-30 12:08:12 -04:00
Joshua Boniface 2a99a27feb Bump version to 0.9.30 2021-07-20 00:01:45 -04:00
Joshua Boniface fa1d93e933 Bump version to 0.9.29 2021-07-19 16:55:41 -04:00
Joshua Boniface 71e4d0b32a Bump version to 0.9.28 2021-07-19 09:29:34 -04:00
Joshua Boniface f16bad4691 Revamp confirmation options for vm modify 
Before, "-y"/"--yes" only confirmed the reboot portion. Instead, modify
this to confirm both the diff portion and the restart portion, and add
separate flags to bypass one or the other independently, ensuring the
administrator has lots of flexibility. UNSAFE mode implies "-y" so both
would be auto-confirmed if that option is set.
 2021-07-19 00:25:43 -04:00
Joshua Boniface 15d92c483f Bump version to 0.9.27 2021-07-19 00:03:40 -04:00
Joshua Boniface 7dd17e71e7 Fix bug with VM editing with file 
Current config is needed for the diff but it was in a conditional.
 2021-07-19 00:02:19 -04:00
Joshua Boniface 602093029c Bump version to 0.9.26 2021-07-18 20:49:52 -04:00
Joshua Boniface 2358ad6bbe Reduce the number of lines per call 
500 was a lot every half second; 200 seems more reasonable. Even a fast
kernel boot should generate < 200 lines in half a second.
 2021-07-18 20:23:45 -04:00
Joshua Boniface a0e9b57d39 Increase log line frequency 2021-07-18 20:19:59 -04:00
Joshua Boniface 2d48127e9c Use even better/faster set comparison 2021-07-18 20:18:35 -04:00
Joshua Boniface 55f2b00366 Add some spaces for better readability 2021-07-18 20:18:23 -04:00
Joshua Boniface ba257048ad Improve output formatting of node logs 2021-07-18 20:06:08 -04:00
Joshua Boniface a088aa4484 Add node log functions to API and CLI 2021-07-18 18:54:28 -04:00
Joshua Boniface 8f3b68d48a Mention multiple option for tags in VM define 2021-07-14 01:12:10 -04:00
Joshua Boniface 6d4c26c8d8 Don't show tag line in info if no tags 2021-07-14 00:59:24 -04:00
Joshua Boniface 75fb60b1b4 Add VM list filtering by tag 
Uses same method as state or node filtering, rather than altering how
the main LIMIT field works.
 2021-07-14 00:59:20 -04:00
Joshua Boniface 9ea9ac3b8a Revamp tag handling and display 
Add an additional protected class, limit manipulation to one at a time,
and ensure future flexibility. Also makes display consistent with other
VM elements.
 2021-07-13 22:39:52 -04:00
Joshua Boniface 2e9f6ac201 Bump version to 0.9.25 2021-07-11 23:19:09 -04:00
Joshua Boniface 551bae2518 Bump version to 0.9.24 2021-07-09 15:58:36 -04:00
Joshua Boniface 9d0eb20197 Mention UUID matching in vm list help 2021-07-09 11:51:20 -04:00
Joshua Boniface 3f5b7045a2 Allow raw listing of cluster names in CLI 2021-07-09 10:53:20 -04:00
Joshua Boniface 7082982a33 Bump version to 0.9.23 2021-07-05 23:40:32 -04:00
Joshua Boniface 54f82a3ea0 Fix bug in VM network list with SR-IOV 2021-07-05 15:14:01 -04:00
Joshua Boniface 37cd278bc2 Bump version to 0.9.22 2021-07-05 14:18:51 -04:00
Joshua Boniface 087c23859c Adjust layout of Provisioner lists output 
Use the same header format as the others.
 2021-07-05 14:06:22 -04:00
Joshua Boniface 6c21a52714 Adjust layout of Ceph/storage lists output 
Use the same header format as node, VM, and network lists.
 2021-07-05 12:57:18 -04:00
Joshua Boniface afde436cd0 Adjust layout of Network lists output 
Use the same header format as node and VM lists.
 2021-07-05 11:48:39 -04:00
Joshua Boniface 1fe71969ca Adjust layout of VM list output 
Matches the new node list output format with the additional header line,
as well as revamps some other aspects:

    1. Adjusts the UUID to be under the name in the info output.
    2. Removes the UUID from the list output to save space, because this
       is generally not needed in day-to-day quick-list output.
    3. Renames the "Node" header to "Current" to better reflect what
       that column actually means and avoid conflicting with the parent
       header.
 2021-07-05 10:52:48 -04:00
Joshua Boniface 2b04df22a6 Add PVC version to node information output 
Also adjusts the layout of the node list output to avoid excessively
long lines. Adds another header line with categories and spacing dashes
for easier visual parsing.
 2021-07-05 10:45:20 -04:00
Joshua Boniface fe0a1d582a Bump version to 0.9.21 2021-06-29 19:21:31 -04:00
Joshua Boniface ffdd6bf3f8 Fix typo in command argument 2021-06-29 18:22:39 -04:00
Joshua Boniface bbb132414c Restore shebang and don't do store if completion 2021-06-23 05:26:50 -04:00
Joshua Boniface 04fa63f081 Only hit the network endpoint once 
Otherwise this is hit for every VM which gets very slow very fast.
 2021-06-23 05:15:48 -04:00
Joshua Boniface f248d579df Convert pvc-client-cli into a proper Python module 
Also fixes up the Debian packaging such that this works how I would
want, with proper module installation while leaving everything else
untouched. Finally implements automatic installation and removal of the
BASH completion for the PVC command.
 2021-06-23 05:03:19 -04:00
Joshua Boniface 1ae34c1960 Fix bad messages in volume remove 2021-06-22 04:31:02 -04:00
Joshua Boniface e6b26745ce Adjust some help messages in pvc.py 2021-06-22 03:40:21 -04:00
Joshua Boniface 7d2a3b5361 Ensure Macvtap NICs can use a model 
Defaults to virtio like a bridged NIC. Otherwise performance is abysmal.
 2021-06-22 02:38:16 -04:00
Joshua Boniface 1787a970ab Fix bug in address check format string 2021-06-22 02:21:32 -04:00
Joshua Boniface b532bc9104 Add missing managed flag for hostdev 2021-06-21 23:22:36 -04:00
Joshua Boniface eeb83da97d Add support for SR-IOV NICs to VMs 2021-06-21 23:18:22 -04:00
Joshua Boniface 13cc0f986f Implement SR-IOV VF config set 
Also fixes some random bugs, adds proper interface sorting, and assorted
tweaks.
 2021-06-21 18:40:11 -04:00
Joshua Boniface e13baf8bd3 Add initial SR-IOV list/info to CLI 2021-06-21 17:12:53 -04:00
Joshua Boniface 164becd3ef Fix info and list matching 2021-06-15 02:32:34 -04:00
Joshua Boniface da48304d4a Avoid hackery in VNI list and support direct type 2021-06-15 00:31:13 -04:00
Joshua Boniface ddd3eeedda Remove needless literal_eval statements 2021-06-14 01:46:30 -04:00
Joshua Boniface a2205bec13 Allow VM dump to file directly 
Similar to the cluster backup task.
 2021-06-13 22:32:54 -04:00
Joshua Boniface 7727221b59 Correctly use the Click file in backups 2021-06-13 22:17:35 -04:00
Joshua Boniface a48bf2d71e More gracefully handle none selectors 
Allow selection of "none" as the node selector, and handle this by
always using the cluster default instead of writing it in.
 2021-06-01 11:13:13 -04:00
Joshua Boniface a1969eb981 Allow overwrite during init command 2021-05-31 00:12:28 -04:00
Joshua Boniface 3128c8fa70 Correct flawed conditional in some commands 2021-05-25 09:59:20 -04:00
Joshua Boniface f46c2e7f6a Implement VM rename functionality 
Closes #125
 2021-05-23 17:21:19 -04:00
Joshua Boniface 1b8b101b64 Fix bugs in log follow command 2021-05-19 16:22:48 -04:00
Joshua Boniface 62c84664fc Handle restart confirmation for VMs more nicely 
For the "vm modify", revamp the way confirmations are presented. Do the
edits/load, show changes, verify XML, then prompt to write and the
restart. The previous order didn't make much sense.

For any of these `--restart` triggered VM modifications, also alter how
the confirmation works. If the user declines the restart, do not abort;
instead, just set restart=False and continue with the modification.
 2021-04-13 10:35:26 -04:00
Joshua Boniface 66f1ac35ab Skip an empty local cluster in cluster list 2021-04-13 10:01:49 -04:00
Joshua Boniface 529f99841d Improve formatting of some help messages 2021-04-12 15:55:59 -04:00
Joshua Boniface 6246b8dfb3 Fix help message output on root command 2021-04-08 14:27:55 -04:00
Joshua Boniface 629cf62385 Add confirmation flag to disruptive VM operations 
Also add some additional output when --restart is not selected.

Closes #118
 2021-04-08 13:33:10 -04:00
Joshua Boniface dfa3432601 Add unsafe envvar/flag option 
Allows another way (beyond --yes) to avoid confirming "unsafe"
operations. While there is probably nearly zero usecase for this (at
least to any sane admin), it is provided to allow maximum flexibility.
 2021-04-08 12:48:38 -04:00
Joshua Boniface 62213fab99 Add description field to CLI clusters 
Allow specifying a textual description of the cluster in the client for
ease of management.
 2021-04-08 12:28:23 -04:00
Joshua Boniface 0bf276fd51 Update copyright year in headers 2021-03-25 17:01:55 -04:00
Joshua Boniface f4ec161aa2 Update file copyright header. 
Remove the option to select a later version of the GPL.
 2021-03-25 16:58:02 -04:00
Joshua Boniface 78925be090 Correct bad help string 2021-03-17 13:58:23 -04:00
Joshua Boniface 1b6613c280 Add live VNC information to domain output 
Sets in the node daemon, returns via the API, and shows in the CLI,
information about the live VNC listen address and port for VNC-enabled
VMs.

Closes #115
 2020-12-20 16:00:55 -05:00
Joshua Boniface 37c3b4ef80 Validate provisioner userdata with SafeLoader 
Given the issues with FullLoader and its eventual deprecation, just use
SafeLoader instead. Any well-formatted Userdata document should conform.
 2020-12-15 00:30:20 -05:00
Joshua Boniface 938d67f96b Make help strings in network modify more detailed 2020-12-04 04:18:06 -05:00
Joshua Boniface 2338aa64f4 Fix bad param for DHCPv4 config 2020-12-04 04:09:23 -05:00
Joshua Boniface c208898b34 Rename null migration method to any 2020-12-03 17:08:49 -05:00
Joshua Boniface 1d5b9c33b5 Unify handling of API list returns 
Ensure that every API return is handled appropriately as it is a list
now.
 2020-12-02 19:15:33 -05:00
Joshua Boniface 185615e6e8 Don't strip single-element lists 
This was a dumb decision that complicated handling of single-item
entries.
 2020-12-01 03:23:18 -05:00
Joshua Boniface a5d495cfaf Update docs name of init command 2020-11-25 10:36:48 -05:00
Joshua Boniface 8f705c9cc2 Add cluster backup + restore functionality 
Adds cluster backup (JSON dump) and restore functions for use in
disaster recovery.

Further, adds additional confirmation to the initialization (as well as
restore) endpoints to avoid accidental triggering, and also groups the
init, backup, and restore commands in the CLI into a new "task"
subsection.
 2020-11-24 02:39:06 -05:00
Joshua Boniface e8914eabb7 Better handle modifying consoles in templates 
Before, the default False was problematic and would reset consoles if
the template was otherwise modified. Instead switch the flags to be full
true/false flags, and on modify, adjust the default to be None so they
will not be changed.
 2020-11-19 10:28:00 -05:00
Joshua Boniface 0383f31086 Fix linting error 2020-11-17 12:37:33 -05:00
Joshua Boniface 7c520ec00c Add short pretty health output 2020-11-17 12:32:16 -05:00
Joshua Boniface f15253210f Ensure all disk stats default to 0 
Prevents issues with converting None to integers and such.
 2020-11-11 13:13:31 -05:00
Joshua Boniface 1a0aedf01c Up line count to 500 to be sure 2020-11-10 16:17:13 -05:00
Joshua Boniface f729a54a2c Obtain more lines during log follow 2020-11-10 16:14:33 -05:00
Joshua Boniface a38e65be47 Correct issues if no interfaces/disks are present 2020-11-10 16:06:43 -05:00
Joshua Boniface 569dcd84a4 Implement disk modification on the CLI 
Adds functions for listing, adding, and removing disks from the CLI,
without editing the XML directly.

References #101
 2020-11-08 00:57:13 -05:00
Joshua Boniface 3ae6a469aa Improve messages in here docs 2020-11-08 00:57:13 -05:00
Joshua Boniface 9c339f4191 Add proper support for cluster networks 
Supports adding the cluster networks (upstream, cluster, storage) as
valid networks for a VM.

References #101
 2020-11-08 00:57:13 -05:00
Joshua Boniface 1ff5d8bf46 Implement network modification on the CLI 
Adds functions for listing, adding, and removing networks from the CLI,
without editing the XML directly.

References #101
 2020-11-08 00:57:13 -05:00
Joshua Boniface 18d3fc8431 Remove serial and VNC console editors 
References #101
 2020-11-08 00:57:13 -05:00
Joshua Boniface b72c415bf1 Implement memory modification on the CLI 
Adds functions for listing and setting the memory values from the CLI,
without editing the XML directly.

References #101
 2020-11-08 00:57:13 -05:00
Joshua Boniface 03d4be79b7 Implement vCPU modification on the CLI 
Adds functions for listing and setting the vCPU and topology values from
the CLI, without editing the XML directly.

References #101
 2020-11-08 00:57:13 -05:00
Joshua Boniface 5f5f4dd421 Add initial VM modification parent commands 
References #101
 2020-11-08 00:57:13 -05:00
Joshua Boniface 0bf130077c Lint: W605 invalid escape sequence '<char>' 
The previous attempt to correct these with character classes failed.
Instead, use the proper `r'blah'` regex formatting.
 2020-11-07 17:41:09 -05:00
Joshua Boniface 3ed97d209f Revert "Lint: W605 invalid escape sequence '\^'" 
This reverts commit 639937f9c2.
 2020-11-07 17:39:34 -05:00
Joshua Boniface d37e5a0c0b Revert "Lint: W605 invalid escape sequence '\+'" 
This reverts commit f787c4cb14.
 2020-11-07 17:38:03 -05:00
Joshua Boniface 893e7a616c Revert "Lint: W605 invalid escape sequence '\-'" 
This reverts commit db9c6eecad.
 2020-11-07 17:38:02 -05:00
Joshua Boniface 6f66b77a00 Lint: E121/E126 continuation line under/over-indented for hanging indent 2020-11-07 15:06:21 -05:00
Joshua Boniface 9dd2a24ce9 Lint: E226 missing whitespace around arithmetic operator 2020-11-07 14:50:57 -05:00
Joshua Boniface 260b39ebf2 Lint: E302 expected 2 blank lines, found X 2020-11-07 14:45:24 -05:00
Joshua Boniface 22355bbec4 Lint: F821 undefined name 're' 2020-11-07 13:19:48 -05:00
Joshua Boniface 0c221be183 Lint: F821 undefined name 'name' 2020-11-07 13:19:38 -05:00
Joshua Boniface 961ebb4c01 Lint: E305 expected 2 blank lines after class or function definition, found X 2020-11-07 13:17:49 -05:00
Joshua Boniface 7932be3948 Lint: E261 at least two spaces before inline comment 2020-11-07 13:11:03 -05:00
Joshua Boniface d2490419c5 Lint: E202 whitespace before ']' 2020-11-07 13:02:54 -05:00
Joshua Boniface 3f242cd437 Lint: E202 whitespace before '}' 2020-11-07 12:57:42 -05:00
Joshua Boniface b7daa8e1f6 E201 whitespace after '[' 2020-11-07 12:39:59 -05:00
Joshua Boniface e333f2b935 Lint: E201 whitespace after '{' 2020-11-07 12:38:31 -05:00
Joshua Boniface 292ccdd94e Lint: E231 missing whitespace after ':' 2020-11-07 12:34:47 -05:00
Joshua Boniface 905b81c47d Lint: E231 missing whitespace after ',' 2020-11-07 12:34:12 -05:00
Joshua Boniface 97f262f5d2 Lint: E221 multiple spaces before operator 2020-11-07 12:19:25 -05:00
Joshua Boniface 4a5d50d0e6 Lint: E225 missing whitespace around operator 2020-11-07 12:16:36 -05:00
Joshua Boniface 6ab261f4cb Lint: E251 unexpected spaces around keyword / parameter equals 2020-11-07 12:11:53 -05:00
Joshua Boniface 69858788c1 Lint: E227 missing whitespace around bitwise or shift operator 2020-11-06 21:41:10 -05:00
Joshua Boniface 4b47a2424c Lint: E303 too many blank lines (2) 2020-11-06 21:16:52 -05:00
Joshua Boniface cb2defbde9 Lint: W391 blank line at end of file 2020-11-06 21:14:19 -05:00
Joshua Boniface 5da314902f Lint: F841 local variable '<variable>' is assigned to but never used 2020-11-06 21:13:13 -05:00
Joshua Boniface aecb845d6a Lint: E713 test for membership should be 'not in' 2020-11-06 20:37:52 -05:00
Joshua Boniface fde8ea2fea Lint: W291 trailing whitespace 2020-11-06 19:44:14 -05:00
Joshua Boniface db9c6eecad Lint: W605 invalid escape sequence '\-' 2020-11-06 19:40:44 -05:00
Joshua Boniface f787c4cb14 Lint: W605 invalid escape sequence '\+' 2020-11-06 19:40:29 -05:00
Joshua Boniface 57c51d3234 Lint: E711 comparison to None should be 'if cond is not None:' 2020-11-06 19:37:13 -05:00
Joshua Boniface ce01b41d81 Lint: E711 comparison to None should be 'if cond is None:' 2020-11-06 19:36:36 -05:00
Joshua Boniface 4d6f36aca0 Lint: E712 comparison to False should be 'if cond is False:' or 'if not cond:' 2020-11-06 19:35:51 -05:00
Joshua Boniface 72ae149cf1 Lint: E712 comparison to True should be 'if cond is True:' or 'if cond:' 2020-11-06 19:35:19 -05:00
Joshua Boniface fb4aafcea9 Lint: E111 indentation is not a multiple of four 2020-11-06 19:26:22 -05:00
Joshua Boniface d9e7b7ec15 Lint: F401 <library> imported but unused 2020-11-06 19:22:49 -05:00
Joshua Boniface ebf254f62d Lint: W293 blank line contains whitespace 2020-11-06 19:11:07 -05:00
Joshua Boniface 2deee9a329 Lint: E262 inline comment should start with '# ' 2020-11-06 19:03:30 -05:00
Joshua Boniface 639937f9c2 Lint: W605 invalid escape sequence '\^' 2020-11-06 18:59:30 -05:00
Joshua Boniface 63f4f9aed7 Lint: E722 do not use bare 'except' 2020-11-06 18:55:10 -05:00
Joshua Boniface 601ab1a181 Add second default too 2020-11-04 23:53:56 -05:00
Joshua Boniface 2266438303 Show fewer lines on a log follow 2020-11-03 11:16:29 -05:00
Joshua Boniface b783588ee6 Use "none" default instead of "default" 2020-10-29 12:19:07 -04:00
Joshua Boniface ec0b8acf90 Support per-VM migration type selectors 
Allow a VM to specify its migration type as a default choice. The valid
options are "default" (i.e. behave as now), "live" which forces a live
migration only, and "shutdown" which forces a shutdown migration only.
The new option is treated as a VM meta option and is set to default if
not found.
 2020-10-29 12:01:29 -04:00
Joshua Boniface 7a27503f1b Allow network-less managed networks 
Allows the specification of network-less managed networks, acting like
bridged networks but over the VXLAN system instead.

Closes #107
 2020-10-18 23:13:12 -04:00
Joshua Boniface a4b80be5ed Add provisioned memory to node info 
Adds a separate field to the node memory, "provisioned", which totals
the amount of memory provisioned to all VMs on the node, regardless of
state, and in contrast to "allocated" which only counts running VMs.

Allows for the detection of potential overprovisioned states when
factoring in non-running VMs.

Includes the supporting code to get this data, since the original
implementation of VM memory selection was dependent on the VM being
running and getting this from libvirt. Now, if the VM is not active, it
gets this from the domain XML instead.
 2020-10-18 14:17:15 -04:00
Joshua Boniface 9d7067469a Correct proper type of uploads 2020-10-16 11:47:09 -04:00
Joshua Boniface 7ce1bfd930 Fix bad integer/string in base convert 2020-08-28 01:08:48 -04:00
Joshua Boniface 423da08f5f Add colour indication if alloc mem is above total 
Shows an "overprovisioned" state clearly without adding a hacky
additional domain state to the system.
 2020-08-28 00:33:50 -04:00
Joshua Boniface 45542bfd67 Avoid verifying SSL on local connections 
Since these will almost always connect to an IP rather than a "real"
hostname, don't verify the SSL cert (if applicable). Also allow the
overriding of SSL verification via an environment variable.

As a consequence, to reduce spam, SSL warnings are disabled for urllib3.
Instead, we warn in the "Using cluster" output whenever verification is
disabled.
 2020-08-27 23:54:18 -04:00
Joshua Boniface 7bf91b1003 Improve store file handling for CLI 
Don't try to chmod every time, instead only chmod when first creating
the file. Also allow loading the default permission from an envvar
rather than hardcoding it.
 2020-08-27 13:14:55 -04:00
Joshua Boniface 6dc6dae26c Disable gtod_reduce for benchmarks 
This ended up disabling latency measurements entirely, so don't use this
option for benchmarks.
 2020-08-25 17:02:06 -04:00
Joshua Boniface 0089ec4e17 Multiple KiB values by 1024 in detail output 
Since these are KiB and not B. Also fix some other anomalies.
 2020-08-25 15:01:24 -04:00
Joshua Boniface 2731aa060c Finalize tests and output formatting 2020-08-25 12:16:23 -04:00
Joshua Boniface 18bcd39b46 Use nicer header format 2020-08-25 02:11:34 -04:00
Joshua Boniface d210eef200 Parse response message properly 2020-08-25 02:08:35 -04:00
Joshua Boniface 887e14a4e2 Add storage benchmarking to API 2020-08-25 01:57:21 -04:00
Joshua Boniface 15e78aa9f0 Add status information in cluster status 
Provide textual explanations for the degraded status, including
specific node/VM/OSD issues as well as detailed Ceph health. "Single
pane of glass" mentality.
 2020-08-17 12:25:23 -04:00
Joshua Boniface 2b4d980685 Display Ceph health in PVC status as well 
Makes this output a little more realistic and allows proper monitoring
of the Ceph cluster status (separate from the PVC status which is
tracking only OSD up/in state).
 2020-08-13 15:10:57 -04:00
Joshua Boniface 42f2dedf6d Add syntax checking of userdata YAML 2020-08-12 14:09:56 -04:00
Joshua Boniface 4438dd401f Add description to example in network add 
A required field so ensure this is in the example.
 2020-08-05 10:35:41 -04:00
Joshua Boniface 6fe74b34b2 Use .get for JSON message responses 2020-07-20 12:31:12 -04:00
Joshua Boniface ad45f6097f Don't output anything if no results and --raw 2020-07-16 02:35:02 -04:00
Joshua Boniface a1ba9d2eeb Allow specifying arbitrary script_args on CLI 
Allow the specifying of arbitrary provisioner script install() args on
the provisioner create CLI, either overriding or adding additional
per-VM arguments to those found in the profile. Reference example is
setting a "vm_fqdn" on a per-run basis.

Closes #100
 2020-07-08 13:18:12 -04:00
Joshua Boniface 37a58d35e8 Implement limiting of node output 
Closes #98
 2020-06-25 11:51:53 -04:00
Joshua Boniface d74f68c904 Add quiet option to CLI 
Closes #99
 2020-06-25 11:09:55 -04:00
Joshua Boniface 15e986c158 Support storing client config in override dir 2020-06-25 11:07:01 -04:00
Joshua Boniface 2967c97f1a Format and display extra VM statistics 2020-06-07 03:04:36 -04:00
Joshua Boniface ce60836c34 Allow enforcement of live migration 
Provides a CLI and API argument to force live migration, which triggers
a new VM state "migrate-live". The node daemon VMInstance during migrate
will read this flag from the state and, if enforced, will not trigger a
shutdown migration.

Closes #95
 2020-06-06 12:00:44 -04:00
Joshua Boniface da20b4493a Properly return the function 2020-06-05 15:50:43 -04:00
Joshua Boniface 440821b136 Refactor cluster validation into a command wrapper 
Instead of using group-based validation, which breaks the help context
for subcommands, use a decorator to validate the cluster status for each
command. The eager help option will then override this decorator for
help commands, while enforcing it for others.
 2020-06-05 14:49:53 -04:00
Joshua Boniface 9ee5ae4826 Volume and Snapshot are not sorted by ID 2020-05-29 13:43:44 -04:00
Joshua Boniface 48711000b0 Ensure stats sorting is by right key 2020-05-29 13:41:52 -04:00
Joshua Boniface 82c067b591 Sort list output in CLI client properly 2020-05-29 13:39:20 -04:00
Joshua Boniface e724e73140 Don't show built-in bridges as invalid 2020-05-12 10:46:10 -04:00
Joshua Boniface e451426c7c Fix minor bugs from change in VM info handling 2020-04-13 22:56:19 -04:00
Joshua Boniface 8faa3bb53d Handle info fuzzy matches better 
If we are calling info, we want one VM. Don't silently discard other
options or try (and fail later) to parse multiple, just say no VM found.
 2020-04-09 10:26:49 -04:00
Joshua Boniface b3a75d8069 Use post instead of get on initialize 2020-04-06 15:05:33 -04:00
Joshua Boniface c3bd6b6ecc Add missing call into cluster initialize function 2020-04-06 14:48:26 -04:00
Joshua Boniface 09aeb33d13 Don't convert non-integer bytes/ops 2020-03-30 19:09:16 -04:00
Joshua Boniface d84e94eff4 Add force_single_node script 2020-03-25 10:48:49 -04:00
Joshua Boniface ce9d0e9603 Add helper scripts to CLI client 2020-03-22 01:19:55 -04:00
Joshua Boniface 3aea5ae34b Correct invalid function call 2020-03-21 16:46:34 -04:00
Joshua Boniface 36588a3a81 Work around bad RequestArgs handling 2020-03-03 16:48:20 -05:00
Joshua Boniface 11f045f100 Support showing individual userdata and script doc 
Closes #89
 2020-02-27 14:31:08 -05:00
Joshua Boniface 9d5f50f82a Implement progress bars for file uploads 
Provide pretty status bars to indicate upload progress for tasks that
perform large file uploads to the API ('provisioner ova upload' and
'storage volume upload') so the administrator can gauge progress and
estimated time to completion.
 2020-02-20 22:42:19 -05:00
Joshua Boniface 0caea03428 Clean up redundant message output 2020-02-19 16:27:14 -05:00
Joshua Boniface 65932b20d2 Handle request failures more gracefully 2020-02-19 16:19:34 -05:00
Joshua Boniface 1b8b32b07c Don't return tuple value on error 2020-02-19 15:47:08 -05:00
Joshua Boniface 39ce704969 Implement wait for node primary/secondary in CLI 
Use a different wait method of querying the node status every
half-second during the transition, in order to wait on the transition to
complete if desired.

Closes #72
 2020-02-19 14:33:31 -05:00
Joshua Boniface 8678dedfea Revert "Implement wait for node coordinator transition" 
This reverts commit 0aefafa7f7.

This does not work since the API goes away during the transition.

References #72
 2020-02-19 10:50:21 -05:00
Joshua Boniface 0aefafa7f7 Implement wait for node coordinator transition 
References #72
 2020-02-19 10:50:04 -05:00
Joshua Boniface 78780039de Add wait support to VM CLI commands 
References #72
 2020-02-19 10:02:32 -05:00
Joshua Boniface ca68321be3 Allow modification of system templates 
Closes #82
 2020-02-18 16:18:27 -05:00
Joshua Boniface dd44f2f42b Correct formatting error in confirmation 2020-02-17 23:31:03 -05:00
Joshua Boniface db558ec91f Complete implementation of OVA handling 
Add functions for uploading, listing, and removing OVA images to the API
and CLI interfaces. Includes improved parsing of the OVF and creation of
a system_template and profile for each OVA.

Also modifies some behaviour around profiles, making most components
option at creation to support both profile types (and incomplete
profiles generally).

Implementation part 2/3 - remaining: OVA VM creation

References #71
 2020-02-17 23:22:50 -05:00
Joshua Boniface 59ca296c58 Add basic OVA profile support 2020-02-17 12:00:51 -05:00
Joshua Boniface b16e2b4925 Handle CLI wildcard addresses properly 
If the local API instance is listening on a wildcard, connect to
127.0.0.1 instead.
 2020-02-16 20:02:08 -05:00
Joshua Boniface 1de57ab6f3 Add CLI client interface to image upload 
Closes #68
 2020-02-09 20:42:56 -05:00
Joshua Boniface ce985234c3 Use consistent naming of components 
Rename "pvcd" to "pvcnoded", and "pvc-api" to "pvcapid" so names for the
daemons are fully consistent. Update the names of the configuration
files as well to match this new formatting.

References #79
 2020-02-08 19:34:07 -05:00
Joshua Boniface 4505b239eb Rename API and common Debian packages 
Closes #79
 2020-02-08 18:50:38 -05:00
Joshua Boniface e7728b8375 Allow modification of a VM profile 
And fix some bugs around this.

Fixes #76
 2020-01-30 11:45:46 -05:00
Joshua Boniface 4868625809 Add try/except around new log lines 
Fixes #73
 2020-01-30 09:28:47 -05:00
Joshua Boniface ebfd9c1882 Gracefully handle API connect timeouts 
Prevents dumping a massive stacktrace if connection fails.
 2020-01-30 09:24:10 -05:00
Joshua Boniface 8c59707cba Prevent writing invalid XML to config 2020-01-28 10:51:32 -05:00
Joshua Boniface 27e73fc6a9 Fix issues with wait on node flush/ready 
Ensure that the strings are converted to booleans on the API side, and
that we're sending a lowercase true/false (for consistency).
 2020-01-26 23:05:30 -05:00
Joshua Boniface 0892e4c842 Fix overlapping name config in VM define 2020-01-23 11:33:09 -05:00
Joshua Boniface 53819b198c Add info about disk/net ordering in templates 
Ensures that the behaviour of 650ed972a1
is reflected in the help messages of these commands.
 2020-01-20 21:20:41 -05:00
Joshua Boniface 194a642c15 Report errors in log -f 2020-01-15 00:47:14 -05:00
Joshua Boniface 3fbd5fcead Reformat output warning message 2020-01-12 16:07:26 -05:00
Joshua Boniface 003171ea18 Correct syntax error 2020-01-12 16:04:42 -05:00
Joshua Boniface 74ff8b41ea Add prov warning to primary/secondary commands 
Warn the administrator if there are active provisioning jobs while
adjusting the current primary node. This is the simplest, cleanest
solution to #69 without trying to implement any hacks or blocking
operations. The administrator can then decide to revert the action
if needed, or will at least know how many jobs are running/queued and
may need to be cancelled.
 2020-01-12 16:03:51 -05:00
Joshua Boniface c4ef73c953 Refactor provisioner data+list format to match 2020-01-12 14:23:11 -05:00
Joshua Boniface b9177c2f26 Add word Job to task ID output 2020-01-12 14:16:11 -05:00
Joshua Boniface 2057859b9f Add Celery task list output 2020-01-12 14:12:01 -05:00
Joshua Boniface 1683a638cf Remove ceph subcommand from storage 
Move everything from under "storage ceph" to "storage" to simplify the
CLI; additional subclasses can be re-added at a future time if and when
additional storage classes are supported.
 2020-01-11 16:03:40 -05:00
Joshua Boniface 83de40307c Fix missing console_log in CLI VM 2020-01-09 13:53:11 -05:00
Joshua Boniface b6474198a4 Implement cluster maintenance mode 
Implements a "maintenance mode" for PVC clusters. For now, the only
thing this mode does is disable node fencing while the state is true.
This allows the administrator to tell PVC that network connectivity,
etc. might be interrupted and to avoid fencing nodes.

Closes #70
 2020-01-09 10:53:27 -05:00
Joshua Boniface 4cda5ebb52 Correct syntax error 2020-01-09 09:46:58 -05:00
Joshua Boniface 6cd3d5c888 Support no-start/no-define in CLI client 2020-01-08 20:13:26 -05:00
Joshua Boniface 9342bc4959 Add extra newline after debug output 2020-01-08 19:41:01 -05:00
Joshua Boniface 4e5bce4975 Update copyright header year to 2020 2020-01-08 19:38:02 -05:00
Joshua Boniface 6e5065511c Move API call to function and implement keys 
Move all API calls to a new common function called call_api to
facilitate easier future changes. Use this function to implement API key
handling via request header value as well as integrate the request URI
generation and debug output handling.

Closes #65
 2020-01-08 19:34:24 -05:00
Joshua Boniface 10d892c698 Support adding API keys to client configs 
Sets the groundwork for the remainder of #65
 2020-01-08 18:43:02 -05:00
Joshua Boniface c515d63340 Add provision state for VMs 2020-01-08 17:40:02 -05:00
Joshua Boniface 9cab1c6eb5 Don't go unhealthy for migrate/unmigrate VMs 2020-01-08 17:37:07 -05:00
Joshua Boniface 23b1f6cb00 Improve handling of progressbar on success 2020-01-08 17:07:57 -05:00
Joshua Boniface a197d9da80 Support wait watching provisioning task 
Addresses #67
 2020-01-08 14:41:46 -05:00
Joshua Boniface c183bc7fef Use click.confirm for all confirmations 
Addresses #67
 2020-01-08 10:34:27 -05:00
Joshua Boniface 9e7d86d5cf Use click.echo_via_pager for VM log output 
Addresses #67
 2020-01-08 10:06:34 -05:00
Joshua Boniface edbe29d06a Use click.edit for modification tasks 
Closes #63
 2020-01-08 09:33:01 -05:00
Joshua Boniface 2005342a2d Correct verb on VM modify 2020-01-08 09:24:17 -05:00
Joshua Boniface d6182d92fc Strip incoming userdata and script documents 2020-01-06 23:52:29 -05:00
Joshua Boniface f4206c2b44 Use correct method for userdata update 2020-01-06 23:39:47 -05:00
Joshua Boniface 62798f9ee8 Use the passed name to modify commands 2020-01-06 23:37:28 -05:00
Joshua Boniface 2aaccfffa7 Use POST body for large data objects 2020-01-06 23:30:34 -05:00
Joshua Boniface a0542d61d0 Fix bad variable name 2020-01-06 16:30:59 -05:00
Joshua Boniface 456a50a73c Send proper delete for OSD deletion 2020-01-06 12:59:00 -05:00
Joshua Boniface 47c49f7672 Fix click import (again) 2020-01-06 12:55:44 -05:00
Joshua Boniface 27279c9686 Properly import click echo 2020-01-06 12:38:20 -05:00
Joshua Boniface 9ea86cc570 Better handle key values 2020-01-06 11:56:34 -05:00
Joshua Boniface 1113234270 Send DELETE instead of POST 2020-01-06 10:47:01 -05:00
Joshua Boniface 66863a0183 Improve output of cluster 2020-01-06 09:23:35 -05:00
Joshua Boniface 007ee56a46 Fix some bugs and where no-cluster message happens 2020-01-06 09:18:22 -05:00
Joshua Boniface be5c67f696 Add messages when adding/removing clusters to CLI 2020-01-06 09:11:00 -05:00
Joshua Boniface 8364d67c51 Correct bad name 2020-01-05 19:19:19 -05:00
Joshua Boniface a9b8403812 Support viewing source volumes 2020-01-05 19:11:52 -05:00
Joshua Boniface 33709828b1 Handle bad response to logs 2020-01-05 17:07:39 -05:00
Joshua Boniface c39c6e8d86 Mention what to do if no cluster found 2020-01-05 15:00:08 -05:00
Joshua Boniface ef293b7778 Fix syntax error 2020-01-05 13:19:21 -05:00
Joshua Boniface b9fdce6728 Use consistent function and stderr for debug 2020-01-05 12:51:06 -05:00
Joshua Boniface baae9b4982 Print initial cluster message to stderr 
Allows it to be easily filtered out
 2020-01-05 12:38:29 -05:00
Joshua Boniface 08570bd7b9 Don't click.echo except in main program 
Return all the strings we printed directly, so that the caller can print
them normally.
 2020-01-05 12:35:00 -05:00
Joshua Boniface 46f3915ab4 Put list handling in the right place 2020-01-05 11:51:03 -05:00
Joshua Boniface 44753c0609 Always handle single-instance results 2020-01-05 02:55:28 -05:00
Joshua Boniface 4fb0d66f6a Use proper variable name 2020-01-05 00:49:50 -05:00
Joshua Boniface 94c60f9daa Mention aborting on no cluster 2020-01-04 15:00:24 -05:00
Joshua Boniface 089b3d7a43 Better output of job status 2020-01-04 14:45:40 -05:00
Joshua Boniface aa2bb7c94c Add create and status functions 2020-01-04 14:31:22 -05:00
Joshua Boniface b88bdc29af Add profile handling 2020-01-04 14:06:36 -05:00
Joshua Boniface a4ad58e64c Add userdata and script options 2020-01-04 13:04:01 -05:00
Joshua Boniface e5d38ec6bd Implement template CLI functions 2020-01-04 11:58:30 -05:00
Joshua Boniface f80b432d3b Handle the empty list better 2020-01-02 12:18:41 -05:00
Joshua Boniface f624cf9e24 Handle no-good-cluster scenarios 2020-01-02 11:19:11 -05:00
Joshua Boniface cb3cc2782a Add initial bits of provisioner CLI 2020-01-02 11:19:08 -05:00
Joshua Boniface d6bd616b31 Fix some bad output 2019-12-30 14:35:44 -05:00
Joshua Boniface 0ced9b2777 Fix incorrect bolding on output 2019-12-30 14:19:17 -05:00
Joshua Boniface 217b163d4b Remove indented empty lines 2019-12-30 13:29:07 -05:00
Joshua Boniface 18e6192178 Add support for multiple clusters in CLI 2019-12-30 13:27:40 -05:00
Joshua Boniface 18c14cbf77 Handle there being no good nodes/VMs 2019-12-30 09:39:04 -05:00
Joshua Boniface 2325f87389 Correct typo in header 2019-12-30 09:07:41 -05:00
Joshua Boniface d2af6f4508 Complete conversion to API client 2019-12-29 20:33:51 -05:00
Joshua Boniface d2f27cc8fe Add network functions to API client 2019-12-29 16:34:32 -05:00
Joshua Boniface d0b6bb4cc3 Convert VM functions to API client 2019-12-29 16:33:50 -05:00
Joshua Boniface a85d8e5c4d Add some QOL improvements 2019-12-29 16:33:50 -05:00
Joshua Boniface e0c7397703 Complete node API client functions 2019-12-29 16:33:50 -05:00
Joshua Boniface 28a0ce772b Node info moved to API 2019-12-29 16:33:50 -05:00
Joshua Boniface 3d178304f1 Convert some node functions to API 2019-12-29 16:33:50 -05:00
Joshua Boniface a9aeb2e836 Implement log viewing/following with API 2019-12-29 16:33:50 -05:00
Joshua Boniface b7eddc6ae9 Add CLI lib 2019-12-29 16:33:50 -05:00
Joshua Boniface 1f6f486b3b Fix broken unwanted change to pvc.py 2019-12-25 13:35:31 -05:00
Joshua Boniface ebcf0cee43 Add doc endpoint 2019-12-23 23:11:55 -05:00
Joshua Boniface 7901ffb5e3 Handle the list of node_limits even better 2019-12-19 13:22:38 -05:00
Joshua Boniface 1924154dce Improve help text 2019-12-08 23:59:17 -05:00
Joshua Boniface 1fb560e996 Add DNS nameservers to networks 2019-12-08 23:55:45 -05:00
Joshua Boniface 32b4ed66e8 Add VM disable command 
Closes #53
 2019-10-23 23:44:47 -04:00
Joshua Boniface 4bff533571 Add cluster status command 2019-10-22 11:23:12 -04:00
Joshua Boniface 03447d3374 Update copyright string year to include 2019 2019-10-13 12:09:51 -04:00
Joshua Boniface 01c8986381 Update help messages in meta command 2019-10-12 02:08:52 -04:00