Commit Graph

391 Commits

Author SHA1 Message Date
Joshua Boniface 78faa90139 Reformat recent changes with Black 2021-11-06 03:27:07 -04:00
Joshua Boniface 66bfad3109 Fix linting errors F522/F523 unused args 2021-11-06 03:24:50 -04:00
Joshua Boniface c41664d2da Reformat code with Black code formatter
Unify the code style along PEP and Black principles using the tool.
2021-11-06 03:02:43 -04:00
Joshua Boniface e88147db4a Bump version to 0.9.42 2021-10-12 15:25:42 -04:00
Joshua Boniface f13cc04b89 Bump version to 0.9.41 2021-10-09 19:39:21 -04:00
Joshua Boniface 24de0f4189 Add MTU to network creation/modification
Addresses #144
2021-10-09 17:51:32 -04:00
Joshua Boniface c27359c4bf Bump version to 0.9.40 2021-10-07 14:42:04 -04:00
Joshua Boniface bdb9db8375 Bump version to 0.9.39 2021-10-07 11:52:38 -04:00
Joshua Boniface c0f7ba0125 Add limit negation to VM list
When using the "state", "node", or "tag" arguments to a VM list, add
support for a "negate" flag to look for all VMs *not in* the state,
node, or tag state.
2021-10-07 11:50:52 -04:00
Joshua Boniface da9248cfa2 Bump version to 0.9.38 2021-10-03 22:32:41 -04:00
Joshua Boniface 32b3af697c Tweak fio tests for benchmarks
1. Remove ramp_time as this was giving very strange results.

2. Up the runtime to 75 seconds to compensate.

3. Print the fio command to the console to validate.
2021-10-03 15:06:18 -04:00
Joshua Boniface 0ff9a6b8c4 Handle benchmark running state properly 2021-10-02 01:54:51 -04:00
Joshua Boniface 28377178d2 Fix missing argument in database insert 2021-10-02 01:49:47 -04:00
Joshua Boniface e06b114c48 Update to storage benchmark format 1
1. Runs `fio` with the `--format=json` option and removes all terse
format parsing from the results.

2. Adds a 15-second ramp time to minimize wonky ramp-up results.

3. Sets group_reporting, which isn't necessary with only a single job,
but is here for consistency.
2021-10-02 01:41:08 -04:00
Joshua Boniface f83a345bfe Add test format versioning to storage benchmarks
Adds a test_format database column and a value in the API return for the
test format version, starting at 0 for the existing format as of 0.9.37.

References #143
2021-10-02 00:55:27 -04:00
Joshua Boniface ce06e4d81b Load benchmark results as JSON
Load the JSON at the API side instead of client side, because that's
what the API doc says it is and it just makes more sense.
2021-09-30 23:40:24 -04:00
Joshua Boniface 23977b04fc Bump version to 0.9.37 2021-09-30 02:08:14 -04:00
Joshua Boniface bb1cca522f Revamp benchmark tests
1. Move to a time-based (60s) benchmark to avoid these taking an absurd
amount of time to show the same information.

2. Eliminate the 256k random benchmarks, since they don't really add
anything.

3. Add in a 4k single-queue benchmark as this might provide valuable
insight into latency.

4. Adjust the output to reflect the above changes.

While this does change the benchmarking, this should not invalidate any
existing benchmarks since most of the test suit is unchanged (especially
the most important 4M sequential and 4K random tests). It simply removes
an unused entry and adds a more helpful one. The time-based change
should not significantly affect the results either, just reduces the
total runtime for long-tests and increase the runtime for quick tests to
provide a better picture.
2021-09-29 20:51:30 -04:00
Joshua Boniface 9a4dce4e4c Add primary node to benchmark job name
Ensures tracking of the current primary node the job was run on, since
this may be relevant for performance reasons.
2021-09-28 09:58:22 -04:00
Joshua Boniface 65df807b09 Add support for configurable OSD DB ratios
The default of 0.05 (5%) is likely ideal in the initial implementation,
but allow this to be set explicitly for maximum flexibility in
space-constrained or performance-critical use-cases.
2021-09-24 01:06:39 -04:00
Joshua Boniface d0f3e9e285 Bump version to 0.9.36 2021-09-23 14:01:38 -04:00
Joshua Boniface adc8a5a3bc Add separate OSD DB device support
Adds in three parts:

1. Create an API endpoint to create OSD DB volume groups on a device.
Passed through to the node via the same command pipeline as
creating/removing OSDs, and creates a volume group with a fixed name
(osd-db).

2. Adds API support for specifying whether or not to use this DB volume
group when creating a new OSD via the "ext_db" flag. Naming and sizing
is fixed for simplicity and based on Ceph recommendations (5% of OSD
size). The Zookeeper schema tracks the block device to use during
removal.

3. Adds CLI support for the new and modified API endpoints, as well as
displaying the block device and DB block device in the OSD list.

While I debated supporting adding a DB device to an existing OSD, in
practice this ended up being a very complex operation involving stopping
the OSD and setting some options, so this is not supported; this can be
specified during OSD creation only.

Closes #142
2021-09-23 13:59:49 -04:00
Joshua Boniface 772807deb3 Bump version to 0.9.35 2021-09-13 02:20:46 -04:00
Joshua Boniface e962743e51 Add VM device hot attach/detach support
Adds a new API endpoint to support hot attach/detach of devices, and the
corresponding client-side logic to use this endpoint when doing VM
network/storage add/remove actions.

The live attach is now the default behaviour for these types of
additions and removals, and can be disabled if needed.

Closes #141
2021-09-12 19:33:00 -04:00
Joshua Boniface 694b8e85a0 Bump version to 0.9.34 2021-08-24 16:15:25 -04:00
Joshua Boniface eb321497ee Properly handle =-separated fsargs 2021-08-24 11:40:22 -04:00
Joshua Boniface a18cef5f25 Bump version to 0.9.33 2021-08-21 03:28:48 -04:00
Joshua Boniface afdf254297 Bump version to 0.9.32 2021-08-19 12:37:58 -04:00
Joshua Boniface 7ecc6a2635 Bump version to 0.9.31 2021-07-30 12:08:12 -04:00
Joshua Boniface c460aa051a Add missing floppy RASD type for compat 2021-07-27 16:32:32 -04:00
Joshua Boniface 2a99a27feb Bump version to 0.9.30 2021-07-20 00:01:45 -04:00
Joshua Boniface fa1d93e933 Bump version to 0.9.29 2021-07-19 16:55:41 -04:00
Joshua Boniface 71e4d0b32a Bump version to 0.9.28 2021-07-19 09:29:34 -04:00
Joshua Boniface 15d92c483f Bump version to 0.9.27 2021-07-19 00:03:40 -04:00
Joshua Boniface 602093029c Bump version to 0.9.26 2021-07-18 20:49:52 -04:00
Joshua Boniface bd7a773d6b Add node log following functionality 2021-07-18 20:37:53 -04:00
Joshua Boniface a088aa4484 Add node log functions to API and CLI 2021-07-18 18:54:28 -04:00
Joshua Boniface 401f102344 Add serial BIOS to default libvirt schema 2021-07-15 10:45:14 -04:00
Joshua Boniface 75fb60b1b4 Add VM list filtering by tag
Uses same method as state or node filtering, rather than altering how
the main LIMIT field works.
2021-07-14 00:59:20 -04:00
Joshua Boniface 9ea9ac3b8a Revamp tag handling and display
Add an additional protected class, limit manipulation to one at a time,
and ensure future flexibility. Also makes display consistent with other
VM elements.
2021-07-13 22:39:52 -04:00
Joshua Boniface 27f1758791 Add tags manipulation to API
Also fixes some checks for Metadata too since these two actions are
almost identical, and adds tags to define endpoint.
2021-07-13 19:05:33 -04:00
Joshua Boniface c0a3467b70 Simplify VM metadata reads
Directly call the new common getDomainMetadata function to avoid
excessive Zookeeper calls for this information.
2021-07-13 19:05:33 -04:00
Joshua Boniface 2e9f6ac201 Bump version to 0.9.25 2021-07-11 23:19:09 -04:00
Joshua Boniface 8c975e5c46 Add chroot context manager example to debootstrap
Closes #132
2021-07-11 23:10:41 -04:00
Joshua Boniface 551bae2518 Bump version to 0.9.24 2021-07-09 15:58:36 -04:00
Joshua Boniface 7082982a33 Bump version to 0.9.23 2021-07-05 23:40:32 -04:00
Joshua Boniface 37cd278bc2 Bump version to 0.9.22 2021-07-05 14:18:51 -04:00
Joshua Boniface 47a522f8af Use manual zkhandler creation in Benchmark job
Like the other Celery job this does not work properly with the
ZKConnection decorator due to conflicting "self", so just connect
manually exactly like the provisioner task does.
2021-07-05 14:12:56 -04:00
Joshua Boniface a69105569f Add node PVC version data to Node information
Allows API client to see the currently-active version of the node
daemon.
2021-07-05 09:57:38 -04:00
Joshua Boniface f12de6727d Adjust logo slightly and add debug state 2021-07-02 02:32:08 -04:00
Joshua Boniface e94f5354e6 Update startup messages with new ASCII logo 2021-07-02 02:21:30 -04:00
Joshua Boniface 61465ef38f Add profiler to several other functions in API 2021-07-02 01:53:19 -04:00
Joshua Boniface 20542c3653 Add profiler to cluster status function 2021-07-01 17:35:29 -04:00
Joshua Boniface fe0a1d582a Bump version to 0.9.21 2021-06-29 19:21:31 -04:00
Joshua Boniface b05c93e260 Fix bad return from initialize call 2021-06-29 18:31:56 -04:00
Joshua Boniface aae9ae2e80 Fix incorrect handling of overwrite flag 2021-06-29 18:22:01 -04:00
Joshua Boniface 4e2a1c3e52 Add worker wrapper to fix Deb incompatibility
Celery 5.x introduced a new worker argument format that is not
backwards-compatible with the older Celery 4.x format. This created a
conundrum since we use one service unit for both Debian 10 (4.x) and
Debian 11 (5.x). Instead of worse hacks, create a wrapper script to
start the worker with the correct arguments instead.
2021-06-28 12:19:29 -04:00
Joshua Boniface 93c2fdec93 Swap order of networks and disks in provisioner
Done to make the resulting config match the expectations when using "vm
network add", which is that networks are below disks, not above.

Not a functional change, just ensures the VM XML is consistent after
many changes.
2021-06-21 21:59:57 -04:00
Joshua Boniface 13cc0f986f Implement SR-IOV VF config set
Also fixes some random bugs, adds proper interface sorting, and assorted
tweaks.
2021-06-21 18:40:11 -04:00
Joshua Boniface ae480d6cc1 Add SR-IOV listing/info endpoints to API 2021-06-21 17:12:45 -04:00
Joshua Boniface a697c2db2e Add SRIOV PF and VF listing to API 2021-06-21 01:42:55 -04:00
Joshua Boniface d2bcfe5cf7 Bump version to 0.9.20 2021-06-14 18:06:27 -04:00
Joshua Boniface d2f93b3a2e Fix call to celery 2021-06-13 14:56:09 -04:00
Joshua Boniface 01c82f5d19 Move backup and restore into common 2021-06-13 14:25:51 -04:00
Joshua Boniface ae269bdfde Add scripts to generate ZK migration JSON 2021-06-09 00:04:38 -04:00
Joshua Boniface cf96bb009f Bump version to 0.9.19 2021-06-06 01:47:41 -04:00
Joshua Boniface 3eedfaa7d5 Collect database model error 2021-06-03 00:22:48 -04:00
Joshua Boniface 7de7e1bc71 Properly handle cluster networks in provisioner 2021-06-02 15:57:46 -04:00
Joshua Boniface 34ef055954 Adjust VNI column for provisioner to text
Allows the storing of the textual cluster labels (e.g. 'upstream') as
valid VNI values in the template.
2021-06-02 15:45:22 -04:00
Joshua Boniface a48bf2d71e More gracefully handle none selectors
Allow selection of "none" as the node selector, and handle this by
always using the cluster default instead of writing it in.
2021-06-01 11:13:13 -04:00
Joshua Boniface 33a54cf7f2 Move configuration keys to /config tree 2021-06-01 10:48:55 -04:00
Joshua Boniface a1969eb981 Allow overwrite during init command 2021-05-31 00:12:28 -04:00
Joshua Boniface c7992000eb Explicitly output JSON cluster data 2021-05-30 23:50:42 -04:00
Joshua Boniface a1e8cc5867 Skip patroni tree during backups 2021-05-30 23:39:37 -04:00
Joshua Boniface ac0c3b0ec9 Ensure temp_dir exists before starting
Otherwise some failures throw the wrong error.
2021-05-30 16:04:38 -04:00
Joshua Boniface 60db800d9c Use full ZKHandler in provisioner
Required due to references to self from Celery that are replaced by the
ZKConnection self instance.
2021-05-30 15:59:37 -04:00
Joshua Boniface 73407e245f Move startup code to an entrypoint function
Prevents further issues with startup.
2021-05-30 00:18:04 -04:00
Joshua Boniface 25f80a4478 Move API version string location to Daemon
Prevents a startup bug with pvcapid-manage.py.
2021-05-30 00:11:24 -04:00
Joshua Boniface 163015bd4a Port remaining helper functions to ZKConnection 2021-05-29 00:30:42 -04:00
Joshua Boniface 49bbad8021 Port provisioner to ZKConnection 2021-05-29 00:26:15 -04:00
Joshua Boniface 2c0bafc313 Port benchmark to ZKConnection 2021-05-29 00:24:53 -04:00
Joshua Boniface 1963f2c336 Convert OVA helper to ZKConnection 2021-05-29 00:22:06 -04:00
Joshua Boniface 9cd121ef9f Convert remaining VM functions 2021-05-29 00:16:26 -04:00
Joshua Boniface ea63a58b21 Port two more functions to new decorator 2021-05-28 23:38:53 -04:00
Joshua Boniface 0eceec0341 Disable SQLAlchemy modifcation tracking 2021-05-28 23:36:36 -04:00
Joshua Boniface c6bececb55 Revamp config parsing and imports
Brings sanity to the passing of the config variable around the various
submodules for use in the ZKConnection decorator.
2021-05-28 23:33:36 -04:00
Joshua Boniface 0c75a127b2 Bump version to 0.9.18 2021-05-23 17:23:10 -04:00
Joshua Boniface f46c2e7f6a Implement VM rename functionality
Closes #125
2021-05-23 17:21:19 -04:00
Joshua Boniface 9de14c46fb Bump version to 0.9.17 2021-05-19 17:06:29 -04:00
Joshua Boniface fe15bdb854 Bump version to 0.9.16 2021-05-10 01:13:21 -04:00
Joshua Boniface 669338c22b Bump version to 0.9.15 2021-04-08 13:37:47 -04:00
Joshua Boniface 3330624f70 Fix bump_version to include API 2021-03-30 10:30:01 -04:00
Joshua Boniface 81c1411b8f Add QEMU guest agent to default VM configs 2021-03-29 15:15:31 -04:00
Joshua Boniface 0bf276fd51 Update copyright year in headers 2021-03-25 17:01:55 -04:00
Joshua Boniface f4ec161aa2 Update file copyright header.
Remove the option to select a later version of the GPL.
2021-03-25 16:58:02 -04:00
Joshua Boniface 4e58b18976 Ensure cloned volume size is int type 2021-03-15 15:12:24 -04:00
Joshua Boniface 593c9a7e0d Fix bugs around cloned volume provisioning 2021-03-09 16:59:49 -05:00
Joshua Boniface 9291ce6ffc Correct output of fromhuman and add size compare
Ensures that the bytes_tohuman returns an integer to avoid the hacky
workaround of stripping off the B.

Adds a verification on the size of a new volume, that it is not larger
than the free space of the pool to prevent errors/excessively-large
volumes from being created.

Closes #120
2021-02-17 11:27:26 -05:00
Joshua Boniface dd87951642 Fix bad calls in pool element 2021-02-17 11:14:50 -05:00
Joshua Boniface 094d25dafa Increase indent further 2021-02-14 16:43:51 -05:00
Joshua Boniface f1c0c9325d Fix indentation issue with API daemon 2021-02-14 16:41:52 -05:00
Joshua Boniface f22f291c8b Add additional field and info to Swagger 2021-02-09 01:49:15 -05:00
Joshua Boniface aba567d6c9 Add nice startup banners to both daemons
Add nicer easy-to-find (yay ASCII art) banners for the startup printouts
of both the node and API daemons. Also adds the safe loader to pvcnoded
to prevent hassle messages and a version string in the API daemon file.
2021-02-08 02:51:43 -05:00
Joshua Boniface 1b6613c280 Add live VNC information to domain output
Sets in the node daemon, returns via the API, and shows in the CLI,
information about the live VNC listen address and port for VNC-enabled
VMs.

Closes #115
2020-12-20 16:00:55 -05:00
Joshua Boniface 2ac31e0a14 Handle issues with state retrieval 2020-12-08 23:26:29 -05:00
Joshua Boniface e8c6df49e6 Fix incorrect method on incoming list 2020-12-04 03:49:28 -05:00
Joshua Boniface 0f8e5c6536 Add VNI to VM network API list
Saves some processing on API clients.
2020-12-01 04:44:33 -05:00
Joshua Boniface 593810e53e Add volume_count to pool API data 2020-12-01 03:40:41 -05:00
Joshua Boniface 185615e6e8 Don't strip single-element lists
This was a dumb decision that complicated handling of single-item
entries.
2020-12-01 03:23:18 -05:00
Joshua Boniface 8f705c9cc2 Add cluster backup + restore functionality
Adds cluster backup (JSON dump) and restore functions for use in
disaster recovery.

Further, adds additional confirmation to the initialization (as well as
restore) endpoints to avoid accidental triggering, and also groups the
init, backup, and restore commands in the CLI into a new "task"
subsection.
2020-11-24 02:39:06 -05:00
Joshua Boniface 3f2c7293d1 Fix inconsistent name helpmsg
In the RequestParser this is called helptext, not helpmsg; make all of
the entries consistent and return the issue as a message.
2020-11-24 02:37:28 -05:00
Joshua Boniface beb62c9f3d Readd erroneously removed blk_file.write 2020-11-10 15:33:29 -05:00
Joshua Boniface e6bca5b6a9 Add override args for RequestParser
Properly fixes the issue with OVA upload bodies by allowing the
restriction of the 'location' directive when parsing specific request
args. Thus the 'form' location can be included by default but removed
for those parsers that have a file body.
2020-11-09 10:26:01 -05:00
Joshua Boniface b169620eee Revert "Ensure args are checked against form body"
This reverts commit d63e757c32.

This did not work; by readding 'form' checking, the attempt to isolate
the large file upload was again thwarted. Another solution, perhaps
specific to the uploads, is needed instead.
2020-11-09 09:59:33 -05:00
Joshua Boniface ee4d682b29 Correct faulty function naming 2020-11-09 09:45:54 -05:00
Joshua Boniface a770b65f6b Correct bad field in libvirt schema 2020-11-08 00:57:13 -05:00
Joshua Boniface 0bf130077c Lint: W605 invalid escape sequence '<char>'
The previous attempt to correct these with character classes failed.
Instead, use the proper `r'blah'` regex formatting.
2020-11-07 17:41:09 -05:00
Joshua Boniface 3ed97d209f Revert "Lint: W605 invalid escape sequence '\^'"
This reverts commit 639937f9c2.
2020-11-07 17:39:34 -05:00
Joshua Boniface f33398458e Revert "Lint: W605 invalid escape sequence '\$'"
This reverts commit 76b1cafdcc.
2020-11-07 17:38:03 -05:00
Joshua Boniface 6213d39c42 Revert "Lint: W605 invalid escape sequence '\.'"
This reverts commit 7aaca92cdb.
2020-11-07 17:38:00 -05:00
Joshua Boniface aa6f088a3e Correct issues with model import 2020-11-07 15:24:18 -05:00
Joshua Boniface 6f66b77a00 Lint: E121/E126 continuation line under/over-indented for hanging indent 2020-11-07 15:06:21 -05:00
Joshua Boniface dad1b330e6 Lint: E241 multiple spaces after ':' 2020-11-07 14:52:27 -05:00
Joshua Boniface 260b39ebf2 Lint: E302 expected 2 blank lines, found X 2020-11-07 14:45:24 -05:00
Joshua Boniface 27663cbf87 Lint: E303 too many blank lines (3) 2020-11-07 13:47:25 -05:00
Joshua Boniface 7475252c8e Lint: W291 trailing whitespace 2020-11-07 13:47:02 -05:00
Joshua Boniface 85d3188eb9 Lint: F821 undefined name 'template' 2020-11-07 13:30:32 -05:00
Joshua Boniface 51de6e57d6 Lint: F821 undefined name 'ceph_pool_list' 2020-11-07 13:29:56 -05:00
Joshua Boniface f707e1075a Lint: F821 undefined name 'reqargs'
These were functions that were missing an expected reqargs pass.
2020-11-07 13:28:59 -05:00
Joshua Boniface 8cfb83916e Lint: F821 undefined name 'API_Doc'
Should have been commented out along with its class.
2020-11-07 13:26:59 -05:00
Joshua Boniface d74cf00feb Lint: F821 undefined name 'data'
Not really a lint, but simply makes the image uploader work the same way
that the OVA uploader does. May need more tweaking if this broke it.
2020-11-07 13:26:12 -05:00
Joshua Boniface 6c56d45345 Lint: F821 undefined name 'config'
This variable is set after importing these files by the flaskapi module.
Thus, simply set a default at the top of each file to avoid linting
errors.
2020-11-07 13:23:34 -05:00
Joshua Boniface 961ebb4c01 Lint: E305 expected 2 blank lines after class or function definition, found X 2020-11-07 13:17:49 -05:00
Joshua Boniface 7932be3948 Lint: E261 at least two spaces before inline comment 2020-11-07 13:11:03 -05:00
Joshua Boniface 975b52ad8e Lint: E128 continuation line under-indented for visual indent 2020-11-07 13:07:07 -05:00
Joshua Boniface d2490419c5 Lint: E202 whitespace before ']' 2020-11-07 13:02:54 -05:00
Joshua Boniface d2e5ede399 Lint: E202 whitespace before ')' 2020-11-07 12:58:54 -05:00
Joshua Boniface 3f242cd437 Lint: E202 whitespace before '}' 2020-11-07 12:57:42 -05:00
Joshua Boniface b7daa8e1f6 E201 whitespace after '[' 2020-11-07 12:39:59 -05:00
Joshua Boniface c88965e898 Lint: E201 whitespace after '(' 2020-11-07 12:39:27 -05:00
Joshua Boniface e333f2b935 Lint: E201 whitespace after '{' 2020-11-07 12:38:31 -05:00
Joshua Boniface 292ccdd94e Lint: E231 missing whitespace after ':' 2020-11-07 12:34:47 -05:00
Joshua Boniface 905b81c47d Lint: E231 missing whitespace after ',' 2020-11-07 12:34:12 -05:00
Joshua Boniface cbb65551be Lint: E203 whitespace before ':' 2020-11-07 12:29:12 -05:00
Joshua Boniface c7f629dffc Lint: E301 expected 1 blank line, found 0 2020-11-07 12:28:51 -05:00
Joshua Boniface c1f25d3426 Lint: F403 'from pvcapid.models import *' used; unable to detect undefined names
Just removed entirely since F401 was given if corrected.
2020-11-07 12:26:09 -05:00
Joshua Boniface 7aaca92cdb Lint: W605 invalid escape sequence '\.' 2020-11-07 12:20:59 -05:00
Joshua Boniface 8ba267a59e Lint: E211 whitespace before '['/'(' 2020-11-07 12:20:01 -05:00
Joshua Boniface 39cc992e9b Lint: E306 expected 1 blank line before a nested definition, found 0 2020-11-07 12:17:38 -05:00
Joshua Boniface 4a5d50d0e6 Lint: E225 missing whitespace around operator 2020-11-07 12:16:36 -05:00
Joshua Boniface 2eef6a1c21 Lint: E265 block comment should start with '# ' 2020-11-06 21:32:17 -05:00
Joshua Boniface 4b47a2424c Lint: E303 too many blank lines (2) 2020-11-06 21:16:52 -05:00
Joshua Boniface cb2defbde9 Lint: W391 blank line at end of file 2020-11-06 21:14:19 -05:00
Joshua Boniface 5da314902f Lint: F841 local variable '<variable>' is assigned to but never used 2020-11-06 21:13:13 -05:00
Joshua Boniface 98a573bbc7 Lint: E402 module level import not at top of file 2020-11-06 20:40:32 -05:00
Joshua Boniface aecb845d6a Lint: E713 test for membership should be 'not in' 2020-11-06 20:37:52 -05:00
Joshua Boniface fde8ea2fea Lint: W291 trailing whitespace 2020-11-06 19:44:14 -05:00
Joshua Boniface 2d8f684fc8 Lint: E722 do not use bare 'except' (2) 2020-11-06 19:24:10 -05:00
Joshua Boniface d9e7b7ec15 Lint: F401 <library> imported but unused 2020-11-06 19:22:49 -05:00
Joshua Boniface ebf254f62d Lint: W293 blank line contains whitespace 2020-11-06 19:11:07 -05:00
Joshua Boniface 76b1cafdcc Lint: W605 invalid escape sequence '\$' 2020-11-06 19:01:22 -05:00
Joshua Boniface 639937f9c2 Lint: W605 invalid escape sequence '\^' 2020-11-06 18:59:30 -05:00
Joshua Boniface 63f4f9aed7 Lint: E722 do not use bare 'except' 2020-11-06 18:55:10 -05:00
Joshua Boniface e984f315f1 Correct bug in finding system template list 2020-10-29 12:14:10 -04:00
Joshua Boniface ec0b8acf90 Support per-VM migration type selectors
Allow a VM to specify its migration type as a default choice. The valid
options are "default" (i.e. behave as now), "live" which forces a live
migration only, and "shutdown" which forces a shutdown migration only.
The new option is treated as a VM meta option and is set to default if
not found.
2020-10-29 12:01:29 -04:00
Joshua Boniface d63e757c32 Ensure args are checked against form body
Required for XML definitions but erroneously removed during file parsing
reworking.
2020-10-27 02:30:59 -04:00
Joshua Boniface f85c2c2a75 Remove PyWSGI and move to Flask server
Gevent was completely failure. The API would block during large file
uploads with no obvious solutions beyond "use gunicorn", which is not
suited to this. I originally had this working with the Flask "debug"
server, so just move to using that all the time. SSL is added using a
custom context with the OpenSSL library, so include that as a
dependency.
2020-10-26 01:58:43 -04:00
Joshua Boniface adfe302f71 Move monkeypatch before all imports 2020-10-24 20:53:44 -04:00
Joshua Boniface ffaa4c033f Improve handling of large file uploads
By default, Werkzeug would require the entire file (be it an OVA or
image file) to be uploaded and saved to a temporary, fake file under
`/tmp`, before any further processing could occur. This blocked most of
the execution of these functions until the upload was completed.

This entirely defeated the purpose of what I was trying to do, which was
to save the uploads directly to the temporary blockdev in each case,
thus avoiding any sort of memory or (host) disk usage.

The solution is two-fold:

  1. First, ensure that the `location='args'` value is set in
  RequestParser; without this, the `files` portion would be parsed
  during the argument parsing, which was the original source of this
  blocking behaviour.

  2. Instead of the convoluted request handling that was being done
  originally here, instead entirely defer the parsing of the `files`
  arguments until the point in the code where they are ready to be
  saved. Then, using an override stream_factory that simply opens the
  temporary blockdev, the upload can commence while being written
  directly out to it, rather than using `/tmp` space.

This does alter the error handling slightly; it is impossible to check
if the argument was passed until this point in the code, so it may take
longer to fail if the API consumer does not specify a file as they
should. This is a minor trade-off and I would expect my API consumers to
be sane here.
2020-10-19 01:00:34 -04:00
Joshua Boniface a4b80be5ed Add provisioned memory to node info
Adds a separate field to the node memory, "provisioned", which totals
the amount of memory provisioned to all VMs on the node, regardless of
state, and in contrast to "allocated" which only counts running VMs.

Allows for the detection of potential overprovisioned states when
factoring in non-running VMs.

Includes the supporting code to get this data, since the original
implementation of VM memory selection was dependent on the VM being
running and getting this from libvirt. Now, if the VM is not active, it
gets this from the domain XML instead.
2020-10-18 14:17:15 -04:00
Joshua Boniface 973c78b8e0 Use monkeypatch to allow multithreaded prod flask
Without this tasks were blocking when other task were active (for
instance, any task with --wait). Using the moneypatch, these no longer
block.
2020-08-28 02:09:31 -04:00
Joshua Boniface b51f0a339d Fix bug in SSL enabled WSGI server 2020-08-26 13:52:45 -04:00
Joshua Boniface 6dc6dae26c Disable gtod_reduce for benchmarks
This ended up disabling latency measurements entirely, so don't use this
option for benchmarks.
2020-08-25 17:02:06 -04:00
Joshua Boniface 486408753b Don't print results to output 2020-08-25 13:38:46 -04:00
Joshua Boniface 169e174d85 Fix size of test volume to 8GB 2020-08-25 13:29:22 -04:00
Joshua Boniface eb06c1494e Add API spec for benchmark results 2020-08-25 12:43:16 -04:00
Joshua Boniface bb7b1a2bd0 Remove aggrpct from results
This value is useless to us since we're not running combined read/write
tests at all.
2020-08-25 12:38:49 -04:00
Joshua Boniface 70b9caedc3 Correct typo 2020-08-25 12:23:12 -04:00
Joshua Boniface 2731aa060c Finalize tests and output formatting 2020-08-25 12:16:23 -04:00
Joshua Boniface 1dcc1f6d55 Rename sample database for API
From pvcprov to pvcapi to facilitate the changing nature of this
database and its expansion to benchmark results.
2020-08-25 01:59:35 -04:00
Joshua Boniface 887e14a4e2 Add storage benchmarking to API 2020-08-25 01:57:21 -04:00
Joshua Boniface 3dbdd12d8f Correct invalid comparison in template VNI add 2020-08-18 09:48:56 -04:00
Joshua Boniface 2b4d980685 Display Ceph health in PVC status as well
Makes this output a little more realistic and allows proper monitoring
of the Ceph cluster status (separate from the PVC status which is
tracking only OSD up/in state).
2020-08-13 15:10:57 -04:00
Joshua Boniface 0d470ae5f6 Work around formatting fail 2020-08-12 12:12:16 -04:00
Joshua Boniface 0dd719a682 Use single-quotes so Python isn't confused 2020-08-11 17:24:11 -04:00
Joshua Boniface 5526e13da9 Move all host provisioner steps to a try block
Make the provisioner a bit more robust. This way, even if a provisioning
step fails, cleanup is still performed this preventing the system from
being left in an undefined state requiring manual correction.

Addresses #91
2020-08-06 12:27:10 -04:00
Joshua Boniface bafdcf9f8c Use new_size to match new_name 2020-08-05 10:25:37 -04:00
Joshua Boniface 9f86f12f1a Only parse script_run_args if not None 2020-07-16 02:36:26 -04:00
Joshua Boniface be405caa11 Remove spurious print statement 2020-07-08 13:28:47 -04:00
Joshua Boniface a1ba9d2eeb Allow specifying arbitrary script_args on CLI
Allow the specifying of arbitrary provisioner script install() args on
the provisioner create CLI, either overriding or adding additional
per-VM arguments to those found in the profile. Reference example is
setting a "vm_fqdn" on a per-run basis.

Closes #100
2020-07-08 13:18:12 -04:00
Joshua Boniface 37a58d35e8 Implement limiting of node output
Closes #98
2020-06-25 11:51:53 -04:00
Joshua Boniface 4cdf1f7247 Add statistics values to the API 2020-06-07 02:15:33 -04:00
Joshua Boniface 654a3cb7fa Improve debug output and use ceph df util data 2020-06-06 22:52:49 -04:00
Joshua Boniface ce60836c34 Allow enforcement of live migration
Provides a CLI and API argument to force live migration, which triggers
a new VM state "migrate-live". The node daemon VMInstance during migrate
will read this flag from the state and, if enforced, will not trigger a
shutdown migration.

Closes #95
2020-06-06 12:00:44 -04:00
Joshua Boniface 73eb4fb457 Fix typo of macaddress in dhcp add 2020-05-09 00:15:25 -04:00
Joshua Boniface b91923735c Move some messages around 2020-05-05 16:19:18 -04:00
Joshua Boniface 34c4690d49 Don't convert bytes into KB in OVA import
Doing so can create an image that is 1 sector (512 bytes) too large,
which will then break qemu-img because it's stupid (or, VMDK is stupid,
I haven't decided which is).. Current Ceph rbd commands seem to accept
--size in bytes so this is fine.
2020-05-05 16:14:18 -04:00
Joshua Boniface 3e351bb84a Add additional error checking for profile creation 2020-05-05 15:28:39 -04:00
Joshua Boniface 4a52ff56b9 Catch failures in getPoolInformation
Fixes #90
2020-03-15 16:58:13 -04:00
Joshua Boniface ca5327b908 Make strtobool even more robust
If strtobool fails, return False always.
2020-03-09 09:30:16 -04:00