parallelvirtualcluster
/
pvc
1
0
Fork 0
Code Issues 11 Pull Requests Projects Releases 111 Wiki Activity
3,385 Commits 1 Branch 113 Tags 28 MiB
Commit Graph

506 Commits

Author SHA1 Message Date
Joshua Boniface 102c3c3106 Port all Celery worker functions to discrete pkg 
Moves all tasks run by the Celery worker into a discrete package/module
for easier installation. Also adjusts several parameters throughout to
accomplish this.
 2023-11-30 02:24:54 -05:00
Joshua Boniface 0c0fb65c62 Rework Flask API to route Celery tasks manually 
Avoids needing to define any of these tasks here; they can all be
defined in the pvcworkerd code.
 2023-11-30 00:40:09 -05:00
Joshua Boniface 03a738f878 Move config parser into daemon_lib 
And reformat/add config values for API.
 2023-11-30 00:05:37 -05:00
Joshua Boniface 647cba3cf5 Expand startup width for new daemon name 2023-11-29 21:21:51 -05:00
Joshua Boniface c8f4cbb39e Fix node entry keys 2023-11-27 13:24:01 -05:00
Joshua Boniface 786fae7769 Improve logo output 2023-11-27 13:01:43 -05:00
Joshua Boniface 17f81e8296 Refactor pvcapid to use new configuration 2023-11-27 12:49:26 -05:00
Joshua Boniface dab7396196 Move to unified pvc.conf configuration file 2023-11-26 14:16:21 -05:00
Joshua Boniface 460a2dd09f Bump version to 0.9.82 2023-11-25 15:38:50 -05:00
Joshua Boniface 24cabd3b99 Fix missing result_backend on Debian 10/11 
For whatever reason, a Celery worker on <5.2.x was not picking these up.
Move them back to the root of the module so they are properly picked up
on these older versions but still prevents calling the routing functions
during an API doc generation.
 2023-11-25 15:35:25 -05:00
Joshua Boniface 3e001b08b6 Bump version to 0.9.81 2023-11-17 01:29:41 -05:00
Joshua Boniface b66cfb07d8 Isolate cluster-dependent Celery startup 
Avoids calling unworkable functions when generating API docs etc. by
isolating them into a Celery startup function called by Daemon.py.

Also update to Celery 4+ settings format.
 2023-11-16 20:32:29 -05:00
Joshua Boniface 9885914abd Remove stray periods from messages 2023-11-16 19:56:24 -05:00
Joshua Boniface e8da3714c0 Convert benchmark to use new Celery step structure 2023-11-16 19:36:23 -05:00
Joshua Boniface 4d23d0419c Fix total stage count 2023-11-16 18:41:43 -05:00
Joshua Boniface c1c22c81e7 Ensure script cleanup is done in chroot 2023-11-16 18:27:23 -05:00
Joshua Boniface 712a50ca27 Avoid use of fail here 
It causes a reraise with a bunch of extra entries that we don't need.
 2023-11-16 18:22:59 -05:00
Joshua Boniface 815041ff20 Fix bugs when main installs fail 2023-11-16 18:20:26 -05:00
Joshua Boniface 823ce8cbf2 Remove duplicate cleanups 2023-11-16 18:19:05 -05:00
Joshua Boniface fca02238d7 Adjust starting text 2023-11-16 18:06:31 -05:00
Joshua Boniface 73a4795967 Avoid fail during yields 
This just causes a double-exception, so don't do it.
 2023-11-16 17:22:53 -05:00
Joshua Boniface 2a637c62e8 Port provisioner scripts to updated framework 
Updates all the example provisioner scripts to use the new functions
exposed by the VMBuilder class as an illustration of how best to use
them.

Also adds a wrapper fail() handler to ensure the cleanup of the script,
as well as the global cleanup, are run on an exception.
 2023-11-16 17:04:46 -05:00
Joshua Boniface 618a1c1c10 Add helper functions to VMBuilder instances 2023-11-16 16:17:17 -05:00
Joshua Boniface f50f170d4e Convert vmbuilder to use new Celery step structure 2023-11-16 16:08:49 -05:00
Joshua Boniface 9ab505ec98 Return and show task_name 2023-11-16 14:50:02 -05:00
Joshua Boniface 0cb81f96e6 Use custom task IDs for Celery tasks 
Full UUIDs were obnoxiously long, so switch to using just the first
8-character section of a UUID instead. Keeps the list nice and short,
makes them easier to copy, and is just generally nicer.

Could this cause uniqueness problems? Perhaps, but I don't see that
happening nearly frequently enough to matter.
 2023-11-16 13:22:14 -05:00
Joshua Boniface 3651885954 Add --events to workers 2023-11-16 12:35:54 -05:00
Joshua Boniface d226e9f4e5 Enable extended Celery results 2023-11-16 12:02:57 -05:00
Joshua Boniface fa361a55d9 Explicitly use kwargs in Celery task calls 2023-11-16 11:55:30 -05:00
Joshua Boniface 0d818017e8 Name the celery workers pvcworkerd@<hostname> 2023-11-16 11:43:17 -05:00
Joshua Boniface 262babc63d Use kwargs for all task arguments 
This will help ensure that the CLI frontend can properly parse the args
in a consistent way.
 2023-11-16 10:10:48 -05:00
Joshua Boniface 289049d223 Properly handle a "primary" run_on value 2023-11-16 02:49:29 -05:00
Joshua Boniface 0bec6abe71 Return proper run_on for ported tasks 2023-11-16 02:28:57 -05:00
Joshua Boniface 484e6542c2 Port remaining tasks to new task handler 
Move the create_vm and run_benchmark tasks to use the new Celery
subsystem, handlers, and wait command. Remove the obsolete, dedicated
API endpoints.

Standardize the CLI client and move the repeated handler code into a
separate common function.
 2023-11-16 02:00:23 -05:00
Joshua Boniface aef38639cf Rename pvcapid-worker to pvcworkerd 2023-11-15 20:31:39 -05:00
Joshua Boniface b5e4c52387 Increase worker concurrency to 3 2023-11-10 00:39:42 -05:00
Joshua Boniface ce17c60a20 Port OSD on-node tasks to Celery worker system 
Adds Celery versions of the osd_add, osd_replace, osd_refresh,
osd_remove, and osd_db_vg_add functions.
 2023-11-09 23:28:08 -05:00
Joshua Boniface 89681d54b9 Port VM on-node tasks to Celery worker system 
Adds Celery versions of the flush_locks, device_attach, and
device_detach functions.
 2023-11-06 20:40:46 -05:00
Joshua Boniface 3dc1f57de2 Revert "Switch to ZK+PG over Redis for Celery queue" 
This reverts commit 54215bab6c.
 2023-11-05 17:10:46 -05:00
Joshua Boniface af8a8d969e Ensure queues are set up for non-coordinator nodes 
Allows a runner to operate on every possible node, not just
coordinators, as OSDs or other things could be on any node.

Also add more comments.
 2023-11-04 15:05:07 -04:00
Joshua Boniface a6caac1b78 Add Celery queue routing for tasks 
By default, tasks will continue to run as they did, on the primary
coordinator's task runner. However this opens the possibility for
defining more tasks that will run on other nodes or coordinators.
 2023-11-04 14:29:59 -04:00
Joshua Boniface ab629f6b51 Use per-host hostname and queues in worker 
Opens up the ability to direct tasks to specific workers.
 2023-11-04 13:02:30 -04:00
Joshua Boniface 54215bab6c Switch to ZK+PG over Redis for Celery queue 
Redis did not provide a distributed solution for the worker, which
precluded several important planned functions. So instead, move to using
Zookeeper + PostgreSQL as the broker and result backend respectively.

Should be a seamless drop-in change but for future uses requires the
database host to be the primary coordinator IP rather than localhost, so
that writes can occur to the database from non-primary hosts.
 2023-11-04 12:46:34 -04:00
Joshua Boniface 64e37ae963 Update OSD replacement functionality 
1. Simplify this by leveraging the existing remove_osd/add_osd
functions, since its task was functionally identical to those two in
sequential order.
2. Add support for split OSDs within the command (replacing all OSDs on
the block device(s) as required).
3. Add additional configurability and flexibility around the old device,
weight, and external DB LVs.
 2023-11-03 01:45:49 -04:00
Joshua Boniface 980ea6a9e9 Adjust handling of ext_db and _count options 
Avoid the use of superfluous flag options, default them to none, and add
support for fixed-size DB LVs.
 2023-11-02 13:29:47 -04:00
Joshua Boniface 526a5f4a74 Add support for split OSD adds 
Allows creating multiple OSDs on a single (NVMe) block device,
leveraging the "ceph-volume lvm batch" command. Replaces the previous
method of creating OSDs.

Also adds a new ZK item for each OSD indicating if it is split or not.
 2023-11-01 21:31:35 -04:00
Joshua Boniface 5b4dd61754 Bump version to 0.9.80 2023-10-27 09:56:31 -04:00
Joshua Boniface 221af3f241 Bump version to 0.9.79 2023-10-24 02:10:24 -04:00
Joshua Boniface c87736eb0a Use consistent path name and format 2023-10-24 01:20:44 -04:00
Joshua Boniface 63d0a85e29 Add backup deletion command 2023-10-24 01:18:27 -04:00
Joshua Boniface 55ca131c2c Handle snapshots on restore and provide options 
Also rename the retain option to remove superfluous plural.
 2023-10-24 00:25:06 -04:00
Joshua Boniface 8d256a1737 Complete VM restore functionality 2023-10-23 22:23:17 -04:00
Joshua Boniface 4fc9b15652 Fix bad function name 2023-10-17 10:56:32 -04:00
Joshua Boniface b997c6f31e Add support for full VM backups 
Adds support for exporting full VM backups, including configuration,
metainfo, and RBD disk images, with incremental support.
 2023-10-17 10:15:06 -04:00
Joshua Boniface 522da3fd95 Adjust wording for volume create too 2023-10-03 09:42:23 -04:00
Joshua Boniface 3a1bf0724e Mention file_size as bytes 2023-10-03 09:39:19 -04:00
Joshua Boniface c6c44bf775 Bump version to 0.9.78 2023-09-30 12:57:55 -04:00
Joshua Boniface bbb940da65 Remove spurious comments 2023-09-30 12:37:58 -04:00
Joshua Boniface 35e27f79ef Fix uploading of non-raw image files 
Adds a new API query parameter to define the file size, which is then
used for the temporary image. This is required for, at least VMDK, files
to work properly in qemu-img convert.
 2023-09-29 16:19:22 -04:00
Joshua Boniface 7c0f12750e Bump version to 0.9.77 2023-09-19 11:05:55 -04:00
Joshua Boniface 51e78480fa Bump version to 0.9.76 2023-09-18 10:15:52 -04:00
Joshua Boniface f46bfc962f Bump version to 0.9.75 2023-09-16 23:06:38 -04:00
Joshua Boniface 1a906b589e Bump version to 0.9.74 2023-09-16 00:18:13 -04:00
Joshua Boniface 311bb69785 Format based on updated Black 2023-09-12 16:41:02 -04:00
Joshua Boniface a8115cafd1 Bump version to 0.9.73 2023-09-02 02:16:19 -04:00
Joshua Boniface fdda47e8a2 Bump version to 0.9.72 2023-09-01 16:34:45 -04:00
Joshua Boniface bb2aac145d Bump version to 0.9.71 2023-09-01 00:36:38 -04:00
Joshua Boniface ec6d3351b2 [Bookworm] Correct Celery invocation 2023-09-01 00:30:01 -04:00
Joshua Boniface 22faaa9bbc [Bookworm] Correct Flask DB migration process 
Move the old manage script to _legacy, and add a new _flask version with
modern Flask tooling. Decide which one to call via pvc-api-db-migrate
using /etc/debian_version call.
 2023-09-01 00:29:17 -04:00
Joshua Boniface 6c407d54c3 Bump version to 0.9.70 2023-08-31 14:15:54 -04:00
Joshua Boniface 9ba7aa5b08 [Bookworm] Remove obsolete package 2023-08-31 14:13:05 -04:00
Joshua Boniface 83b8ce7b62 Bump version to 0.9.69 (nice) 2023-08-29 22:02:13 -04:00
Joshua Boniface 83d475bd15 Bump version to 0.9.68 2023-08-27 20:59:23 -04:00
Joshua Boniface 705ec802a3 Bump version to 0.9.67 2023-08-27 14:47:20 -04:00
Joshua Boniface 0b90f37518 Bump version to 0.9.66 2023-08-27 11:41:22 -04:00
Joshua Boniface 1e083d7652 Bump version to 0.9.65 2023-08-23 01:56:57 -04:00
Joshua Boniface 075dbe7cc9 Bump version to 0.9.64 2023-08-18 12:34:27 -04:00
Joshua Boniface e298d10561 Ensure ACPI is included in Deb VMs 2023-08-17 11:16:08 -04:00
Joshua Boniface fc8cf9ed44 Ensure consistency in variable names and fix bug 2023-08-17 11:09:51 -04:00
Joshua Boniface 3a90fda109 Bump version to 0.9.63 2023-04-28 14:47:04 -04:00
Joshua Boniface 45ad3b9a17 Bump version to 0.9.62 2023-02-22 18:13:45 -05:00
Joshua Boniface e773211293 Add PVC version to cluster status output 2023-02-22 16:09:24 -05:00
Joshua Boniface 96defebd0b Add last item to swagger doc 2023-02-22 00:25:27 -05:00
Joshua Boniface e9aa545e9b Update API specification 2023-02-22 00:06:52 -05:00
Joshua Boniface aeb238f43c Bump version to 0.9.61 2023-02-08 10:08:05 -05:00
Joshua Boniface e945fd8590 Remove bad casting to int in string compare 2023-01-01 13:55:10 -05:00
Joshua Boniface a49510ecc8 Bump version to 0.9.60 2022-12-06 15:42:55 -05:00
Joshua Boniface 6d7730ab52 Disable RBD caching by default 
Results in a massive (~2x) performance boost for random block I/O inside
VMs, and thus a worthwhile default change.
 2022-12-05 17:56:59 -05:00
Joshua Boniface 8135426973 Fix bad ref in example scripts 2022-11-18 12:54:28 -05:00
Joshua Boniface 20d436a745 Update description 2022-11-16 22:48:40 -05:00
Joshua Boniface 28f6819726 Fix up remaining bugs in Rinse test script 2022-11-16 13:32:24 -05:00
Joshua Boniface 35c07f0384 Ensure transient dirs are cleaned up 2022-11-16 13:01:15 -05:00
Joshua Boniface 6127387be4 Ensure swap is skipped during cleanup too 2022-11-16 12:52:24 -05:00
Joshua Boniface 343d66875b Skip swap volumes during mounting 2022-11-16 12:42:28 -05:00
Joshua Boniface 92feeefd26 Bump version to 0.9.59 2022-11-15 15:50:15 -05:00
Joshua Boniface 38d63d9837 Flip behaviour of memory selectors 
It didn't make any sense to me for mem(prov) to be the default selector,
since this has too many caveats versus mem(free). Switch to using
mem(free) as the default (i.e. "mem") and make memprov the alternative.
 2022-11-15 15:45:59 -05:00
Joshua Boniface 095bcb2373 Bump version to 0.9.58 2022-11-07 12:27:48 -05:00
Joshua Boniface 91e450f399 Remove extra lower() call where not needed 2022-11-07 12:26:50 -05:00
Joshua Boniface 79eb994a5e Ensure equality of none and None for selector 2022-11-07 11:59:53 -05:00
Joshua Boniface d65f512897 Bump version to 0.9.57 2022-11-06 01:39:50 -04:00
Joshua Boniface ea7a4b2b85 Make benchmarker function as a module 
1. Move the test_matrix, volume name, and size to module-level variables
so they can be accessed externally if this is imported.
2. Separate the volume creation and volume cleanup into functions.
3. Separate the individual benchmark runs into a function.

This should enable easier calling of the various subcomponents
externally, e.g. for external benchmark scripts.
 2022-11-03 21:33:32 -04:00
Joshua Boniface c3bc55eff8 Bump version to 0.9.56 2022-10-27 14:21:04 -04:00
Joshua Boniface 666e02fbfd Ensure None filesystem is valid 2022-10-21 15:13:52 -04:00
Joshua Boniface 46dde055c4 Move /dev umount to cleanup step 2022-10-21 14:47:48 -04:00
Joshua Boniface ef437c3dbf Adjust help message text 2022-10-21 14:22:15 -04:00
Joshua Boniface bd2208e8f6 Add ova as valid name in addition to default_ova 2022-10-21 14:13:40 -04:00
Joshua Boniface 62d5ff11df Fix console config and domain argument 2022-10-21 14:04:17 -04:00
Joshua Boniface 0019881cfa Add cloud-init configuration to debootstrap script 
Prevents errors trying to find the cloud-init metadata source.
 2022-10-21 14:03:34 -04:00
Joshua Boniface d46133802b Add pfsense example provisioner script 2022-10-21 13:35:48 -04:00
Joshua Boniface fcadde057e Include /proc in chroot mounts 2022-10-20 15:00:10 -04:00
Joshua Boniface 2608f38d64 Properly handle missing source_volume from OVAs 2022-10-19 13:18:12 -04:00
Joshua Boniface 89f05ced3f Add missing ceph import 2022-10-19 13:10:40 -04:00
Joshua Boniface 729481126c Move conversion to install() step 
Seems more clear to me than doing it in prepare()
 2022-10-19 13:09:29 -04:00
Joshua Boniface 41eccb9c7d Add missing imports 2022-10-19 13:07:34 -04:00
Joshua Boniface e550e39a5a Add output messages during OVA prepare 2022-10-19 12:58:11 -04:00
Joshua Boniface dff156b2b0 Fix bad comparison 2022-10-19 12:46:15 -04:00
Joshua Boniface e26ff8a975 Fix missing f-string marker 2022-10-15 16:26:47 -04:00
Joshua Boniface a34d64a71b Add rinse example configuration 
Provisions Rocky Linux 8 and 9 systems, and potentially older
CentOS/Fedora/Scientific Linux/SuSE systems. Depends on a custom build
of rinse (3.7.1) with Rocky 9 support.
 2022-10-07 19:55:56 -04:00
Joshua Boniface 71297e0179 Add host-model to CPU config in VMs 2022-10-07 09:36:22 -04:00
Joshua Boniface 45c9909428 Add output message to debootstrap install 2022-10-07 02:27:20 -04:00
Joshua Boniface 7268592c87 Fix braces in fstring example 2022-10-06 15:57:31 -04:00
Joshua Boniface 726d0a562b Update copyright header year 2022-10-06 11:55:27 -04:00
Joshua Boniface 39e1fc50ed Fix titles 2022-10-06 11:54:36 -04:00
Joshua Boniface 7a3870fc44 Add OVA script support 
1. Ensure that system_template and script are not nullable in the DB.
2. Ensure that the CLI and API enforce the above and clean up CLI
arguments for profile add.
3. Ensure that, before uploading OVAs, a 'default_ova' provisioning
script is present.
4. Use the 'default_ova' script for new OVA uploads.
5. Ensure that OVA details are properly added to the vm_data dict in the
provisioner vmbuilder.
 2022-10-06 10:48:12 -04:00
Joshua Boniface bffab7a5a1 Reverse numbering of example scripts 2022-10-06 10:14:37 -04:00
Joshua Boniface 6cbaeb5dc8 Complete OVA provisioning script 2022-10-06 10:14:04 -04:00
Joshua Boniface 58ce133c8d Remove lingering OVA references 2022-10-06 00:13:36 -04:00
Joshua Boniface 3a5d8c61da Reorganize and add more comments to examples 2022-10-05 23:35:53 -04:00
Joshua Boniface 1e0b502250 Fix remaining bugs in example scripts 2022-10-05 22:37:11 -04:00
Joshua Boniface fe17d28385 Ensure inner cleanup and end message response 2022-10-05 22:36:42 -04:00
Joshua Boniface 8aaac33056 Fix bad variable reference 2022-10-05 17:43:23 -04:00
Joshua Boniface cc7952c232 Add additional import for config 2022-10-05 17:41:37 -04:00
Joshua Boniface 16915ed507 Add better exception handling with ctxtmgrs 2022-10-05 17:35:05 -04:00
Joshua Boniface 2c624ceb2c Add additional missing import in examples 2022-10-05 17:29:34 -04:00
Joshua Boniface da85480488 Improve error messages 2022-10-05 17:26:09 -04:00
Joshua Boniface 47b0704555 Fix bad ref in examples 2022-10-05 17:25:56 -04:00
Joshua Boniface 7c49967586 Add proper imports to examples 2022-10-05 17:22:04 -04:00
Joshua Boniface e3f96ac87e Better handle cleanups and fix chroot bug 2022-10-05 17:21:30 -04:00
Joshua Boniface 4df70cf086 Implement new provisioner setup 2022-10-05 16:03:05 -04:00
Joshua Boniface f1df1cfe93 Bump version to 0.9.55 2022-10-04 13:21:40 -04:00
Joshua Boniface 239c392892 Bump version to 0.9.54 2022-08-23 11:01:05 -04:00
Joshua Boniface 172d0a86e4 Use proper SSLContext and enable TLSv1 
It's bad, but sometimes you need to access the API from a very old
software version. So just enable it for now and clean it up later.
 2022-08-23 10:58:47 -04:00
Joshua Boniface 9b499b9f48 Bump version to 0.9.53 2022-08-12 17:47:11 -04:00
Joshua Boniface 2a21d48128 Bump version to 0.9.52 2022-08-12 11:09:25 -04:00
Joshua Boniface 645b525ad7 Bump version to 0.9.51 2022-07-25 23:25:41 -04:00
Joshua Boniface 932b3c55a3 Bump version to 0.9.50 2022-07-06 16:01:14 -04:00
Joshua Boniface d8d3feee22 Add selector help and adjust flag name 
1. Add documentation on the node selector flags. In the API, reference
the daemon configuration manual which now includes details in this
section; in the CLI, provide the help in "pvc vm define" in detail and
then reference that command's help in the other commands that use this
field.

2. Ensure the naming is consistent in the CLI, using the flag name
"--node-selector" everywhere (was "--selector" for "pvc vm" commands and
"--node-selector" for "pvc provisioner" commands).
 2022-06-10 02:42:06 -04:00
Joshua Boniface b1357cafdb Add memfree to selector and use proper defaults 2022-06-10 02:03:12 -04:00
Joshua Boniface 51ad2058ed Bump version to 0.9.49 2022-05-06 15:49:39 -04:00
Joshua Boniface c401a1f655 Use consistent language for primary mode 
I didn't call it "router" anywhere else, but the state in the list is
called "coordinator" so, call it "coordinator mode".
 2022-05-06 15:40:52 -04:00