Joshua Boniface
0bec6abe71
Return proper run_on for ported tasks
2023-11-16 02:28:57 -05:00
Joshua Boniface
484e6542c2
Port remaining tasks to new task handler
...
Move the create_vm and run_benchmark tasks to use the new Celery
subsystem, handlers, and wait command. Remove the obsolete, dedicated
API endpoints.
Standardize the CLI client and move the repeated handler code into a
separate common function.
2023-11-16 02:00:23 -05:00
Joshua Boniface
aef38639cf
Rename pvcapid-worker to pvcworkerd
2023-11-15 20:31:39 -05:00
Joshua Boniface
b5e4c52387
Increase worker concurrency to 3
2023-11-10 00:39:42 -05:00
Joshua Boniface
ce17c60a20
Port OSD on-node tasks to Celery worker system
...
Adds Celery versions of the osd_add, osd_replace, osd_refresh,
osd_remove, and osd_db_vg_add functions.
2023-11-09 23:28:08 -05:00
Joshua Boniface
89681d54b9
Port VM on-node tasks to Celery worker system
...
Adds Celery versions of the flush_locks, device_attach, and
device_detach functions.
2023-11-06 20:40:46 -05:00
Joshua Boniface
3dc1f57de2
Revert "Switch to ZK+PG over Redis for Celery queue"
...
This reverts commit 54215bab6c
2023-11-05 17:10:46 -05:00
Joshua Boniface
af8a8d969e
Ensure queues are set up for non-coordinator nodes
...
Allows a runner to operate on every possible node, not just
coordinators, as OSDs or other things could be on any node.
Also add more comments.
2023-11-04 15:05:07 -04:00
Joshua Boniface
a6caac1b78
Add Celery queue routing for tasks
...
By default, tasks will continue to run as they did, on the primary
coordinator's task runner. However this opens the possibility for
defining more tasks that will run on other nodes or coordinators.
2023-11-04 14:29:59 -04:00
Joshua Boniface
ab629f6b51
Use per-host hostname and queues in worker
...
Opens up the ability to direct tasks to specific workers.
2023-11-04 13:02:30 -04:00
Joshua Boniface
54215bab6c
Switch to ZK+PG over Redis for Celery queue
...
Redis did not provide a distributed solution for the worker, which
precluded several important planned functions. So instead, move to using
Zookeeper + PostgreSQL as the broker and result backend respectively.
Should be a seamless drop-in change but for future uses requires the
database host to be the primary coordinator IP rather than localhost, so
that writes can occur to the database from non-primary hosts.
2023-11-04 12:46:34 -04:00
Joshua Boniface
64e37ae963
Update OSD replacement functionality
...
1. Simplify this by leveraging the existing remove_osd/add_osd
functions, since its task was functionally identical to those two in
sequential order.
2. Add support for split OSDs within the command (replacing all OSDs on
the block device(s) as required).
3. Add additional configurability and flexibility around the old device,
weight, and external DB LVs.
2023-11-03 01:45:49 -04:00
Joshua Boniface
980ea6a9e9
Adjust handling of ext_db and _count options
...
Avoid the use of superfluous flag options, default them to none, and add
support for fixed-size DB LVs.
2023-11-02 13:29:47 -04:00
Joshua Boniface
526a5f4a74
Add support for split OSD adds
...
Allows creating multiple OSDs on a single (NVMe) block device,
leveraging the "ceph-volume lvm batch" command. Replaces the previous
method of creating OSDs.
Also adds a new ZK item for each OSD indicating if it is split or not.
2023-11-01 21:31:35 -04:00
Joshua Boniface
5b4dd61754
Bump version to 0.9.80
2023-10-27 09:56:31 -04:00
Joshua Boniface
221af3f241
Bump version to 0.9.79
2023-10-24 02:10:24 -04:00
Joshua Boniface
c87736eb0a
Use consistent path name and format
2023-10-24 01:20:44 -04:00
Joshua Boniface
63d0a85e29
Add backup deletion command
2023-10-24 01:18:27 -04:00
Joshua Boniface
55ca131c2c
Handle snapshots on restore and provide options
...
Also rename the retain option to remove superfluous plural.
2023-10-24 00:25:06 -04:00
Joshua Boniface
8d256a1737
Complete VM restore functionality
2023-10-23 22:23:17 -04:00
Joshua Boniface
4fc9b15652
Fix bad function name
2023-10-17 10:56:32 -04:00
Joshua Boniface
b997c6f31e
Add support for full VM backups
...
Adds support for exporting full VM backups, including configuration,
metainfo, and RBD disk images, with incremental support.
2023-10-17 10:15:06 -04:00
Joshua Boniface
522da3fd95
Adjust wording for volume create too
2023-10-03 09:42:23 -04:00
Joshua Boniface
3a1bf0724e
Mention file_size as bytes
2023-10-03 09:39:19 -04:00
Joshua Boniface
c6c44bf775
Bump version to 0.9.78
2023-09-30 12:57:55 -04:00
Joshua Boniface
bbb940da65
Remove spurious comments
2023-09-30 12:37:58 -04:00
Joshua Boniface
35e27f79ef
Fix uploading of non-raw image files
...
Adds a new API query parameter to define the file size, which is then
used for the temporary image. This is required for, at least VMDK, files
to work properly in qemu-img convert.
2023-09-29 16:19:22 -04:00
Joshua Boniface
7c0f12750e
Bump version to 0.9.77
2023-09-19 11:05:55 -04:00
Joshua Boniface
51e78480fa
Bump version to 0.9.76
2023-09-18 10:15:52 -04:00
Joshua Boniface
f46bfc962f
Bump version to 0.9.75
2023-09-16 23:06:38 -04:00
Joshua Boniface
1a906b589e
Bump version to 0.9.74
2023-09-16 00:18:13 -04:00
Joshua Boniface
311bb69785
Format based on updated Black
2023-09-12 16:41:02 -04:00
Joshua Boniface
a8115cafd1
Bump version to 0.9.73
2023-09-02 02:16:19 -04:00
Joshua Boniface
fdda47e8a2
Bump version to 0.9.72
2023-09-01 16:34:45 -04:00
Joshua Boniface
bb2aac145d
Bump version to 0.9.71
2023-09-01 00:36:38 -04:00
Joshua Boniface
ec6d3351b2
[Bookworm] Correct Celery invocation
2023-09-01 00:30:01 -04:00
Joshua Boniface
22faaa9bbc
[Bookworm] Correct Flask DB migration process
...
Move the old manage script to _legacy, and add a new _flask version with
modern Flask tooling. Decide which one to call via pvc-api-db-migrate
using /etc/debian_version call.
2023-09-01 00:29:17 -04:00
Joshua Boniface
6c407d54c3
Bump version to 0.9.70
2023-08-31 14:15:54 -04:00
Joshua Boniface
9ba7aa5b08
[Bookworm] Remove obsolete package
2023-08-31 14:13:05 -04:00
Joshua Boniface
83b8ce7b62
Bump version to 0.9.69 (nice)
2023-08-29 22:02:13 -04:00
Joshua Boniface
83d475bd15
Bump version to 0.9.68
2023-08-27 20:59:23 -04:00
Joshua Boniface
705ec802a3
Bump version to 0.9.67
2023-08-27 14:47:20 -04:00
Joshua Boniface
0b90f37518
Bump version to 0.9.66
2023-08-27 11:41:22 -04:00
Joshua Boniface
1e083d7652
Bump version to 0.9.65
2023-08-23 01:56:57 -04:00
Joshua Boniface
075dbe7cc9
Bump version to 0.9.64
2023-08-18 12:34:27 -04:00
Joshua Boniface
e298d10561
Ensure ACPI is included in Deb VMs
2023-08-17 11:16:08 -04:00
Joshua Boniface
fc8cf9ed44
Ensure consistency in variable names and fix bug
2023-08-17 11:09:51 -04:00
Joshua Boniface
3a90fda109
Bump version to 0.9.63
2023-04-28 14:47:04 -04:00
Joshua Boniface
45ad3b9a17
Bump version to 0.9.62
2023-02-22 18:13:45 -05:00
Joshua Boniface
e773211293
Add PVC version to cluster status output
2023-02-22 16:09:24 -05:00
Joshua Boniface
96defebd0b
Add last item to swagger doc
2023-02-22 00:25:27 -05:00
Joshua Boniface
e9aa545e9b
Update API specification
2023-02-22 00:06:52 -05:00
Joshua Boniface
aeb238f43c
Bump version to 0.9.61
2023-02-08 10:08:05 -05:00
Joshua Boniface
e945fd8590
Remove bad casting to int in string compare
2023-01-01 13:55:10 -05:00
Joshua Boniface
a49510ecc8
Bump version to 0.9.60
2022-12-06 15:42:55 -05:00
Joshua Boniface
6d7730ab52
Disable RBD caching by default
...
Results in a massive (~2x) performance boost for random block I/O inside
VMs, and thus a worthwhile default change.
2022-12-05 17:56:59 -05:00
Joshua Boniface
8135426973
Fix bad ref in example scripts
2022-11-18 12:54:28 -05:00
Joshua Boniface
20d436a745
Update description
2022-11-16 22:48:40 -05:00
Joshua Boniface
28f6819726
Fix up remaining bugs in Rinse test script
2022-11-16 13:32:24 -05:00
Joshua Boniface
35c07f0384
Ensure transient dirs are cleaned up
2022-11-16 13:01:15 -05:00
Joshua Boniface
6127387be4
Ensure swap is skipped during cleanup too
2022-11-16 12:52:24 -05:00
Joshua Boniface
343d66875b
Skip swap volumes during mounting
2022-11-16 12:42:28 -05:00
Joshua Boniface
92feeefd26
Bump version to 0.9.59
2022-11-15 15:50:15 -05:00
Joshua Boniface
38d63d9837
Flip behaviour of memory selectors
...
It didn't make any sense to me for mem(prov) to be the default selector,
since this has too many caveats versus mem(free). Switch to using
mem(free) as the default (i.e. "mem") and make memprov the alternative.
2022-11-15 15:45:59 -05:00
Joshua Boniface
095bcb2373
Bump version to 0.9.58
2022-11-07 12:27:48 -05:00
Joshua Boniface
91e450f399
Remove extra lower() call where not needed
2022-11-07 12:26:50 -05:00
Joshua Boniface
79eb994a5e
Ensure equality of none and None for selector
2022-11-07 11:59:53 -05:00
Joshua Boniface
d65f512897
Bump version to 0.9.57
2022-11-06 01:39:50 -04:00
Joshua Boniface
ea7a4b2b85
Make benchmarker function as a module
...
1. Move the test_matrix, volume name, and size to module-level variables
so they can be accessed externally if this is imported.
2. Separate the volume creation and volume cleanup into functions.
3. Separate the individual benchmark runs into a function.
This should enable easier calling of the various subcomponents
externally, e.g. for external benchmark scripts.
2022-11-03 21:33:32 -04:00
Joshua Boniface
c3bc55eff8
Bump version to 0.9.56
2022-10-27 14:21:04 -04:00
Joshua Boniface
666e02fbfd
Ensure None filesystem is valid
2022-10-21 15:13:52 -04:00
Joshua Boniface
46dde055c4
Move /dev umount to cleanup step
2022-10-21 14:47:48 -04:00
Joshua Boniface
ef437c3dbf
Adjust help message text
2022-10-21 14:22:15 -04:00
Joshua Boniface
bd2208e8f6
Add ova as valid name in addition to default_ova
2022-10-21 14:13:40 -04:00
Joshua Boniface
62d5ff11df
Fix console config and domain argument
2022-10-21 14:04:17 -04:00
Joshua Boniface
0019881cfa
Add cloud-init configuration to debootstrap script
...
Prevents errors trying to find the cloud-init metadata source.
2022-10-21 14:03:34 -04:00
Joshua Boniface
d46133802b
Add pfsense example provisioner script
2022-10-21 13:35:48 -04:00
Joshua Boniface
fcadde057e
Include /proc in chroot mounts
2022-10-20 15:00:10 -04:00
Joshua Boniface
2608f38d64
Properly handle missing source_volume from OVAs
2022-10-19 13:18:12 -04:00
Joshua Boniface
89f05ced3f
Add missing ceph import
2022-10-19 13:10:40 -04:00
Joshua Boniface
729481126c
Move conversion to install() step
...
Seems more clear to me than doing it in prepare()
2022-10-19 13:09:29 -04:00
Joshua Boniface
41eccb9c7d
Add missing imports
2022-10-19 13:07:34 -04:00
Joshua Boniface
e550e39a5a
Add output messages during OVA prepare
2022-10-19 12:58:11 -04:00
Joshua Boniface
dff156b2b0
Fix bad comparison
2022-10-19 12:46:15 -04:00
Joshua Boniface
e26ff8a975
Fix missing f-string marker
2022-10-15 16:26:47 -04:00
Joshua Boniface
a34d64a71b
Add rinse example configuration
...
Provisions Rocky Linux 8 and 9 systems, and potentially older
CentOS/Fedora/Scientific Linux/SuSE systems. Depends on a custom build
of rinse (3.7.1) with Rocky 9 support.
2022-10-07 19:55:56 -04:00
Joshua Boniface
71297e0179
Add host-model to CPU config in VMs
2022-10-07 09:36:22 -04:00
Joshua Boniface
45c9909428
Add output message to debootstrap install
2022-10-07 02:27:20 -04:00
Joshua Boniface
7268592c87
Fix braces in fstring example
2022-10-06 15:57:31 -04:00
Joshua Boniface
726d0a562b
Update copyright header year
2022-10-06 11:55:27 -04:00
Joshua Boniface
39e1fc50ed
Fix titles
2022-10-06 11:54:36 -04:00
Joshua Boniface
7a3870fc44
Add OVA script support
...
1. Ensure that system_template and script are not nullable in the DB.
2. Ensure that the CLI and API enforce the above and clean up CLI
arguments for profile add.
3. Ensure that, before uploading OVAs, a 'default_ova' provisioning
script is present.
4. Use the 'default_ova' script for new OVA uploads.
5. Ensure that OVA details are properly added to the vm_data dict in the
provisioner vmbuilder.
2022-10-06 10:48:12 -04:00
Joshua Boniface
bffab7a5a1
Reverse numbering of example scripts
2022-10-06 10:14:37 -04:00
Joshua Boniface
6cbaeb5dc8
Complete OVA provisioning script
2022-10-06 10:14:04 -04:00
Joshua Boniface
58ce133c8d
Remove lingering OVA references
2022-10-06 00:13:36 -04:00
Joshua Boniface
3a5d8c61da
Reorganize and add more comments to examples
2022-10-05 23:35:53 -04:00
Joshua Boniface
1e0b502250
Fix remaining bugs in example scripts
2022-10-05 22:37:11 -04:00
Joshua Boniface
fe17d28385
Ensure inner cleanup and end message response
2022-10-05 22:36:42 -04:00
Joshua Boniface
8aaac33056
Fix bad variable reference
2022-10-05 17:43:23 -04:00
Joshua Boniface
cc7952c232
Add additional import for config
2022-10-05 17:41:37 -04:00
Joshua Boniface