Commit Graph

3354 Commits

Author SHA1 Message Date
Joshua Boniface 6ace2ebf6a Set expected PVC version for mirroring 2024-10-15 11:31:50 -04:00
Joshua Boniface 962fba7621 Bump up startup waits slightly
Ensures there's more time for daemons (specifically Zookeeper) to start
up and synchronize between nodes.
2024-10-15 11:10:23 -04:00
Joshua Boniface 49bf51da38 Fix indentation of previous fix 2024-10-15 10:57:33 -04:00
Joshua Boniface 1293e8ae7e Fix bugs in lock freeing function
1. The destination state on an error was invalid; should be "stop".

2. If a lock was listed but removing it fails (because it was already
cleared somehow, this would error. In turn this would cause the VM to
not migrate and be left in an undefined state. Fix that when unlocking
is forced.
2024-10-15 10:43:52 -04:00
Joshua Boniface ae2cf8a070 Add some time for Zookeeper to synchronize 2024-10-15 10:43:44 -04:00
Joshua Boniface ab5bd3c57d Fix handling of invalid nets in list
Ensure we add the difference in length between the visual output and the
ANSI-coded output to avoid the format handler mishandling the length.
2024-10-14 12:51:02 -04:00
Joshua Boniface 35153cd6b6 Fix path handling for zkhandler
Using full paths broke the local schema generator, so convert these to
proper class instance methods and use them along with a new default +
settable override.
2024-10-11 16:03:40 -04:00
Joshua Boniface 7f7047dd52 Add one more instance of mirror as purple 2024-10-11 14:44:14 -04:00
Joshua Boniface 9a91767405 Add proper return codes to API handlers 2024-10-11 14:43:44 -04:00
Joshua Boniface bcfa6851e1 Use purple for mirror state colour 2024-10-11 10:44:39 -04:00
Joshua Boniface 28b8b3bb44 Use proper response parsing instead of raise_for 2024-10-11 10:32:15 -04:00
Joshua Boniface 02425159ef Update Grafana graphs 2024-10-11 09:47:19 -04:00
Joshua Boniface a6f8500309 Improve fence handling to prevent anomalies
1. Move fence monitoring to its own thread rather than doing the listing
and triggering within the main keepalive thread.
2. Add a global lock key at /config/fence_lock and use this lock key to
prevent multiple nodes from trying to run fences simultaneously.
3. Run the fencing monitor for each node sequentially within the context
of the main fence monitoring thread, to ensure that fences of multiple
nodes happen sequentially rather than in parallel.

All of these should help to prevent any anomalies where one node can try
to fence multiple nodes at once without recourse.
2024-10-10 16:42:57 -04:00
Joshua Boniface ebec1332e9 Return to relative paths for SCHEMA_ROOT_PATH 2024-10-10 16:20:02 -04:00
Joshua Boniface c08c3b2d7d Improve thread timeouts in keepalive
Avoids various parts of the keepalive deadlocking waiting on data that
will never come when various internal processes fail. This should ensure
based on testing that the keepalive will always finish in <5 seconds.
2024-10-10 15:33:47 -04:00
Joshua Boniface 4c0d90b517 Add read lock timeouts to prevent deadlocks 2024-10-10 15:19:05 -04:00
Joshua Boniface 70c588d3a8 Add confirmation option for mirror promote 2024-10-10 01:57:06 -04:00
Joshua Boniface 214e7f835a Properly preserve state on promotion
Ensure if the state is start, stop, or disable, that state is preserved;
if it's anything else, the remote side will be started.
2024-10-10 01:21:05 -04:00
Joshua Boniface 96cebfb42a Handle cross-cluster Ceph storage secrets 2024-10-10 00:47:50 -04:00
Joshua Boniface c4763ac596 Fix invalid responses during promote 2024-10-09 01:14:19 -04:00
Joshua Boniface ea5512e3d8 Only shut down VM if it is running 2024-10-09 01:10:42 -04:00
Joshua Boniface ac00f7c4c8 Fix boolean state of remove_on_source 2024-10-09 01:04:08 -04:00
Joshua Boniface 6d31bf439e Update error text 2024-10-09 01:00:51 -04:00
Joshua Boniface c714093a2e Ensure VM start is forced 2024-10-09 00:58:43 -04:00
Joshua Boniface 04a09b9269 Fix invalid data in state change 2024-10-09 00:55:13 -04:00
Joshua Boniface 3ede0c7d38 Name mirror snapshots like autobackup snapshots 2024-10-09 00:49:22 -04:00
Joshua Boniface ab9390fdb8 Fix another bad stage counting instance 2024-10-09 00:44:20 -04:00
Joshua Boniface 1c83584788 Set correct verbage 2024-10-09 00:38:59 -04:00
Joshua Boniface 7f3ab4e119 Fix stage counting in tasks 2024-10-09 00:37:13 -04:00
Joshua Boniface 16eb09dc22 Fix ordering bug with vm_detail 2024-10-09 00:33:00 -04:00
Joshua Boniface 7ba75adef4 Fix bug if destination is missing 2024-10-09 00:27:42 -04:00
Joshua Boniface a691d26c30 Add check for scheme in destination
Allows handling invalid cluster names properly.
2024-10-09 00:25:13 -04:00
Joshua Boniface 1d90b066bc Add guard rails against manipulating mirrors
Snapshot mirrors should normally be promoted using "mirror promote", and
not started manually. This adds guard rails against that to the "start",
"stop", and "disable" state commands to prevent changing mirror states
without an explicit "--force" option.
2024-10-08 23:51:48 -04:00
Joshua Boniface 3ea7421f09 Implement friendlier VM mirror commands
Adds two helper commands which automate sending and promoting VM
snapshots as "vm mirror" commands.

"vm mirror create" replicates the functionality of "snapshot create" and
"snapshot send", performing both in one single task using an
autogenerated dated snapshot name for automatic cross-cluster
replication.

"vm mirror promote" replicates the functionality of "vm shutdown",
"snapshot create", "snapshot send", "vm start" (remote), and,
optionally, "vm remove", performing in one single task an entire
cross-cluster VM move with or without retaining the copy on the local
cluster (if retained, the local copy becomes a snapshot mirror of the
remote, flipping their statuses).
2024-10-08 23:51:39 -04:00
Joshua Boniface df4d437d31 Update the description of VM define endpoint 2024-10-01 13:30:44 -04:00
Joshua Boniface 8295e2089d Add proper response schema for 202 responses 2024-10-01 13:25:11 -04:00
Joshua Boniface 4ccb570762 Enhance documentation of snapshot send command 2024-09-30 23:54:53 -04:00
Joshua Boniface 235299942a Add volume resize if changed 2024-09-30 20:51:59 -04:00
Joshua Boniface 9aa32134a9 Fix bug in API specification 2024-09-30 20:51:49 -04:00
Joshua Boniface 75eac356d5 Increase send blocksize and add total speed
It's much faster and seems to cause no issues.
2024-09-30 20:11:12 -04:00
Joshua Boniface fb8561cc5d Actually fix incremental sending 2024-09-30 17:00:18 -04:00
Joshua Boniface 5f7aa0b2d6 Improve incremental send speed 2024-09-30 04:15:17 -04:00
Joshua Boniface 7fac7a62cf Clean up debug print statements 2024-09-30 03:51:39 -04:00
Joshua Boniface b19642aa2e Fix bug where snapshot rollback was never called 2024-09-30 03:04:35 -04:00
Joshua Boniface 974e0d6ac2 Shorten progress bars to 20 characters
They were needlessly long and this limited the message size.
2024-09-30 03:04:10 -04:00
Joshua Boniface 7785166a7e Finish working implementation of send/receive
Required some significant refactoring due to issues with the diff send,
but it works.
2024-09-30 02:53:23 -04:00
Joshua Boniface 34f0a2f388 Add mostly complete implementation of VM send 2024-09-29 01:31:13 -04:00
Joshua Boniface 8fa37d21c0 Fix handling of invalid network lengths 2024-09-29 00:39:53 -04:00
Joshua Boniface f462ebbc6b Add VM snapshot send (initial) 2024-09-28 10:49:35 -04:00
Joshua Boniface 0d533f3658 Rework task output bar operation
Allows sending constant updates including changes to the message within
the same task.
2024-09-28 10:48:39 -04:00