parallelvirtualcluster
/
pvc
1
0
Fork 0
Code Issues 11 Pull Requests Projects Releases 111 Wiki Activity
2,907 Commits 1 Branch 113 Tags 28 MiB
Commit Graph

438 Commits

Author SHA1 Message Date
Joshua Boniface 5e43f9bd7c Ensure Patroni failures do not block takeover 2023-08-29 22:00:11 -04:00
Joshua Boniface ed087d83c2 Found cpuload to 2 decimal places 2023-08-29 21:41:44 -04:00
Joshua Boniface 83d475bd15 Bump version to 0.9.68 2023-08-27 20:59:23 -04:00
Joshua Boniface 705ec802a3 Bump version to 0.9.67 2023-08-27 14:47:20 -04:00
Joshua Boniface 0b90f37518 Bump version to 0.9.66 2023-08-27 11:41:22 -04:00
Joshua Boniface 1e083d7652 Bump version to 0.9.65 2023-08-23 01:56:57 -04:00
Joshua Boniface 075dbe7cc9 Bump version to 0.9.64 2023-08-18 12:34:27 -04:00
Joshua Boniface b5f996febd Fix bugs for node flush for stop/shutdown/restart 
Previously VMs in stop/shutdown/restart states wouldn't be properly
handled during a node flush. This fixes the bugs and ensures that the
transient VM states (shutdown/restart) are completed before proceeding,
and then avoids setting a stopped/shutdown VM to shutdown/auotstart.
 2023-08-18 11:25:59 -04:00
Joshua Boniface 3a90fda109 Bump version to 0.9.63 2023-04-28 14:47:04 -04:00
Joshua Boniface 2c3a3cdf52 Use try when watching health value in NodeInstance 2023-03-07 09:53:01 -05:00
Joshua Boniface 7c07fbefff Adjust keepalive health printing and ordering 2023-02-24 11:08:30 -05:00
Joshua Boniface 202dc3ed59 Correct error handling if monitoring plugins fail 2023-02-24 10:19:41 -05:00
Joshua Boniface 45ad3b9a17 Bump version to 0.9.62 2023-02-22 18:13:45 -05:00
Joshua Boniface e45b3108a2 Add health delta change to message output 2023-02-22 15:02:08 -05:00
Joshua Boniface 118237a53b Fix bad string value for message 2023-02-22 15:02:08 -05:00
Joshua Boniface 1093ca6264 Disallow health less than 0 2023-02-15 16:50:24 -05:00
Joshua Boniface f4eef30770 Add JSON health to cluster data 2023-02-15 15:26:57 -05:00
Joshua Boniface 0ecf219910 Run setup during plugin loads 2023-02-15 10:11:38 -05:00
Joshua Boniface 0f4edc54d1 Use percentage in keepalie output 2023-02-15 01:56:02 -05:00
Joshua Boniface 14d29f2986 Adjust text on log message 2023-02-13 22:21:23 -05:00
Joshua Boniface bc88d764b0 Add logging flag for montioring plugin output 2023-02-13 22:04:39 -05:00
Joshua Boniface b07396c39a Fix bugs if plugins fail to load 2023-02-13 21:51:48 -05:00
Joshua Boniface 1ea4800212 Set node health to None when restarting 2023-02-13 15:54:46 -05:00
Joshua Boniface 9c14d84bfc Add node health value and send out API 2023-02-13 15:53:39 -05:00
Joshua Boniface d8f346abdd Move Ceph cluster health reporting to plugin 
Also removes several outputs from the normal keepalive that were
superfluous/static so that the main output fits on one line.
 2023-02-13 13:29:40 -05:00
Joshua Boniface 2ee52e44d3 Move Ceph cluster health reporting to plugin 
Also removes several outputs from the normal keepalive that were
superfluous/static so that the main output fits on one line.
 2023-02-13 12:13:56 -05:00
Joshua Boniface 3c742a827b Initial implementation of monitoring plugin system 2023-02-13 12:06:26 -05:00
Joshua Boniface aeb238f43c Bump version to 0.9.61 2023-02-08 10:08:05 -05:00
Joshua Boniface a49510ecc8 Bump version to 0.9.60 2022-12-06 15:42:55 -05:00
Joshua Boniface 92feeefd26 Bump version to 0.9.59 2022-11-15 15:50:15 -05:00
Joshua Boniface 095bcb2373 Bump version to 0.9.58 2022-11-07 12:27:48 -05:00
Joshua Boniface d65f512897 Bump version to 0.9.57 2022-11-06 01:39:50 -04:00
Joshua Boniface c3bc55eff8 Bump version to 0.9.56 2022-10-27 14:21:04 -04:00
Joshua Boniface 726d0a562b Update copyright header year 2022-10-06 11:55:27 -04:00
Joshua Boniface f1df1cfe93 Bump version to 0.9.55 2022-10-04 13:21:40 -04:00
Joshua Boniface 5942aa50fc Avoid raise/handle deadlocks 
Can cause log flooding in some edge cases and isn't really needed any
longer. Use a proper conditional followed by an actual error handler.
 2022-10-03 14:04:12 -04:00
Joshua Boniface 239c392892 Bump version to 0.9.54 2022-08-23 11:01:05 -04:00
Joshua Boniface 9b499b9f48 Bump version to 0.9.53 2022-08-12 17:47:11 -04:00
Joshua Boniface 2a21d48128 Bump version to 0.9.52 2022-08-12 11:09:25 -04:00
Joshua Boniface 8d0f26ff7a Add additional kb_ values to OSD stats 
Allows for easier parsing later to get e.g. % values and more details on
the used amounts.
 2022-08-11 11:06:36 -04:00
Joshua Boniface 645b525ad7 Bump version to 0.9.51 2022-07-25 23:25:41 -04:00
Joshua Boniface 932b3c55a3 Bump version to 0.9.50 2022-07-06 16:01:14 -04:00
Joshua Boniface 92e2ff7449 Fix bug with space-containing detect strings 2022-07-06 15:58:57 -04:00
Joshua Boniface 51ad2058ed Bump version to 0.9.49 2022-05-06 15:49:39 -04:00
Joshua Boniface 7a40c7a55b Add support for replacing/refreshing OSDs 
Adds commands to both replace an OSD disk, and refresh (reimport) an
existing OSD disk on a new node. This handles the cases where an OSD
disk should be replaced (either due to upgrades or failures) or where a
node is rebuilt in-place and an existing OSD must be re-imported to it.

This should avoid the need to do a full remove/add sequence for either
case.

Also cleans up some aspects of OSD removal that are identical between
methods (e.g. using safe-to-destroy and sleeping after stopping) and
fixes a bug if an OSD does not truly exist when the daemon starts up.
 2022-05-06 15:32:06 -04:00
Joshua Boniface 3801fcc07b Fix bug with initial JSON for stats 2022-05-02 13:28:19 -04:00
Joshua Boniface c741900baf Refactor OSD removal to use new ZK data 
With the OSD LVM information stored in Zookeeper, we can use this to
determine the actual block device to zap rather than relying on runtime
determination and guestimation.
 2022-05-02 12:52:22 -04:00
Joshua Boniface 464f0e0356 Store additional OSD information in ZK 
Ensures that information like the FSIDs and the OSD LVM volume are
stored in Zookeeper at creation time and updated at daemon start time
(to ensure the data is populated at least once, or if the /dev/sdX
path changes).

This will allow safer operation of OSD removals and the potential
implementation of re-activation after node replacements.
 2022-05-02 12:11:39 -04:00
Joshua Boniface cea8832f90 Ensure initial OSD stats is populated 
Values are all invalid but this ensures the client won't error out when
trying to show an OSD that has never checked in yet.
 2022-04-29 16:50:30 -04:00
Joshua Boniface 5807351405 Bump version to 0.9.48 2022-04-29 15:03:52 -04:00
Joshua Boniface d6ca74376a Fix bugs with forced removal 2022-04-29 14:03:07 -04:00
Joshua Boniface 4d698be34b Add OSD removal force option 
Ensures a removal can continue even in situations where some step(s)
might fail, for instance removing an obsolete OSD from a replaced node.
 2022-04-29 11:16:33 -04:00
Joshua Boniface ea709f573f Bump version to 0.9.47 2021-12-28 22:03:08 -05:00
Joshua Boniface 58d57d7037 Bump version to 0.9.46 2021-12-28 15:02:14 -05:00
Joshua Boniface 00d2c67c41 Allow single-node clusters to restart and timeout 
Prevents a daemon from waiting forever to terminate if it is primary,
and avoids this entirely if there is only a single node in the cluster.
 2021-12-28 03:06:03 -05:00
Joshua Boniface 67131de4f6 Fix bug when removing OSDs 
Ensure the OSD is down as well as out or purge might fail.
 2021-12-28 03:05:34 -05:00
Joshua Boniface abc23ebb18 Handle detect strings as arguments for blockdevs 
Allows specifying blockdevs in the OSD and OSD-DB addition commands as
detect strings rather than actual block device paths. This provides
greater flexibility for automation with pvcbootstrapd (which originates
the concept of detect strings) and in general usage as well.
 2021-12-28 02:53:02 -05:00
Joshua Boniface f164d898c1 Bump version to 0.9.45 2021-11-25 09:34:20 -05:00
Joshua Boniface 817dffcf30 Bump version to 0.9.44 2021-11-11 16:20:38 -05:00
Joshua Boniface 6e9fcd38a3 Bump version to 0.9.43 2021-11-08 02:29:17 -05:00
Joshua Boniface 78faa90139 Reformat recent changes with Black 2021-11-06 03:27:07 -04:00
Joshua Boniface 23b1501f40 Fix linting error F541 f-string placeholders 2021-11-06 03:26:03 -04:00
Joshua Boniface 66bfad3109 Fix linting errors F522/F523 unused args 2021-11-06 03:24:50 -04:00
Joshua Boniface c41664d2da Reformat code with Black code formatter 
Unify the code style along PEP and Black principles using the tool.
 2021-11-06 03:02:43 -04:00
Joshua Boniface 2e7b9b28b3 Add some delay and additional tries to fencing 2021-10-27 16:24:17 -04:00
Joshua Boniface 55f397a347 Fix bad location of config sets 2021-10-12 17:23:04 -04:00
Joshua Boniface dfebb2d3e5 Also validate on failures 2021-10-12 17:11:03 -04:00
Joshua Boniface e88147db4a Bump version to 0.9.42 2021-10-12 15:25:42 -04:00
Joshua Boniface b8204d89ac Go back to passing if exception 
Validation already happened and the set happens again later.
 2021-10-12 14:21:52 -04:00
Joshua Boniface fe73dfbdc9 Use current live value for bridge_mtu 
This will ensure that upgrading without the bridge_mtu config key set
will keep things as they are.
 2021-10-12 12:24:03 -04:00
Joshua Boniface 8f906c1f81 Use power off in fence instead of reset 
Use a power off (and then make the power on a requirement) during a node
fence. Removes some potential ambiguity in the power state, since we
will know for certain if it is off.
 2021-10-12 11:04:27 -04:00
Joshua Boniface 2d9fb9688d Validate network MTU after initial read 2021-10-12 10:53:17 -04:00
Joshua Boniface f13cc04b89 Bump version to 0.9.41 2021-10-09 19:39:21 -04:00
Joshua Boniface 95e01f38d5 Adjust log type of object setup message 2021-10-09 19:23:12 -04:00
Joshua Boniface 3122d73bf5 Avoid duplicate runs of MTU set 
It wasn't the validator duplicating, but the update duplicating, so
avoid that happening properly this time.
 2021-10-09 19:21:47 -04:00
Joshua Boniface 7ed8ef179c Revert "Avoid duplicate runs of MTU validator" 
This reverts commit 56021c443a.
 2021-10-09 19:11:42 -04:00
Joshua Boniface caead02b2a Set all log messages to information state 
None of these were "success" messages and thus shouldn't have been ok
state.
 2021-10-09 19:09:38 -04:00
Joshua Boniface 87bc5f93e6 Avoid duplicate runs of MTU validator 2021-10-09 19:07:41 -04:00
Joshua Boniface 203893559e Use correct isinstance instead of type 2021-10-09 19:03:31 -04:00
Joshua Boniface 2c51bb0705 Move MTU validation to function 
Prevents code duplication and ensures validation runs when an MTU is
updated, not just on network creation.
 2021-10-09 19:01:45 -04:00
Joshua Boniface 46d3daf686 Add logger message when setting MTU 2021-10-09 18:56:18 -04:00
Joshua Boniface e9d05aa24e Ensure vx_mtu is always an int() 2021-10-09 18:52:50 -04:00
Joshua Boniface 6ce28c43af Add MTU value checking and log messages 
Ensures that if a specified MTU is more than the maximum it is set to
the maximum instead, and adds warning messages for both situations.
 2021-10-09 18:48:56 -04:00
Joshua Boniface c45f8f5bd5 Have VXNetworkInstance set MTU if unset 
Makes this explicit in Zookeeper if a network is unset, post-migration
(schema version 6).

Addresses #144
 2021-10-09 17:52:57 -04:00
Joshua Boniface 3690a2c1e0 Fix migration bugs and invalid vx_mtu 
Addresses #144
 2021-10-09 17:35:10 -04:00
Joshua Boniface 50d8aa0586 Add handlers for client network MTUs 
Refactors some of the code in VXNetworkInterface to handle MTUs in a
more streamlined fashion. Also fixes a bug whereby bridge client
networks were being explicitly given the cluster dev MTU which might not
be correct. Now adds support for this option explicitly in the configs,
and defaults to 1500 for safety (the standard Ethernet MTU).

Addresses #144
 2021-10-09 17:02:27 -04:00
Joshua Boniface 6ee4c55071 Correct flawed conditional in verify_ipmi 2021-10-07 15:11:19 -04:00
Joshua Boniface c27359c4bf Bump version to 0.9.40 2021-10-07 14:42:04 -04:00
Joshua Boniface 46078932c3 Correct bad stop_keepalive_timer call 2021-10-07 14:41:12 -04:00
Joshua Boniface bdb9db8375 Bump version to 0.9.39 2021-10-07 11:52:38 -04:00
Joshua Boniface da9248cfa2 Bump version to 0.9.38 2021-10-03 22:32:41 -04:00
Joshua Boniface 23977b04fc Bump version to 0.9.37 2021-09-30 02:08:14 -04:00
Joshua Boniface f6f6f07488 Add timeouts to queue gets and adjust 
Ensure that all keepalive timeouts are set (prevent the queue.get()
actions from blocking forever) and set the thread timeouts to line up as
well. Everything here is thus limited to keepalive_interval seconds
(default 5s) to keep it uniform.
 2021-09-27 16:10:27 -04:00
Joshua Boniface 142c999ce8 Re-add success log output during migration 2021-09-27 11:50:55 -04:00
Joshua Boniface 1de069298c Fix missing character in log message 2021-09-27 00:49:43 -04:00
Joshua Boniface 55221b3d97 Simplify VM migration down to 3 steps 
Remove two superfluous synchronization steps which are not needed here,
since the exclusive lock handles that situation anyways.

Still does not fix the weird flush->unflush lock timeout bug, but is
better worked-around now due to the cancelling of the other wait freeing
this up and continuing.
 2021-09-27 00:03:20 -04:00
Joshua Boniface 0d72798814 Work around synchronization lock issues 
Make the block on stage C only wait for 900 seconds (15 minutes) to
prevent indefinite blocking.

The issue comes if a VM is being received, and the current unflush is
cancelled for a flush. When this happens, this lock acquisition seems to
block for no obvious reason, and no other changes seem to affect it.
This is certainly some sort of locking bug within Kazoo but I can't
diagnose it as-is. Leave a TODO to look into this again in the future.
 2021-09-26 23:26:21 -04:00
Joshua Boniface 3638efc77e Improve log messages during VM migration 2021-09-26 23:15:38 -04:00
Joshua Boniface c2c888d684 Use event to non-block wait and fix inf wait 2021-09-26 22:55:39 -04:00
Joshua Boniface febef2e406 Track status of VM state thread 2021-09-26 22:55:21 -04:00