Commit Graph

3228 Commits

Author SHA1 Message Date
Joshua Boniface 4c3f235e05 Avoid running fault updates in maintenance mode
When the cluster is in maintenance mode, all faults should be ignored.
2023-12-01 17:38:28 -05:00
Joshua Boniface 3dc48c1783 Lower default monitoring interval to 15s
Faults are also reported on the monitoring interval, so 60s seems like
too long. Lower this to 15 seconds by default instead.
2023-12-01 17:38:28 -05:00
Joshua Boniface 9c2b1b29ee Add node health to fault states
Adjusts ordering and ensures that node health states are included in
faults if they are less than 50%.

Also adjusts fault ID generation and runs fault checks only coordinator
nodes to avoid too many runs.
2023-12-01 17:38:28 -05:00
Joshua Boniface 8594eb697f Add initial fault generation in pvchealthd
References: #164
2023-12-01 17:38:27 -05:00
Joshua Boniface 988de1218f Bump version to 0.9.83 2023-12-01 17:37:42 -05:00
Joshua Boniface 0ffcbf3152 Fix bad file paths 2023-12-01 17:25:12 -05:00
Joshua Boniface ad8d8cf7a7 Avoid removing changelog file until the end
Avoids losing a changelog if something else fails.
2023-12-01 17:23:43 -05:00
Joshua Boniface 915a84ee3c Fix psql check for new configs 2023-12-01 03:58:21 -05:00
Joshua Boniface 6315a068d1 Use SafeLoader for config load 2023-12-01 02:01:24 -05:00
Joshua Boniface 2afd064445 Update CLI to read from pvc.conf 2023-12-01 01:53:33 -05:00
Joshua Boniface 7cb9ebae6b Remove legacy configuration handler
This is not going to be needed.
2023-12-01 01:25:40 -05:00
Joshua Boniface 1fb0463dea Adjust daemon service startup
Add healthd, adjust workerd, lower waittime
2023-11-30 03:28:02 -05:00
Joshua Boniface 13549fc995 Depend pvcnoded on pvcworkerd 2023-11-30 03:24:01 -05:00
Joshua Boniface 102c3c3106 Port all Celery worker functions to discrete pkg
Moves all tasks run by the Celery worker into a discrete package/module
for easier installation. Also adjusts several parameters throughout to
accomplish this.
2023-11-30 02:24:54 -05:00
Joshua Boniface 0c0fb65c62 Rework Flask API to route Celery tasks manually
Avoids needing to define any of these tasks here; they can all be
defined in the pvcworkerd code.
2023-11-30 00:40:09 -05:00
Joshua Boniface 03a738f878 Move config parser into daemon_lib
And reformat/add config values for API.
2023-11-30 00:05:37 -05:00
Joshua Boniface 4df5fdbca6 Update description of example conf 2023-11-29 21:21:51 -05:00
Joshua Boniface 97eb63ebab Clean up config naming and dead files 2023-11-29 21:21:51 -05:00
Joshua Boniface 4a2eba0961 Improve node output messages (from pvchealthd)
1. Output startup "list" entries in cyan with s state
2. Add start of keepalive run message
2023-11-29 21:21:51 -05:00
Joshua Boniface 077dd8708f Add check start message 2023-11-29 21:21:51 -05:00
Joshua Boniface b6b5786c3b Output list in cyan (s state) 2023-11-29 21:21:51 -05:00
Joshua Boniface ad738dec40 Clean up plugin pycache too 2023-11-29 21:21:51 -05:00
Joshua Boniface d2b764a2c7 Output more details on startup 2023-11-29 21:21:51 -05:00
Joshua Boniface b8aecd9c83 Wait less time between restarts 2023-11-29 21:21:51 -05:00
Joshua Boniface 11db3c5b20 Fix ordering during termination 2023-11-29 21:21:51 -05:00
Joshua Boniface 7a7c975eff Ensure return from health shutdown 2023-11-29 21:21:51 -05:00
Joshua Boniface fa12a3c9b1 Permit buffered log appending 2023-11-29 21:21:51 -05:00
Joshua Boniface 787f4216b3 Expand Zookeeper log daemon prefix to match 2023-11-29 21:21:51 -05:00
Joshua Boniface 647cba3cf5 Expand startup width for new daemon name 2023-11-29 21:21:51 -05:00
Joshua Boniface 921ecb3a05 Fix name in kydb plugin 2023-11-29 21:21:51 -05:00
Joshua Boniface 6a68cf665b Wait between service restarts 2023-11-29 21:21:51 -05:00
Joshua Boniface 41f4e4fb2f Split health monitoring into discrete daemon/pkg 2023-11-29 21:21:51 -05:00
Joshua Boniface 74a416165d Move default autobackup config to pvc.conf 2023-11-29 21:21:37 -05:00
Joshua Boniface 83ceb41138 Add daemon name to Logger entries 2023-11-29 15:18:37 -05:00
Joshua Boniface 2e5958640a Remove erroneous time from message 2023-11-29 15:12:41 -05:00
Joshua Boniface d65b18f15b Improve handling of loglines on client 2023-11-29 15:12:41 -05:00
Joshua Boniface 7abc697c8a Improve Zookeeper log handling
Ensures that messages are fully read before each append. Adds more
Zookeeper hits, but ensures logs won't be overwritten by multiple
daemons.

Also don't use a set on the client side, to avoid "removing duplicate"
entries erroneously.
2023-11-29 15:12:41 -05:00
Joshua Boniface 1adc3674b6 Install everything uploaded 2023-11-29 15:12:28 -05:00
Joshua Boniface bd811408f9 Remove "Python 3" from package descriptions 2023-11-29 15:12:09 -05:00
Joshua Boniface 6090b286fe Improve copying to avoid leaving stale files 2023-11-29 14:46:59 -05:00
Joshua Boniface 2545a7b744 Allow similar for IPMI hostnames 2023-11-28 16:09:01 -05:00
Joshua Boniface ce907ff26a Allow specifying static IPs instead of a file 2023-11-28 15:28:31 -05:00
Joshua Boniface 71e589e461 Remove superflous debug output
This is printed in the startup logo block anyways.
2023-11-27 13:46:30 -05:00
Joshua Boniface fc3d292081 Add missing subdirectory configs 2023-11-27 13:40:07 -05:00
Joshua Boniface eab1ae873b Ensure upstream_gateway key will exist 2023-11-27 13:37:57 -05:00
Joshua Boniface eaf93cdf96 Readd missing subsystem configurations 2023-11-27 13:33:41 -05:00
Joshua Boniface c8f4cbb39e Fix node entry keys 2023-11-27 13:24:01 -05:00
Joshua Boniface 786fae7769 Improve logo output 2023-11-27 13:01:43 -05:00
Joshua Boniface 17f81e8296 Refactor pvcapid to use new configuration 2023-11-27 12:49:26 -05:00
Joshua Boniface bcc57638a9 Refactor pvcnoded to use new configuration 2023-11-26 15:41:25 -05:00