a3171b666b
Split node health into separate function
2023-12-10 16:52:10 -05:00
48e41d7b05
Port Faults getFault and getAllFaults to read_many
2023-12-10 16:05:16 -05:00
d6aecf195e
Port Node getNodeInformation to read_many
2023-12-10 15:53:28 -05:00
9329784010
Implement async ZK read function
...
Adds a function, "read_many", which can take in multiple ZK keys and
return the values from all of them, using asyncio to avoid reading
sequentially.
Initial tests show a marked improvement in read performance of multiple
read()-heavy functions (e.g. "get_list()" functions) with this method.
2023-12-10 15:35:40 -05:00
9dc5097dbc
Bump version to 0.9.85
2023-12-10 01:00:33 -05:00
5776cb3a09
Remove Prometheus client dependencies
...
We don't actually use this (yet!) so remove the dependency for now.
2023-12-10 00:58:09 -05:00
53d632f283
Fix bug in example PVC Grafana dashboard
2023-12-10 00:50:05 -05:00
7bc0760b78
Add time to "starting keepalive" message
...
Matches the pvchealthd output and provides a useful message detail to
this otherwise contextless message.
2023-12-10 00:40:32 -05:00
9aee2a9075
Bump version to 0.9.84
2023-12-09 23:05:40 -05:00
8f0ae3e2dd
Fix config file for database migrations
2023-12-09 22:51:54 -05:00
946d3eaf43
Add wait after stopping VM
2023-12-09 18:14:03 -05:00
1f6347d24b
Add Prometheus monitoring examples
2023-12-09 17:42:51 -05:00
e8552b471b
Require at least one FAULT_ID
2023-12-09 17:31:56 -05:00
fc443a323b
Allow ack/delete of multiple faults at once
2023-12-09 17:28:13 -05:00
b0557edb76
Ensure entry in name is uppercase
2023-12-09 17:01:41 -05:00
47bd7bf2f5
Only run cluster-wide health checks on primary
...
Avoids multiple coordinators trying to write updated cluster-wide fault
events. Instead, they are now only written by the primary (or the
incoming primary if still in a transition).
2023-12-09 16:50:51 -05:00
b9fbfe2ed5
Improve fault ID format
...
Instead of using random hex characters from an md5sum, use a nice name
in all-caps similar to how Ceph does. This further helps prevent dupes
but also permits a changing health delta within a single event (which
would really only ever apply to plugin faults).
2023-12-09 16:48:14 -05:00
764e3e3722
Fix bug in fault header format
2023-12-09 16:47:56 -05:00
7e6d922877
Improve fault detail handling further
...
Since we already had a "details" field, simply move where it gets added
to the message later, in generate_fault, after the main message value
was used to generate the ID.
2023-12-09 16:13:36 -05:00
4ca2381077
Rework metrics output and add combined endpoint
2023-12-09 15:47:40 -05:00
4003204f14
Remove bracketed text from fault_str
...
This ensures that certain faults e.g. Ceph status faults, will be
combined despite the added text in brackets, while still keeping them
mostly separate.
Also ensure the health text is updated each time to assist with this, as
this health text may now change independent of the fault ID.
2023-12-09 15:34:18 -05:00
a70c1d63b0
Separate state totals from states, separate states
2023-12-09 13:59:17 -05:00
2bea78d25e
Make all remaining limits optional
2023-12-09 13:43:58 -05:00
fd717b702d
Use external list of fault states
2023-12-09 12:51:41 -05:00
132cde5591
Add totals and nice-format states
...
Avoids tons of annoying rewriting in the UI later.
2023-12-09 12:50:19 -05:00
ba565ead4c
Report all state combinations in Prom metrics
...
Ensures that every state combination is always shown to metrics, even if
it contains 0 entries.
2023-12-09 12:40:37 -05:00
317ca4b98c
Move defined state combinations into common
2023-12-09 12:36:32 -05:00
2b8abea8df
Remove debug printing
2023-12-09 12:22:36 -05:00
9b3c9f1be5
Add Ceph metrics proxy and health fault counts
2023-12-09 12:22:36 -05:00
7373bfed3f
Add Prometheus metric exporter
...
Adds a "fake" Prometheus metrics endpoint which returns cluster status
information in Prometheus format.
2023-12-09 12:22:36 -05:00
d0e7c19602
Add prometheus client dependencies
2023-12-09 12:22:36 -05:00
f01c12c86b
Import from pvcworkerd not pvcapid
2023-12-09 12:22:19 -05:00
0bda095571
Move libvirt_schema and fix other imports
2023-12-09 12:20:29 -05:00
7976e1d2d0
Correct import location in scripts
2023-12-09 12:18:33 -05:00
813aef1463
Fix incorrect UUID key name
2023-12-09 12:14:57 -05:00
5a7ea25266
Fix incorrect database name entries
2023-12-09 12:12:00 -05:00
82a7fd3c80
Add more debugging info to psql
2023-12-07 21:36:05 -05:00
ddd9d9ee07
Adjust psql check to avoid weird failures
2023-12-07 15:07:59 -05:00
9e2e749c55
Combine pvchealthd output into single log message
2023-12-07 14:00:43 -05:00
157b8c20bf
Add Patroni output to debug logs
2023-12-07 14:00:35 -05:00
bf158dc2d9
Shorten debug output
2023-12-07 13:31:20 -05:00
1b84553405
Use passed coordinator state
2023-12-07 11:19:26 -05:00
60dac143f2
Use simpler health calculation
2023-12-07 11:17:31 -05:00
a13273335d
Add colon to result text
2023-12-07 11:15:42 -05:00
e7f21b7058
Enhance and fix bugs in psql plugin
...
1. Check Patronictl statuses
2. Don't error during node primary transitions
2023-12-07 11:14:16 -05:00
9dbadfdd6e
Move back to per-plugin fault reporting
2023-12-07 11:13:56 -05:00
61b39d0739
Fix incorrect cluster health calculation
2023-12-07 11:13:36 -05:00
4bf80a5913
Fix missing datetime shrink
2023-12-06 17:15:36 -05:00
6c0dfe16cf
Improve word splitting for fault messages
...
This ensures that fault messages are split on word boundaries and that
the column length is equal to the longest of these if applicable.
2023-12-06 17:10:19 -05:00
3fde494fc5
Add status back to short fault list
2023-12-06 16:53:23 -05:00
