Commit Graph

30 Commits

Author SHA1 Message Date
Joshua Boniface 5089aec178 Use node names rather than IPs 2023-12-28 11:54:33 -05:00
Joshua Boniface ae4237c173 Use all coordinators 2023-12-28 11:47:38 -05:00
Joshua Boniface ce14cdb938 Rename file for consistency 2023-12-28 11:38:33 -05:00
Joshua Boniface 09eba9de80 Add Prometheus Zookeeper exporter 2023-12-10 12:39:35 -05:00
Joshua Boniface 7a2c16e2a5 Disable Zookeeper Prometheus export
This does not work as this functionality is built out of the Debian
packages. Hopefully it will be enabled one day...
2023-12-10 01:41:06 -05:00
Joshua Boniface 23bb3cb4d9 Add JVM info to Zookepeer metric 2023-12-10 01:16:44 -05:00
Joshua Boniface 54bf70d336 Enable Prometheus metrics in Zookeeper too 2023-12-10 00:32:37 -05:00
Joshua Boniface c11f896a60 Fix zk_status check target znode 2023-10-22 00:42:43 -04:00
Joshua Boniface d24cb8a8ef Unify and standardize inventory_hostname
This was causing some confusing conflicts, so create a new fact called
"this_node" which is inventory_hostname.split('.')[0], i.e. the short
name, and use that everywhere instead of an FQDN or true inventory
hostname.
2023-09-01 15:42:28 -04:00
Joshua Boniface f25a80ff53 Add additional CMK checks 2023-09-01 15:42:26 -04:00
Joshua Boniface 3d64ad2420 Typo fix 2023-09-01 15:42:26 -04:00
Joshua Boniface eaea860b61 Lower autopurge interval to 1 hour 2023-09-01 15:42:26 -04:00
Joshua Boniface 524f857f56 Add some Zookeeper configuration tweaks 2023-09-01 15:42:26 -04:00
Joshua Boniface 13556918d7 Disable any systemd start rate limiting
Because Zookeeper is supremely stupid (see last commit) we want to
disable start limiting. It needs to keep trying forever until it starts.
2023-09-01 15:42:26 -04:00
Joshua Boniface 8eecc95f2f Ensure Zookeeper restarts itself
The Zookeeper daemon does not appear to exit with any status other than
0, even after a fatal error. Work around this.
2023-09-01 15:42:26 -04:00
Joshua Boniface b03ecf0125 Add -XX:+AlwaysPreTouch option for Zookeeper 2023-09-01 15:42:26 -04:00
Joshua Boniface b842276002 Lower keep count for Zookeeper vacuum to 3
Required to keep disk space growth down when using zookeeper_logging
functionality.
2023-09-01 15:42:26 -04:00
Joshua Boniface 2d31e6c8ea Fix memory tuning issues 2023-09-01 15:42:26 -04:00
Joshua Boniface d455b31905 Bump max connections in Zookeeper to 200 2023-09-01 15:42:26 -04:00
Joshua Boniface f105f0497c Configure Zookeeper only on Cluster address 2023-09-01 15:42:26 -04:00
Joshua Boniface be94bc134f Add configurable ZK memory limits 2023-09-01 15:42:24 -04:00
Joshua Boniface 2bd5cc5a25 Tune Zookeeper memory usage
Use Xms and Xmx=128M to reduce overall Zookeeper memory usage.
2023-09-01 15:42:24 -04:00
Joshua Boniface bcb5962353 Add jute.maxbuffer to Zookeeper environment ops
Adds this option based on the findings of
https://github.com/python-zk/kazoo/issues/630, whereby restores of >1MB
in size would fail. This is considered an unsafe option, but given our
usecase no actual znode should ever exceed this limit; this is purely
for the large transactions that come from a `pvc task restore` action to
an empty Zookeeper instance.
2023-09-01 15:42:24 -04:00
Joshua Boniface 72df058684 Ensure ZK prioritizes IPv4 2023-09-01 15:42:24 -04:00
Joshua Boniface 457e18a850 Use FQDN for Zookeeper server entries 2023-09-01 15:42:24 -04:00
Joshua Boniface 157c56fd46 Use shortname for Zookeeper 2023-09-01 15:42:23 -04:00
Joshua Boniface 94ef3490ab Add daily Zookeeper data cleanup 2023-09-01 15:42:21 -04:00
Joshua Boniface 15a2bf1418 Add custom systemd unit for Zookeeper
We're 100% systemd here, and the lack of control/information that the
old-school ZK initscript provides is frustrating. Replace it with our
own simple unit file.
2023-09-01 15:42:21 -04:00
Joshua Boniface 2cee9f08a0 Add Zookeeper autopurge @72h 2023-09-01 15:42:21 -04:00
Joshua Boniface 6dfaf433dc Initial commit of PVC Ansible role 2023-09-01 15:42:19 -04:00