Commit Graph

115 Commits

Author SHA1 Message Date
Joshua Boniface b4c2b9bdf8 Lower sync and init tick limits
Lower both of these to 5 seconds to ensure Zookeeper doesn't linger on
startup or synchronization while pvcnoded is starting up (15s in
0.9.101).
2024-10-15 11:11:10 -04:00
Joshua Boniface e35842675f Add new is-primary condition to autobackup timers
Avoids running these except on the primary coordinator at the time of
firing, to avoid duplicate runs being passed to pvcworkerd.
2024-08-25 22:18:20 -04:00
Joshua Boniface 5a3233a905 Fix shutdown bug with keydb 2024-03-07 13:21:10 -05:00
Joshua Boniface db8c437a31 Add email reporting for autobackups (0.9.90) 2024-01-11 00:32:38 -05:00
Joshua Boniface b7f1c55279 Update and add default vars for autobackup 2024-01-09 09:54:38 -05:00
Joshua Boniface f6878c69ad Add enable_prometheus switch 2023-12-29 09:29:29 -05:00
Joshua Boniface 5089aec178 Use node names rather than IPs 2023-12-28 11:54:33 -05:00
Joshua Boniface ae4237c173 Use all coordinators 2023-12-28 11:47:38 -05:00
Joshua Boniface ce14cdb938 Rename file for consistency 2023-12-28 11:38:33 -05:00
Joshua Boniface 09eba9de80 Add Prometheus Zookeeper exporter 2023-12-10 12:39:35 -05:00
Joshua Boniface 1eb618e8f3 Fix typo 2023-12-10 03:15:02 -05:00
Joshua Boniface 1f34427a0c Add user as well 2023-12-10 03:05:11 -05:00
Joshua Boniface c7d914696b Add configuration for Postgres Prom exporter 2023-12-10 02:53:11 -05:00
Joshua Boniface 7a2c16e2a5 Disable Zookeeper Prometheus export
This does not work as this functionality is built out of the Debian
packages. Hopefully it will be enabled one day...
2023-12-10 01:41:06 -05:00
Joshua Boniface 23bb3cb4d9 Add JVM info to Zookepeer metric 2023-12-10 01:16:44 -05:00
Joshua Boniface 54bf70d336 Enable Prometheus metrics in Zookeeper too 2023-12-10 00:32:37 -05:00
Joshua Boniface 18054c01a0 Remove obsolete config templates 2023-12-09 23:05:16 -05:00
Joshua Boniface b1d6915cf4 Write new pvc.conf style configuration (0.9.82+) 2023-11-28 16:10:23 -05:00
Joshua Boniface c8764159f6 Readd queue configuration with updated options 2023-11-05 23:37:49 -05:00
Joshua Boniface 523f7da71e Add KeyDB (Redis clone) to configuration
Replaces Redis for PVC >= 0.9.81
2023-11-05 19:24:30 -05:00
Joshua Boniface 2f9603c82f Adjust pvcapid.yaml for 0.9.81 worker queue config 2023-11-04 12:52:21 -04:00
Joshua Boniface 103e9fe147 Add restart overrides for ceph-mgr
Needed because ceph-mgr seems to crash frequently under Debian 12 when
adding or removing OSDs. The default settings do not restart it
properly, so this override does.
2023-11-03 14:25:31 -04:00
Joshua Boniface 90417621d7 Add autobackup support to pvc-ansible 2023-10-27 02:08:20 -04:00
Joshua Boniface c11f896a60 Fix zk_status check target znode 2023-10-22 00:42:43 -04:00
Joshua Boniface 83636388f0 Add configurable monitoring interval 2023-09-15 22:31:16 -04:00
Joshua Boniface 353399a407 Ensure core pg_hba entries are present 2023-09-01 15:42:30 -04:00
Joshua Boniface b21778f117 Fix Patroni upgrade and D12 support 2023-09-01 15:42:30 -04:00
Joshua Boniface f79d1da5be Update other commands to use new CLI format 2023-09-01 15:42:30 -04:00
Joshua Boniface cab4deac26 Add configuration field for plugins 2023-09-01 15:42:29 -04:00
Joshua Boniface c2b576334f Adjust plugin log config field for 0.9.62 2023-09-01 15:42:29 -04:00
Joshua Boniface 07d75573d6 Add updated tuning configuration
Uses a much nicer CPU tuning configuration, leveraging systemd's
AllowedCPUs and CPUAffinity options within a set of slices (some
default, some custom).

Configuration is also greatly simplified versus the previous
implementation, simply asking for a number of CPUS for both the system
and OSDs, and calculating everything else that is required.

Also switches (back) to the v2 unified cgroup hierarchy by default as
required by the systemd AllowedCPUs directive.
2023-09-01 15:42:29 -04:00
Joshua Boniface fa4f1cff0f Adjust variable used for migrate selector 2023-09-01 15:42:29 -04:00
Joshua Boniface 1d35fec8a8 Remove cpuset configurations
This functionality simply did not work, with Libvirt continuing to dump
its processes into the root cset thus defeating the purpose entirely.

Just remove it, from some very initial testing it isn't worth the
headache.
2023-09-01 15:42:29 -04:00
Joshua Boniface f51fc2ce64 Fix setting of csets for OSDs 2023-09-01 15:42:29 -04:00
Joshua Boniface e4ccafee73 Add cgroup delegation override
Required to solve the occasional
  libvirt: QEMU Driver error : Requested operation is not valid:
  cgroup CPUACCT controller is not mounted
problem, as per:
  https://answers.launchpad.net/ubuntu/+question/665132
2023-09-01 15:42:28 -04:00
Joshua Boniface 9fe43efac2 Convert default libvirtd to template 2023-09-01 15:42:28 -04:00
Joshua Boniface 1838f8ff56 Add proper PostgreSQL versioning 2023-09-01 15:42:28 -04:00
Joshua Boniface d6cb28b639 Add immutability to PVC subrole
1. Remove the obsolete pvc-vacuum script install.

2. Remove notifies when modifying configs; we do not want to restart the
daemons uncontrolled.

3. Add bootstrap check to package installs so they only happen on
bootstrap.

This ensures this part of the role, on re-runs, will *only* update
configs and not actually touch the running daemon. This makes it safe to
run before a oneshot/update-pvc-daemons.yml playbook run.
2023-09-01 15:42:28 -04:00
Joshua Boniface 77be96bf6f Fix a few more splits 2023-09-01 15:42:28 -04:00
Joshua Boniface 87803cb7a2 Remove extraneous splits
The node.hostname should always be short.
2023-09-01 15:42:28 -04:00
Joshua Boniface d24cb8a8ef Unify and standardize inventory_hostname
This was causing some confusing conflicts, so create a new fact called
"this_node" which is inventory_hostname.split('.')[0], i.e. the short
name, and use that everywhere instead of an FQDN or true inventory
hostname.
2023-09-01 15:42:28 -04:00
Joshua Boniface 4cb2d7835c Add setting bridge_mtu to config 2023-09-01 15:42:27 -04:00
Joshua Boniface 6e2d661134 Adjust documentation and behaviour of cpuset
1. Detail the caveats and specific situations and ref the documentation
which will provide more details.

2. Always install the configs, but use /etc/default/ceph-osd-cpuset to
control if the script does anything or not (so, the "osd" cset set is
always active just not set in a special way.
2023-09-01 15:42:27 -04:00
Joshua Boniface 7927ec4f11 Allow dynamic enabling/disabling of cset
Add a separate config to handle enable/disable on the system itself.
2023-09-01 15:42:27 -04:00
Joshua Boniface 6e48d6fe84 Add Ceph OSD cpuset tuning options
Allows an administrator to set CPU pinning with the cpuset tool for Ceph
OSDs, in situations where CPU contention with VMs or other system tasks
may be negatively affecting OSD performance. This is optional, advanced
tuning and is disabled by default.
2023-09-01 15:42:27 -04:00
Joshua Boniface f25a80ff53 Add additional CMK checks 2023-09-01 15:42:26 -04:00
Joshua Boniface 3d64ad2420 Typo fix 2023-09-01 15:42:26 -04:00
Joshua Boniface eaea860b61 Lower autopurge interval to 1 hour 2023-09-01 15:42:26 -04:00
Joshua Boniface 524f857f56 Add some Zookeeper configuration tweaks 2023-09-01 15:42:26 -04:00
Joshua Boniface 13556918d7 Disable any systemd start rate limiting
Because Zookeeper is supremely stupid (see last commit) we want to
disable start limiting. It needs to keep trying forever until it starts.
2023-09-01 15:42:26 -04:00