From 673a53331af387f326ca089243e38bcf8e8d36a6 Mon Sep 17 00:00:00 2001 From: "Joshua M. Boniface" Date: Sun, 1 Dec 2024 12:11:41 -0500 Subject: [PATCH] Update Patroni endpoints --- .../patroni-and-haproxy-agent-checks/index.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/content/en/posts/patroni-and-haproxy-agent-checks/index.md b/content/en/posts/patroni-and-haproxy-agent-checks/index.md index e6f4f77..dbf35a1 100644 --- a/content/en/posts/patroni-and-haproxy-agent-checks/index.md +++ b/content/en/posts/patroni-and-haproxy-agent-checks/index.md @@ -13,7 +13,7 @@ Once you have a working Patroni cluster, managing client access to it becomes th ### The problem - Do you like `DOWN` hosts? -However, the [official HAProxy configuration template](https://github.com/zalando/patroni/blob/master/extras/confd/templates/haproxy.tmpl) has a problem - in a read-write backend, you want your non-`master` hosts to be inaccessable to clients, to prevent write attempts against a read-only replica. However this configuration results in the `replica` hosts being marked `DOWN` in HAProxy. +However, the [official HAProxy configuration template](https://github.com/zalando/patroni/blob/master/extras/confd/templates/haproxy.tmpl) has a problem - in a read-write backend, you want your non-`primary` hosts to be inaccessable to clients, to prevent write attempts against a read-only replica. However this configuration results in the `replica` hosts being marked `DOWN` in HAProxy. Now, some people might ask "well, why is that a big deal"? And they may be right. However, as soon as you start trying to monitor your HAProxy backends via an external monitoring tool, you see the problem: "CRITICAL" alerts during normal operation! After all, a `DOWN` host is considered a _problem_ in 99.9% of HAProxy usecases. But with Patroni, it's expected behaviour, which is not ideal. @@ -23,7 +23,7 @@ So what can we do? HAProxy, since at least version 1.5, supports [a feature called `agent-check`](https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#5.2-agent-check). In short, this "enable[s] an auxiliary agent check which is run independently of a regular health check". The `agent-check` will connect to a specific port on either the backend host or another target, and will modify the backend status based on the response, which must be one of the common HAProxy keyworks (eg. `MAINT` or `READY`). -So how does this help us? Well, if we had some way to obtain Patroni's `master`/`replica` status for each host, we could, instead of having the `replica` machines marked `DOWN`, put them into `MAINT` mode instead. This provides cleanliness for monitoring purposes while still letting us use the typical Patroni HAProxy configuration, with just minimal modifications to the HAProxy configuation and deploying an additional daemon on the Patroni hosts. +So how does this help us? Well, if we had some way to obtain Patroni's `primary`/`replica` status for each host, we could, instead of having the `replica` machines marked `DOWN`, put them into `MAINT` mode instead. This provides cleanliness for monitoring purposes while still letting us use the typical Patroni HAProxy configuration, with just minimal modifications to the HAProxy configuation and deploying an additional daemon on the Patroni hosts. ### The Code - Python 3 daemon @@ -38,7 +38,7 @@ Here is the code - I'm sure it can be improved significantly but it works for me ``` #!/usr/bin/env python3 -# Simple agent check for HAProxy to determine Patroni master/replica status +# Simple agent check for HAProxy to determine Patroni primary/replica status import socket, requests @@ -75,8 +75,8 @@ while True: conn, addr = sock.accept() state = getstate() - # Set our response based on the state; only `master` should be READY in read-write mode - if state == 'master': + # Set our response based on the state; only `primary` should be READY in read-write mode + if state == 'primary': data = b'READY\n' else: data = b'MAINT\n' @@ -124,7 +124,7 @@ Now finally, configure your HAProxy backend to use the agent check. Here's my (l backend mast-pgX_psql_readwrite mode tcp option tcpka - option httpchk OPTIONS /master + option httpchk OPTIONS /primary http-check expect status 200 server mast-pg1 mast-pg1:5432 resolvers nsX resolve-prefer ipv4 maxconn 100 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008 server mast-pg2 mast-pg2:5432 resolvers nsX resolve-prefer ipv4 maxconn 100 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008 @@ -138,3 +138,5 @@ And here it is in action: ### Conclusion I hope that this provides some help to those who want to use Patroni fronted by HAProxy but don't want `DOWN` backends all the time! And of course, I'm open to suggestions for improvement or questions - just send me an email! + +UPDATE 2024-12-01: Updated various instances of `master` to `primary` to reflect changes in Patroni since this post was originally written. Thanks to [Gary T. Giesen][https://fosstodon.org/@ggiesen] for pointing these out!