joshuaboniface
/
blog
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Update Patroni endpoints

This commit is contained in:
Joshua Boniface 2024-12-01 12:11:41 -05:00
parent 53593e4ab4
commit 673a53331a
1 changed files with 8 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
content/en/posts/patroni-and-haproxy-agent-checks/index.md
View File

@ -13,7 +13,7 @@ Once you have a working Patroni cluster, managing client access to it becomes th
### The problem - Do you like `DOWN` hosts? ### The problem - Do you like `DOWN` hosts?
However, the [official HAProxy configuration template](https://github.com/zalando/patroni/blob/master/extras/confd/templates/haproxy.tmpl) has a problem - in a read-write backend, you want your non-`master` hosts to be inaccessable to clients, to prevent write attempts against a read-only replica. However this configuration results in the `replica` hosts being marked `DOWN` in HAProxy. However, the [official HAProxy configuration template](https://github.com/zalando/patroni/blob/master/extras/confd/templates/haproxy.tmpl) has a problem - in a read-write backend, you want your non-`primary` hosts to be inaccessable to clients, to prevent write attempts against a read-only replica. However this configuration results in the `replica` hosts being marked `DOWN` in HAProxy.
Now, some people might ask "well, why is that a big deal"? And they may be right. However, as soon as you start trying to monitor your HAProxy backends via an external monitoring tool, you see the problem: "CRITICAL" alerts during normal operation! After all, a `DOWN` host is considered a _problem_ in 99.9% of HAProxy usecases. But with Patroni, it's expected behaviour, which is not ideal. Now, some people might ask "well, why is that a big deal"? And they may be right. However, as soon as you start trying to monitor your HAProxy backends via an external monitoring tool, you see the problem: "CRITICAL" alerts during normal operation! After all, a `DOWN` host is considered a _problem_ in 99.9% of HAProxy usecases. But with Patroni, it's expected behaviour, which is not ideal.
@ -23,7 +23,7 @@ So what can we do?
HAProxy, since at least version 1.5, supports [a feature called `agent-check`](https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#5.2-agent-check). In short, this "enable[s] an auxiliary agent check which is run independently of a regular health check". The `agent-check` will connect to a specific port on either the backend host or another target, and will modify the backend status based on the response, which must be one of the common HAProxy keyworks (eg. `MAINT` or `READY`). HAProxy, since at least version 1.5, supports [a feature called `agent-check`](https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#5.2-agent-check). In short, this "enable[s] an auxiliary agent check which is run independently of a regular health check". The `agent-check` will connect to a specific port on either the backend host or another target, and will modify the backend status based on the response, which must be one of the common HAProxy keyworks (eg. `MAINT` or `READY`).
So how does this help us? Well, if we had some way to obtain Patroni's `master`/`replica` status for each host, we could, instead of having the `replica` machines marked `DOWN`, put them into `MAINT` mode instead. This provides cleanliness for monitoring purposes while still letting us use the typical Patroni HAProxy configuration, with just minimal modifications to the HAProxy configuation and deploying an additional daemon on the Patroni hosts. So how does this help us? Well, if we had some way to obtain Patroni's `primary`/`replica` status for each host, we could, instead of having the `replica` machines marked `DOWN`, put them into `MAINT` mode instead. This provides cleanliness for monitoring purposes while still letting us use the typical Patroni HAProxy configuration, with just minimal modifications to the HAProxy configuation and deploying an additional daemon on the Patroni hosts.
### The Code - Python 3 daemon ### The Code - Python 3 daemon
@ -38,7 +38,7 @@ Here is the code - I'm sure it can be improved significantly but it works for me
``` ```
#!/usr/bin/env python3 #!/usr/bin/env python3
# Simple agent check for HAProxy to determine Patroni master/replica status # Simple agent check for HAProxy to determine Patroni primary/replica status
import socket, requests import socket, requests
@ -75,8 +75,8 @@ while True:
conn, addr = sock.accept() conn, addr = sock.accept()
state = getstate() state = getstate()
# Set our response based on the state; only `master` should be READY in read-write mode # Set our response based on the state; only `primary` should be READY in read-write mode
if state == 'master': if state == 'primary':
data = b'READY\n' data = b'READY\n'
else: else:
data = b'MAINT\n' data = b'MAINT\n'
@ -124,7 +124,7 @@ Now finally, configure your HAProxy backend to use the agent check. Here's my (l
backend mast-pgX_psql_readwrite backend mast-pgX_psql_readwrite
mode tcp mode tcp
option tcpka option tcpka
option httpchk OPTIONS /master option httpchk OPTIONS /primary
http-check expect status 200 http-check expect status 200
server mast-pg1 mast-pg1:5432 resolvers nsX resolve-prefer ipv4 maxconn 100 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008 server mast-pg1 mast-pg1:5432 resolvers nsX resolve-prefer ipv4 maxconn 100 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008
server mast-pg2 mast-pg2:5432 resolvers nsX resolve-prefer ipv4 maxconn 100 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008 server mast-pg2 mast-pg2:5432 resolvers nsX resolve-prefer ipv4 maxconn 100 check agent-check agent-port 5555 inter 1s fall 2 rise 2 on-marked-down shutdown-sessions port 8008
@ -138,3 +138,5 @@ And here it is in action:
### Conclusion ### Conclusion
I hope that this provides some help to those who want to use Patroni fronted by HAProxy but don't want `DOWN` backends all the time! And of course, I'm open to suggestions for improvement or questions - just send me an email! I hope that this provides some help to those who want to use Patroni fronted by HAProxy but don't want `DOWN` backends all the time! And of course, I'm open to suggestions for improvement or questions - just send me an email!
UPDATE 2024-12-01: Updated various instances of `master` to `primary` to reflect changes in Patroni since this post was originally written. Thanks to [Gary T. Giesen][https://fosstodon.org/@ggiesen] for pointing these out!