Move to Zookeeper as API worker broker #165

New Issue

joshuaboniface · 2023-11-04T03:26:52-04:00

joshuaboniface commented

2023-11-04 03:26:52 -04:00

The Redis broker has served well, but has some big limitations:

No replication between coordinators. While replication is possible, it is statically configured and not multi-master (dealbreaker).
The lack of replication makes each node separate, causing issues.
Prevents using the Celery worker for anything that isn't on "this node" i.e. the primary coordinator.

Investigate the experimental Zookeeper broker instead. If this works, it can open up several interesting avenues:

Provisioner can work against any arbitrary node, and can continue to report status if the primary node changes.
Other tasks can be delegated to the API worker, e.g. Ceph OSD operations or other similar node-specific operations.

The Redis broker has served well, but has some big limitations: 1. No replication between coordinators. While replication is possible, it is statically configured and not multi-master (dealbreaker). 2. The lack of replication makes each node separate, causing issues. 3. Prevents using the Celery worker for anything that isn't on "this node" i.e. the primary coordinator. Investigate the experimental Zookeeper broker instead. If this works, it can open up several interesting avenues: 1. Provisioner can work against any arbitrary node, and can continue to report status if the primary node changes. 2. Other tasks can be delegated to the API worker, e.g. Ceph OSD operations or other similar node-specific operations.

joshuaboniface added the

API

improvement

labels 2023-11-04 03:27:11 -04:00

joshuaboniface commented

2023-11-04 03:47:33 -04:00

Well, that's impossible, because the Redis devs are lazy. Support every random-ass thing under the sun but not ETCd or Zookeeper (i.e. sensible KV stores).

I guess the only option is to hack in Redis multi-master using Dynomite.

https://fatihmalakci.com/how-to-setup-redis-master-master-replication/

Well, that's impossible, because the Redis devs are lazy. Support every random-ass thing under the sun but not ETCd or Zookeeper (i.e. sensible KV stores). I guess the only option is to hack in Redis multi-master using Dynomite. https://fatihmalakci.com/how-to-setup-redis-master-master-replication/

joshuaboniface commented

2023-11-04 12:56:55 -04:00

Was able to get this to work with Zookeeper as the message transport and PostgreSQL as the results backend, and it seems to work fine. I was just confused as Zookeeper is only usable as a Message Broker while I needed something else as the Results Backend, which PostgreSQL works fine as.

joshuaboniface closed this issue

2023-11-04 12:56:55 -04:00

joshuaboniface referenced this issue

2023-11-04 15:15:41 -04:00

Move to Celery jobs for per-node tasks #166

joshuaboniface commented

2023-11-04 15:29:59 -04:00

Code written and merged. Can use a custom kwarg in the function calls (likely always run_on in combination with the @celery.task(..., routing_key='run_on'), to set the worker delegation for tasks that require it. All nodes will have workers, started along side the node daemon, so these tasks can be properly delegated to them leveraging the above queues.

This necessitated a couple config changes:

In pvcapid.yaml, remove the queue: subsection as it is not required.
In pvcapid.yaml, adjust the default PostgreSQL connection to be the cluster floating IP address instead of localhost.
Logic for this in https://git.bonifacelabs.ca/parallelvirtualcluster/pvc-ansible

Code written and merged. Can use a custom kwarg in the function calls (likely always `run_on` in combination with the `@celery.task(..., routing_key='run_on')`, to set the worker delegation for tasks that require it. All nodes will have workers, started along side the node daemon, so these tasks can be properly delegated to them leveraging the above queues. This necessitated a couple config changes: 1. In `pvcapid.yaml`, remove the `queue:` subsection as it is not required. 2. In `pvcapid.yaml`, adjust the default PostgreSQL connection to be the cluster floating IP address instead of localhost. 3. Logic for this in https://git.bonifacelabs.ca/parallelvirtualcluster/pvc-ansible

Sign in to join this conversation.