Gracefully handle node primary transitions in provisioner #69

New Issue

2020-01-08T21:04:15-05:00

JoshuaBoniface commented

2020-01-08 21:04:15 -05:00

(Migrated from git.bonifacelabs.ca)

As-is, transitioning the primary node while a provisioner task is running can have undefined results. Investigate this in more detail and correct in an optimal way.

JoshuaBoniface commented

2020-01-08 21:04:40 -05:00

(Migrated from git.bonifacelabs.ca)

mentioned in issue #68

JoshuaBoniface commented

2020-01-08 21:11:44 -05:00

(Migrated from git.bonifacelabs.ca)

The only real option I can think of is to block node primary transition if a job is running in the provisioner. This is not ideal, since there could still be queued jobs that would then be discarded without implementing some sort of Redis mirroring or a non-Redis datastore (not currently supported unfortunately, PostgreSQL support for Celery was dropped) and could delay needed transitions for an arbitrary amount of time, but would still be a valid solution to the immediate problem.

Other options might be possible that require more work, so feedback requested.

The only real option I can think of is to block node primary transition if a job is running in the provisioner. This is not ideal, since there could still be queued jobs that would then be discarded without implementing some sort of Redis mirroring or a non-Redis datastore (not currently supported unfortunately, PostgreSQL support for Celery was dropped) and could delay needed transitions for an arbitrary amount of time, but would still be a valid solution to the immediate problem. Other options might be possible that require more work, so feedback requested.

JoshuaBoniface commented

2020-01-12 16:04:22 -05:00

(Migrated from git.bonifacelabs.ca)

mentioned in commit 74ff8b41ea

mentioned in commit 74ff8b41ea79f4088c1f793a008a80ccc75471d9

JoshuaBoniface commented

2020-01-12 18:27:42 -05:00

(Migrated from git.bonifacelabs.ca)

Implemented a warning in the CLI for primary/secondary commands. Otherwise, this can remain as-is for the foreseeable future.

$ pvc node primary hv1
Using cluster "bl" - Host: "10.100.0.252:7370"  Scheme: "http"  Prefix: "/api/v1"

Note: There are currently 1 active or queued provisioner jobs on the current primary node.
      These jobs will continue executing, but status will not be visible until the current
      node returns to primary state.

Setting node hv1 in primary router mode.

Implemented a warning in the CLI for primary/secondary commands. Otherwise, this can remain as-is for the foreseeable future. ``` $ pvc node primary hv1 Using cluster "bl" - Host: "10.100.0.252:7370" Scheme: "http" Prefix: "/api/v1" Note: There are currently 1 active or queued provisioner jobs on the current primary node. These jobs will continue executing, but status will not be visible until the current node returns to primary state. Setting node hv1 in primary router mode. ```

JoshuaBoniface commented

2020-01-12 18:30:44 -05:00

(Migrated from git.bonifacelabs.ca)

closed

Sign in to join this conversation.