Joshua Boniface
20ae2186f9
Run VM state actions in a thread
...
Prevents blocking the main thread(s) while a VM is changing state. In
particular, this caused some issues with nodes not responding to
cancellation/reversal of a flush/ready state until the previous
migration was finished, which could cause issues. This entire subset of
actions is now threaded and so can run on its own in the background.
2019-12-26 11:08:16 -05:00
Joshua Boniface
b3483fa810
Add explicit returns from flush/ready threads
2019-12-26 11:08:00 -05:00
Joshua Boniface
47cf0a8006
Ensure migration out occurs
2019-12-25 21:11:02 -05:00
Joshua Boniface
77db36a891
Ensure migration out occurs
2019-12-25 21:02:46 -05:00
Joshua Boniface
9a39d739e8
Ensure we empty of flush_thread
2019-12-25 20:29:17 -05:00
Joshua Boniface
e8d8fb161b
Add wait functionality to API domain-state
2019-12-25 20:24:12 -05:00
Joshua Boniface
b1c19a21ba
Move int cast to helper try block
2019-12-25 19:31:51 -05:00
Joshua Boniface
1f77b382ef
Support providing console log lines from API
2019-12-25 19:13:53 -05:00
Joshua Boniface
8a0a278fe9
Cast numbers to strings for length
2019-12-25 13:47:46 -05:00
Joshua Boniface
1f6f486b3b
Fix broken unwanted change to pvc.py
2019-12-25 13:35:31 -05:00
Joshua Boniface
b92c1876d6
Make sure we return True from init function
2019-12-25 12:18:26 -05:00
Joshua Boniface
9b81a02805
Disable doc endpoint to avoid flask_swagger dep
2019-12-24 11:19:06 -05:00
Joshua Boniface
88f1be8cc9
Add API reference link
2019-12-24 10:53:58 -05:00
Joshua Boniface
e4c96ee43d
Add flask-restful dependency
2019-12-24 10:48:15 -05:00
Joshua Boniface
f48d1e6d40
Add documentation for API_Doc endpoint
2019-12-24 08:59:17 -05:00
Joshua Boniface
ebcf0cee43
Add doc endpoint
2019-12-23 23:11:55 -05:00
Joshua Boniface
2dec61ce88
Fix the host in the output file
2019-12-23 21:22:09 -05:00
Joshua Boniface
914b3d84e1
Fix URL hosts
2019-12-23 21:17:30 -05:00
Joshua Boniface
142e1f186d
Fix in-app host URL
2019-12-23 21:15:26 -05:00
Joshua Boniface
d75a9749ac
Regenerate host value
2019-12-23 21:14:59 -05:00
Joshua Boniface
398fb2dfe9
Set URL and information in local swagger.json
2019-12-23 21:09:58 -05:00
Joshua Boniface
60ffe534a1
Add proper (local) URL to swagger docs
2019-12-23 21:06:08 -05:00
Joshua Boniface
05f08a714a
Update link to API reference document
2019-12-23 20:59:46 -05:00
Joshua Boniface
14b811e3c7
Rename API reference page
2019-12-23 20:59:24 -05:00
Joshua Boniface
1d59453252
API documentation page cleanup
2019-12-23 20:58:20 -05:00
Joshua Boniface
57127d2fa8
Move gen-doc helper script
2019-12-23 20:51:44 -05:00
Joshua Boniface
fd4f033a04
Update return from initialize
2019-12-23 20:51:18 -05:00
Joshua Boniface
9076208a92
Fix missing code end block
2019-12-23 20:47:01 -05:00
Joshua Boniface
e6a8ed0734
Replace in-markdown API docs with Swagger
2019-12-23 20:45:30 -05:00
Joshua Boniface
a6683d5b17
Rebuild API using Flask-RESTful and Swagger docs
2019-12-23 20:43:20 -05:00
Joshua Boniface
91cb542e37
Make sure ACLs are name-unique in both directions
2019-12-22 22:33:26 -05:00
Joshua Boniface
66078223a4
Fix several bugs
2019-12-22 22:28:34 -05:00
Joshua Boniface
de8c79755b
Add documentation about the provisioner API
2019-12-19 23:44:10 -05:00
Joshua Boniface
eecc07b731
Depend daemons on systemd
...
Numerous parts of PVC call systemctl commands or otherwise require a
functioning systemd-based system. Make the dependencies explicitly
reflect this.
2019-12-19 19:04:25 -05:00
Joshua Boniface
a66b834ae4
Fix several small bugs
2019-12-19 18:58:53 -05:00
Joshua Boniface
b17b7bf22b
Add black magic to minimize ping losses
...
This particular arping interval/count, along with forcing it to run in
the foreground, seems to minimize the packet loss when the primary
coordinator transitions. Through extensive testing, this value results
in the, consistently, least amount of loss: 1-2 pings, at an 0.025s ping
interval, return "TTL exceeded", with no other loss, and only when the
node the test VM is on is the one switching to secondary state. No other
combination of values here, nor tweaks to other parts of the code, seem
able to reduce this further, therefore this is likely the best
configuration possible.
2019-12-19 18:57:32 -05:00
Joshua Boniface
2fe808f812
Make this even nicer
2019-12-19 13:29:15 -05:00
Joshua Boniface
7901ffb5e3
Handle the list of node_limits even better
2019-12-19 13:22:38 -05:00
Joshua Boniface
684499cce3
Better handle list elements when defining VM
2019-12-19 12:08:44 -05:00
Joshua Boniface
796fa4692f
Pass remaining required args to vm_define
2019-12-19 11:53:24 -05:00
Joshua Boniface
78089afb4e
Remove spurious argument to vm_define
2019-12-19 11:43:59 -05:00
Joshua Boniface
2d14fabe62
Don't throw exception while defining new VM
...
This would throw an exception since we tried to find the current host of
a VM that did not exist yet. If this happens, just get None.
2019-12-19 11:30:13 -05:00
Joshua Boniface
8c252aeecc
Implemented coordinated locked node transitions
...
The previous method was a "throw it in the sea"-type migration with some
(very arbitrary) sleep statements thrown in for good measure.
Reimplement this with some hard locking. During each phase of the
transition, the nodes acquire read/write shared locks to a Zookeeper key
so that they can tightly coordinate the actions of transferring each
part of the primary state between them. This is done in a subthread to
prevent strange blocking issues that were encountered, likely due to
business in the existing main thread.
2019-12-19 10:56:34 -05:00
Joshua Boniface
0841ddf8b0
Handle integrity errors in DNS aggregator
2019-12-19 10:45:06 -05:00
Joshua Boniface
98764f1edd
Clean up some aspects of node switchover
2019-12-18 21:39:40 -05:00
Joshua Boniface
23188199cb
Handle failing Patroni events more gracefully
2019-12-18 21:12:22 -05:00
Joshua Boniface
669a8fbacd
Clean up pycache files before (re)installing
2019-12-18 18:39:03 -05:00
Joshua Boniface
2b1b78622e
Fix invalid arping option
...
It made little difference and didn't error, but was incorrect.
2019-12-18 12:06:40 -05:00
Joshua Boniface
5eec3e99f8
Lower delay during build-and-deploy
2019-12-18 11:56:50 -05:00
Joshua Boniface
355e16e23a
Add missing dependencies
2019-12-18 11:56:22 -05:00
Joshua Boniface