Joshua Boniface
b17b7bf22b
Add black magic to minimize ping losses
...
This particular arping interval/count, along with forcing it to run in
the foreground, seems to minimize the packet loss when the primary
coordinator transitions. Through extensive testing, this value results
in the, consistently, least amount of loss: 1-2 pings, at an 0.025s ping
interval, return "TTL exceeded", with no other loss, and only when the
node the test VM is on is the one switching to secondary state. No other
combination of values here, nor tweaks to other parts of the code, seem
able to reduce this further, therefore this is likely the best
configuration possible.
2019-12-19 18:57:32 -05:00
Joshua Boniface
2fe808f812
Make this even nicer
2019-12-19 13:29:15 -05:00
Joshua Boniface
7901ffb5e3
Handle the list of node_limits even better
2019-12-19 13:22:38 -05:00
Joshua Boniface
684499cce3
Better handle list elements when defining VM
2019-12-19 12:08:44 -05:00
Joshua Boniface
796fa4692f
Pass remaining required args to vm_define
2019-12-19 11:53:24 -05:00
Joshua Boniface
78089afb4e
Remove spurious argument to vm_define
2019-12-19 11:43:59 -05:00
Joshua Boniface
2d14fabe62
Don't throw exception while defining new VM
...
This would throw an exception since we tried to find the current host of
a VM that did not exist yet. If this happens, just get None.
2019-12-19 11:30:13 -05:00
Joshua Boniface
8c252aeecc
Implemented coordinated locked node transitions
...
The previous method was a "throw it in the sea"-type migration with some
(very arbitrary) sleep statements thrown in for good measure.
Reimplement this with some hard locking. During each phase of the
transition, the nodes acquire read/write shared locks to a Zookeeper key
so that they can tightly coordinate the actions of transferring each
part of the primary state between them. This is done in a subthread to
prevent strange blocking issues that were encountered, likely due to
business in the existing main thread.
2019-12-19 10:56:34 -05:00
Joshua Boniface
0841ddf8b0
Handle integrity errors in DNS aggregator
2019-12-19 10:45:06 -05:00
Joshua Boniface
98764f1edd
Clean up some aspects of node switchover
2019-12-18 21:39:40 -05:00
Joshua Boniface
23188199cb
Handle failing Patroni events more gracefully
2019-12-18 21:12:22 -05:00
Joshua Boniface
669a8fbacd
Clean up pycache files before (re)installing
2019-12-18 18:39:03 -05:00
Joshua Boniface
2b1b78622e
Fix invalid arping option
...
It made little difference and didn't error, but was incorrect.
2019-12-18 12:06:40 -05:00
Joshua Boniface
5eec3e99f8
Lower delay during build-and-deploy
2019-12-18 11:56:50 -05:00
Joshua Boniface
355e16e23a
Add missing dependencies
2019-12-18 11:56:22 -05:00
Joshua Boniface
364ab10673
Add slight delay when stopping the metadata API
2019-12-18 11:56:04 -05:00
Joshua Boniface
39c9f911cc
Increase arping interval to 0.2s
2019-12-15 14:55:34 -05:00
Joshua Boniface
2b5c134970
Add missing distutils dep
2019-12-15 13:53:22 -05:00
Joshua Boniface
686af31c08
Reduce arping interval to 0.1s
2019-12-15 12:30:45 -05:00
Joshua Boniface
0a94fac407
Fix bugs around passing master
...
Was not passing properly and getting stuck sometimes, so modify the
checking and route creation a bit to prevent it. Seems to work.
2019-12-15 00:08:18 -05:00
Joshua Boniface
8202d7e64b
Make output a little nicer
2019-12-14 16:56:33 -05:00
Joshua Boniface
292c2d9a44
Install newly moved schema file
2019-12-14 16:52:09 -05:00
Joshua Boniface
41479edc3d
Remove separated provisioner
2019-12-14 16:47:46 -05:00
Joshua Boniface
a2b30158f8
Move schema file
2019-12-14 16:47:32 -05:00
Joshua Boniface
b3e21a5bf8
Integrate metadata API into node daemon
2019-12-14 16:41:01 -05:00
Joshua Boniface
8c36e7618a
Modify node daemon to follow API
2019-12-14 14:13:26 -05:00
Joshua Boniface
0727a7f6ed
Move all provisioner API functionality into main
2019-12-14 14:12:55 -05:00
Joshua Boniface
45dbc0eef8
Merge branch 'provisioner' into 'master'
...
Implement Provisioner daemon
Closes #56 and #22
See merge request parallelvirtualcluster/pvc!1
2019-12-13 15:05:57 -05:00
Joshua Boniface
c0757405bc
Increase delay in deploy
2019-12-13 00:06:37 -05:00
Joshua Boniface
78f053d81f
Recreate network in aggregator if DNS changes
2019-12-13 00:03:47 -05:00
Joshua Boniface
0a8dd30a48
Restart dnsmasq when network details change
2019-12-12 23:51:22 -05:00
Joshua Boniface
6fa828e721
Don't stop the provisioner worker
...
It should probably just be running on all nodes all the time already,
but is started when a node first becomes primary.
2019-12-12 23:08:02 -05:00
Joshua Boniface
c1b6ce0ff7
Reorder starting clients
2019-12-12 23:03:34 -05:00
Joshua Boniface
b854d53fab
Add API management to node daemon
2019-12-12 22:59:07 -05:00
Joshua Boniface
22ae087836
Add examples to deb install
2019-12-12 22:52:12 -05:00
Joshua Boniface
f5fb741dad
Unify the APIs
2019-12-12 22:50:42 -05:00
Joshua Boniface
c91c9ae6d5
Fix formatting of node_limit when writing
2019-12-12 21:36:21 -05:00
Joshua Boniface
fff44ab44e
Make node limit output nicer
2019-12-12 20:52:56 -05:00
Joshua Boniface
708de48065
Finish the provisioner and metadata server
2019-12-12 19:41:23 -05:00
Joshua Boniface
88924497c2
Define with profile and allow options
2019-12-11 17:05:05 -05:00
Joshua Boniface
88a181b20d
Allow metadata API in nft rules
2019-12-11 17:04:29 -05:00
Joshua Boniface
52127f2938
Add provisioner profile to VM information
2019-12-11 17:04:16 -05:00
Joshua Boniface
a95e7d1f76
Add profile from provisioner to data stored in ZK
2019-12-11 16:49:11 -05:00
Joshua Boniface
4236493912
Add DHCP in example debootstrap script
2019-12-10 23:22:12 -05:00
Joshua Boniface
911965c1f3
Revamp ordering and spacing of libvirt XML
2019-12-10 22:31:35 -05:00
Joshua Boniface
1377c15948
Fix text in postinst
2019-12-09 13:33:47 -05:00
Joshua Boniface
867634cea7
Fix missing argument
2019-12-09 11:31:56 -05:00
Joshua Boniface
6c4eb7fc7b
Use distutils to convert text to bools
2019-12-09 11:12:12 -05:00
Joshua Boniface
2027809bfd
Add service unit for provisioner worker
2019-12-09 10:43:53 -05:00
Joshua Boniface
57e8fba602
Add provisioner to Debian packages
2019-12-09 10:40:27 -05:00
Joshua Boniface