Allow viewing of node logs via API/CLI #135

New Issue

2021-07-05T17:02:40-04:00

JoshuaBoniface commented

2021-07-05 17:02:40 -04:00

(Migrated from git.bonifacelabs.ca)

Similar to pvc vm log, allow pvc node log which should show the various logs (node daemon, API daemon, API worker daemon) in a similar fashion, i.e. less-view or tail -f-view. Implementation should be identical to the pvc vm log except that the source will likely be journalctl (if file logging is turned off).

Similar to `pvc vm log`, allow `pvc node log` which should show the various logs (node daemon, API daemon, API worker daemon) in a similar fashion, i.e. `less`-view or `tail -f`-view. Implementation should be identical to the `pvc vm log` except that the source will likely be `journalctl` (if file logging is turned off).

JoshuaBoniface commented

2021-07-06 00:29:15 -04:00

(Migrated from git.bonifacelabs.ca)

Would need to store each node's log lines in Zookeeper to prevent shenannegans, and probably update them every keepalive. This will get really hairy quick though, so, as nice as it might be, as-is I don't think this will work out.

That said, there might be a better way to do this with some sort of syslog forwarding. Perhaps using journalctl, to send all the logs to the primary which can then read them. But that is it's own kind of weird.

Might be better to avoid this despite it seeming nice and easy on the surface.

Example implementation of the diff for the basic function to read logs:

diff --git a/daemon-common/node.py b/daemon-common/node.py
index 82f43ea..4080f6f 100644
--- a/daemon-common/node.py
+++ b/daemon-common/node.py
@@ -182,6 +182,34 @@ def ready_node(zkhandler, node, wait=False):
     return True, retmsg
 
 
+def get_node_log(daemon='all', lines=1000, raw=False):
+    """
+    Return {lines} number of log lines from journalctl for {daemon}, optionally in
+    {raw} format without timestamps.
+    """
+
+    # Map of the possible daemon values to service unit names for journalctl
+    daemon_service_map = {
+        'all': ['pvcnoded.service', 'pvcapid.service', 'pvcapid-worker.service'],
+        'node': ['pvcnoded.service'],
+        'api': ['pvcapid.service', 'pvcapid-worker.service'],
+        'api-daemon': ['pvcapid.service'],
+        'api-worker': ['pvcapid-worker.service']
+    }
+
+    journalctl_service_string = '--unit=' + ' --unit='.join(daemon_service_map[daemon])
+    journalctl_lines_string = '--lines={}'.format(lines)
+    journalctl_output_string = '--output=cat' if raw else ''
+    journalctl_command = 'journalctl --all --full --no-pager {} {} {}'.format(
+        journalctl_service_string, journalctl_lines_string, journalctl_output_string
+    )
+
+    # Get the data from journalctl
+    retcode, stdout, stderr = common.run_os_command(journalctl_command)
+
+    return True if retcode == 0 else False, stdout
+
+
 def get_info(zkhandler, node):
     # Verify node is valid
     if not common.verifyNode(zkhandler, node):

Would need to store each node's log lines in Zookeeper to prevent shenannegans, and probably update them every keepalive. This will get really hairy quick though, so, as nice as it might be, as-is I don't think this will work out. That said, there might be a better way to do this with some sort of syslog forwarding. Perhaps using journalctl, to send all the logs to the primary which can then read them. But that is it's own kind of weird. Might be better to avoid this despite it seeming nice and easy on the surface. Example implementation of the diff for the basic function to read logs: ``` diff --git a/daemon-common/node.py b/daemon-common/node.py index 82f43ea..4080f6f 100644 --- a/daemon-common/node.py +++ b/daemon-common/node.py @@ -182,6 +182,34 @@ def ready_node(zkhandler, node, wait=False): return True, retmsg +def get_node_log(daemon='all', lines=1000, raw=False): + """ + Return {lines} number of log lines from journalctl for {daemon}, optionally in + {raw} format without timestamps. + """ + + # Map of the possible daemon values to service unit names for journalctl + daemon_service_map = { + 'all': ['pvcnoded.service', 'pvcapid.service', 'pvcapid-worker.service'], + 'node': ['pvcnoded.service'], + 'api': ['pvcapid.service', 'pvcapid-worker.service'], + 'api-daemon': ['pvcapid.service'], + 'api-worker': ['pvcapid-worker.service'] + } + + journalctl_service_string = '--unit=' + ' --unit='.join(daemon_service_map[daemon]) + journalctl_lines_string = '--lines={}'.format(lines) + journalctl_output_string = '--output=cat' if raw else '' + journalctl_command = 'journalctl --all --full --no-pager {} {} {}'.format( + journalctl_service_string, journalctl_lines_string, journalctl_output_string + ) + + # Get the data from journalctl + retcode, stdout, stderr = common.run_os_command(journalctl_command) + + return True if retcode == 0 else False, stdout + + def get_info(zkhandler, node): # Verify node is valid if not common.verifyNode(zkhandler, node): ```

JoshuaBoniface commented

2021-07-06 00:37:09 -04:00

(Migrated from git.bonifacelabs.ca)

https://serverfault.com/questions/758244/how-to-configure-systemd-journal-remote might provide a slightly more elegant way to move the logs around. I'd envision, if possible, some sort of full mesh among the coordinators so that all coordinators can receive all messages from all nodes and then store them in their own independent journal, which could then be dynamically called by whichever node is the primary at that moment using a similar method to the above. This would avoid storing arbitrary system logs in Zookeeper.

JoshuaBoniface commented

2021-07-06 01:04:36 -04:00

(Migrated from git.bonifacelabs.ca)

Journal forwarding doesn't seem very nice, since you can't really limit it to any sort of specific service/unit like journalctl can. Uploading to Zookeeper might be the better option but this need some more investigation of exactly how Zookeeper works, since unlike daemon logs which are bursty but generally quite unchanging, these are constantly updating and rotating.

Journal forwarding doesn't seem very nice, since you can't really limit it to any sort of specific service/unit like `journalctl` can. Uploading to Zookeeper might be the better option but this need some more investigation of exactly how Zookeeper works, since unlike daemon logs which are bursty but generally quite unchanging, these are constantly updating and rotating.

JoshuaBoniface commented

2021-07-18 20:50:42 -04:00

(Migrated from git.bonifacelabs.ca)

Ended up implementing this in a very similar way to VM console logs. The main downside is that this requires a lot more Zookeeper heap memory, so there is an adjustment in the PVC Ansible configuration to handle this. The option can be turned off in the node daemon configuration if desired.

Included in PVC 0.9.26

Ended up implementing this in a very similar way to VM console logs. The main downside is that this requires a lot more Zookeeper heap memory, so there is an adjustment in the PVC Ansible configuration to handle this. The option can be turned off in the node daemon configuration if desired. Included in PVC 0.9.26

JoshuaBoniface commented

2021-07-18 20:50:43 -04:00

(Migrated from git.bonifacelabs.ca)

closed

Sign in to join this conversation.