Notifier: Add event queueing and auto-retries on failed notifications

[ben-willmore]

chiadog tried to notify me using pushover while the network was briefly down. I think this brought down the whole thread, leading to multiple incorrect harvester errors. I have just wrapped it in try..except for now but I suppose the right solution is to queue up any undelivered messages and try again after a delay.

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ben/chiadog/src/chia_log/log_consumer.py", line 72, in _consume_loop
    self._notify_subscribers(log_line)
  File "/home/ben/chiadog/src/chia_log/log_consumer.py", line 48, in _notify_subscribers
    subscriber.consume_logs(logs)
  File "/home/ben/chiadog/src/chia_log/log_handler.py", line 30, in consume_logs
    self._notify_manager.process_events(events)
  File "/home/ben/chiadog/src/notifier/notify_manager.py", line 47, in process_events
    notifier.send_events_to_user(events)
  File "/home/ben/chiadog/src/notifier/pushover_notifier.py", line 26, in send_events_to_user
    conn.request(
  File "/usr/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/usr/lib/python3.8/http/client.py", line 1417, in connect
    super().connect()
  File "/usr/lib/python3.8/http/client.py", line 921, in connect
    self.sock = self._create_connection(
  File "/usr/lib/python3.8/socket.py", line 787, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.8/socket.py", line 918, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
[2021-04-22 22:28:12] [ WARNING] --- Your harvester appears to be offline! No events for the past 584 seconds. (
keep_alive_monitor.py:71)
[2021-04-22 22:33:12] [ WARNING] --- Your harvester appears to be offline! No events for the past 885 seconds. (
keep_alive_monitor.py:71)
[2021-04-22 22:38:13] [ WARNING] --- Your harvester appears to be offline! No events for the past 1186 seconds. 
(keep_alive_monitor.py:71)
[2021-04-22 22:43:14] [ WARNING] --- Your harvester appears to be offline! No events for the past 1487 seconds. 
(keep_alive_monitor.py:71)

[martomi]

Yep, I've added handling for the uncaught exceptions yesterday to dev. Queueing and retry is yet to be implemented though.

If someone want to implement queueing, I'd suggest doing so in the base class. The notifier manager will then queue up the events for each notifier and the notifiers can process the queues with auto-retries in their own threads.