cmd410
Handling of connection error exceptions
I'm running OrigamiBot as part of a multi thread server process. From within the telegram communication thread, I run bot.start() (which starts more Origamibot threads in the background), then I use that thread for forwarding messages from the system to the bot asynchronously (i.e.: from the main thread, not triggered by incoming bot messages; I don't use the asyncio concept). This all works quite well.
Things seem to become a bit ugly when for example the network is down, or the server is not reachable for any other reason.
This leads to a flurry of messages like these:
2024-03-11 19:23:03,398 | bot.py:1443 | PID 2282 | ERROR | Exception while polling
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1017, in _validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7b90347264d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /BOTID:BOTTOKEN/getUpdates (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7b90347264d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/bot.py", line 1441, in _listen_loop
updates = self.get_updates()
File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/bot.py", line 314, in get_updates
updates = get_updates(
File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/api_request.py", line 115, in get_updates
updates = request(
File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/api_request.py", line 91, in request
response = requests.post(
File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /BOTID:BOTTOKEN/getUpdates (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7b90347264d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))
(Bot data has been replaced with BOTID and BOTTOKEN in above log).
I would like to somewhat gracefully catch these, log them as a single concise message to syslog ("Telegram Server Connection lost"), and then try to recover (i.e. the system should continue to operate, and silently try to reconnect to the telegram server in the background).
How is this best done? Is it even possible with Origamibot? Is the above behavior a bug? Or am I doing something wrong?
My code looks about as this (edited for brevity):
bot = Bot(bot_token)
bot.add_listener(MessageListener(bot))
bot.add_commands(BotsCommands(bot))
bot.start()
...
while(True):
bot.send_message(chat_id, "testmessage") # chat_id stored globally, the chat we want to send info to
# sleep for a while
Funny: The bot seems to reconnect, and commands I type in the telegram channel do get through to the system. But messages my systems send to the telegram channel via the bot (see above example) are lost.
Maybe you have already found a solution yourself. If not, here is my guess.
First of all, you posted the log of the update polling running in the background. This is logged, but should run properly again as soon as the internet connection is re-established. (you wrote this as the funny postscript)
As far as I can see, this library has no integrated error handling for sending a message. However, you should be able to integrate this yourself quite easily.
bot = Bot(bot_token)
bot.add_listener(MessageListener(bot))
bot.add_commands(BotsCommands(bot))
bot.start()
...
while True: # your default task loop
while True: # loop to retry after exception
try:
bot.send_message(chat_id, "testmessage") # chat_id stored globally, the chat we want to send info to
except IOError: # may be more precise, e.g. requests.exceptions.ConnectionError
logging.exception("Telegram Server Connection lost")
time.sleep(5) # wait 5 seconds before retry
continue
else:
break # break inner loop if send_message was successfully
# sleep for a while
But my example has one big problem. It keeps trying to send the message until it succeeds. This could end in an endless loop.
I hope this helps you.
I did not find a solution, I ran into more issues like this and decided to switch to a different bot framework that offers better error handling. Thanks for trying to help!