telegram-bot-api
telegram-bot-api copied to clipboard
tdlib
Console errors, API unavailability
Errors are being observed in the console. During these errors, the response from curl is Сurl error Symfony\Component\HttpClient\Exception\TimeoutException (Idle timeout reached for "http://127.0.0.1:8081/bot***/getChatMemberCount".). Sometimes it goes away by itself, but today the errors went on until I killed the process and the API did not work for 3 hours, it is not very convenient, there is not even a parameter for the application to close itself during such errors. Error example:
------- Stack Backtrace -------
/usr/local/bin/telegram-bot-api(+0x14a8c01)[0x64b870929c01]
/usr/local/bin/telegram-bot-api(+0x14a8f15)[0x64b870929f15]
/usr/local/bin/telegram-bot-api(+0x17b5e6)[0x64b86f5fc5e6]
/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7c0e45e45330]
/lib/x86_64-linux-gnu/libc.so.6(epoll_wait+0x52)[0x7c0e45f2a042]
/usr/local/bin/telegram-bot-api(+0x14adf71)[0x64b87092ef71]
/usr/local/bin/telegram-bot-api(+0x30e122)[0x64b86f78f122]
/usr/local/bin/telegram-bot-api(+0x306a61)[0x64b86f787a61]
/usr/local/bin/telegram-bot-api(+0x178f20)[0x64b86f5f9f20]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7c0e45e2a1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7c0e45e2a28b]
/usr/local/bin/telegram-bot-api(+0x17abe5)[0x64b86f5fbbe5]
-------------------------------
You seem to use a very old Bot API version.
Nevertheless, presence of the traces means that your operating system hanged up in the call to epoll_wait. You need to check that CPU and RAM weren't exhausted on the server.
You seem to use a very old Bot API version.
Nevertheless, presence of the traces means that your operating system hanged up in the call to
epoll_wait. You need to check that CPU and RAM weren't exhausted on the server.
telegram@mafia:~$ telegram-bot-api --version
Bot API 9.0
There are enough resources, I have 2 servers running on port 8081 and 8082, 8082 - worked without problems, 8081 - wrote an error and did not respond to http requests
I just got an error from curl "Сurl error Symfony\Component\HttpClient\Exception\TimeoutException (Idle timeout reached for "http://127.0.0.1:8081/bot***/editMessageText".)". There was no Stack Backtrace, it was like this in the logs at the same time:
[ 2][t12][1749033505.026620864][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.190718%
[ 2][t12][1749033506.025826454][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.046652%
[ 2][t12][1749033507.026228427][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.936194%
[ 2][t12][1749033508.025997638][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.848423%
[ 2][t12][1749033509.025893211][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.241467%
[ 2][t12][1749033510.026321411][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.205259%
[ 2][t12][1749033511.025705814][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.324737%
[ 2][t12][1749033512.025748491][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.412698%
[ 2][t12][1749033513.026068925][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.482370%
[ 2][t12][1749033514.025476217][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.436548%
[ 2][t12][1749033515.025969743][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.700422%
[ 2][t12][1749033517.025444746][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.532447%
[ 2][t12][1749033518.025734663][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.589361%
[ 2][t12][1749033519.025592803][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.634245%
[ 2][t12][1749033520.025790214][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.762431%
[ 2][t 1][1749033520.726402044][UpdatesManager.cpp:456][#6261664674][!Timeout] Filling gap in QTS from 552637459 to 552637462-552637464 by running getDifference. Receive no updates from session with auth key 12335658319609676851 for 3.661044 seconds, active for 5917.302263 seconds and having 5012 received updates
[ 2][t12][1749033521.025575876][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.702703%
[ 2][t12][1749033522.025707483][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.752626%
[ 2][t12][1749033523.025960445][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.662441%
[ 2][t12][1749033524.025996208][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.705750%
[ 2][t12][1749033525.026108026][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.739726%
[ 2][t12][1749033526.025758266][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.638255%
[ 2][t12][1749033527.025855541][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.704987%
[ 2][t12][1749033528.026234388][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.752626%
[ 2][t12][1749033529.025236368][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.663285%
[ 2][t12][1749033530.025735855][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.590090%
[ 2][t12][1749033531.026050090][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.632911%
[ 2][t12][1749033532.026264905][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.533784%
[ 2][t12][1749033533.026424407][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.607751%
[ 2][t12][1749033534.025798559][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.537266%
[ 2][t12][1749033535.025845766][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.476780%
[ 2][t12][1749033536.026140928][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.436548%
[ 2][t12][1749033537.025389909][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.197802%
[ 2][t12][1749033538.025769948][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.464661%
[ 2][t12][1749033539.025944948][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.538877%
The timeout in curl is 10 seconds.
There were no issues on the Bot API side. You can increase log level to 4 to see whether the request was received by the Bot API server or enable verbose logs for cURL.
Right now, timeout errors started happening on one of the two servers until I did a kill -9 telegram-bot-api.
One of the curl errors, they are all of the same type, with a timeout of 10 seconds:
App\Exceptions\CurlException: Сurl error Symfony\Component\HttpClient\Exception\TimeoutException (Idle timeout reached for "http://127.0.0.1:8082/bot***/getChatMemberCount".)
START: 1749626919.7743; END: 1749626929.778; DIFF: 10.00362610817
Console log telegram-bot-api:
[ 2][t12][1749626848.569614171][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.875225%
[ 2][t12][1749626849.568868875][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.893939%
[ 2][t12][1749626850.568885803][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.907969%
[ 2][t12][1749626851.568873882][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.919677%
[ 2][t12][1749626852.569135665][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.928375%
[ 2][t 1][1749626852.722080945][UpdatesManager.cpp:456][#6261664674][!Timeout] Filling gap in PTS from 759798842 to 759798845(-1)-759798845 updateDeleteMessages { messages = vector[1] { 74904759 } pts = 759798845 pts_count = 1 } by running getDifference. Receive no updates from session with auth key *** for 8.087209 seconds, active for 9242.702944 seconds and having 5085 received updates
[ 2][t12][1749626853.569197654][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.853412%
[ 2][t12][1749626854.569652080][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.167630%
[ 2][t 1][1749626854.857659578][UpdatesManager.cpp:456][#6261664674][!Timeout] Filling gap in QTS from 376658361 to 376658363-376658366 by running getDifference. Receive no updates from session with auth key *** for 10.222788 seconds, active for 9244.838523 seconds and having 5085 received updates
[ 2][t12][1749626855.569478988][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.151218%
[ 2][t12][1749626856.569571733][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.023040%
[ 2][t12][1749626857.569359302][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.123894%
[ 2][t12][1749626858.568849086][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.114943%
[ 2][t12][1749626859.568949460][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.360877%
[ 2][t12][1749626860.569608449][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.459313%
[ 2][t12][1749626861.569148063][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.405825%
[ 2][t12][1749626862.569221496][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.250352%
[ 2][t12][1749626863.569321393][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.329114%
[ 2][t12][1749626864.569392681][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.431611%
[ 2][t12][1749626865.568840980][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.025316%
[ 2][t12][1749626866.568854570][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.167630%
[ 2][t12][1749626867.569509506][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.896933%
[ 2][t12][1749626868.568890333][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.023609%
[ 2][t12][1749626869.569468021][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.021734%
[ 2][t12][1749626870.568998336][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.682086%
[ 2][t12][1749626871.569551944][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.440922%
[ 2][t12][1749626872.569475889][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.514196%
[ 2][t12][1749626873.568830490][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.570827%
[ 2][t12][1749626874.569655895][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.938941%
[ 2][t12][1749626875.569349050][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 4.276986%
[ 2][t12][1749626876.569227457][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 4.067797%
[ 2][t12][1749626877.568713426][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.627131%
[ 2][t12][1749626878.569401502][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.299492%
[ 2][t12][1749626879.569334030][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.044827%
[ 2][t12][1749626880.569254159][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.152810%
[ 2][t12][1749626881.570240259][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.858893%
[ 2][t12][1749626882.569504261][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.881331%
[ 2][t12][1749626883.569377660][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.026599%
[ 2][t12][1749626884.569409132][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.139640%
[ 2][t12][1749626885.569029569][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.025829%
[ 2][t12][1749626886.569480419][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.939954%
[ 2][t12][1749626887.568887472][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.034760%
[ 2][t12][1749626888.568985939][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.033406%
[ 2][t12][1749626889.569748640][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.157360%
[ 2][t12][1749626890.569259643][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.029884%
[ 2][t 1][1749626891.029226541][UpdatesManager.cpp:456][#6261664674][!Timeout] Filling gap in QTS from 376658398 to 376658400-376658400 by running getDifference. Receive no updates from session with auth key *** for 46.394355 seconds, active for 9281.010090 seconds and having 5085 received updates
[ 2][t12][1749626891.569794178][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.129278%
[ 2][t12][1749626892.570674896][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.219980%
[ 2][t12][1749626893.569720268][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.356902%
[ 2][t12][1749626894.569447994][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.165283%
[ 2][t12][1749626895.569344043][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.145787%
[ 2][t12][1749626896.568957090][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.019635%
[ 2][t12][1749626897.568860292][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.120677%
[ 2][t12][1749626898.569655656][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.022756%
[ 2][t12][1749626899.569053411][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.021661%
[ 2][t12][1749626900.568857908][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.024035%
[ 2][t12][1749626901.569203615][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.024747%
[ 2][t12][1749626902.569342613][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.024291%
[ 2][t12][1749626903.569025754][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.023919%
[ 2][t 1][1749626903.797386407][UpdatesManager.cpp:456][#6261664674][!Timeout] Filling gap in QTS from 376658408 to 376658410-376658411 by running getDifference. Receive no updates from session with auth key *** for 59.162515 seconds, active for 9293.778250 seconds and having 5085 received updates
[ 2][t12][1749626904.568793773][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.025316%
[ 2][t12][1749626905.569375514][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.025316%
[ 2][t12][1749626906.569629907][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.024035%
[ 2][t12][1749626907.569216966][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.911723%
[ 2][t12][1749626908.569614410][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.821401%
[ 2][t12][1749626909.569431304][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.822785%
[ 2][t12][1749626910.569434404][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.026171%
[ 2][t 1][1749626911.369740724][UpdatesManager.cpp:456][#6261664674][!Timeout] Filling gap in QTS from 376658419 to 376658421-376658421 by running getDifference. Receive no updates from session with auth key *** for 66.734869 seconds, active for 9301.350604 seconds and having 5085 received updates
[ 2][t12][1749626911.569080114][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.169982%
[ 2][t12][1749626912.569142818][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.151899%
[ 2][t12][1749626913.569692373][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.249719%
[ 2][t12][1749626914.569432973][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.126044%
[ 2][t12][1749626915.569031476][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.363867%
[ 2][t12][1749626916.569697380][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.314647%
[ 2][t12][1749626917.569421768][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.149858%
[ 2][t12][1749626918.568741083][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.137233%
[ 2][t12][1749626919.569241046][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.127121%
[ 2][t12][1749626920.569291353][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.026342%
[ 2][t12][1749626921.568632602][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.027883%
[ 2][t12][1749626922.569509267][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.882012%
[ 2][t12][1749626923.569461584][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.902346%
[ 2][t12][1749626924.568916559][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.915493%
[ 2][t12][1749626925.568716764][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.028912%
[ 2][t12][1749626926.568720817][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 1.860465%
[ 2][t12][1749626927.569357872][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.173913%
[ 2][t12][1749626928.569646358][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.027883%
[ 2][t 2][1749626928.693092346][Session.cpp:665][#6261664674][!Session:2:main#12] Session connection with 1 pending requests was closed: [Error : 0 : No answer from auth key *** for [Query:[id:2220032000][tl:0x08736a09][msg_id:0x68492fe557961394]] for 75.3s after ping sent at 1749626917.564081 and answered at 1749626917.564082 with the current server time 1749626928.665393] Session:2:main#12::Connect::TCP::[149.154.167.51:443] to DcId{2} from [***:53140]
[ 2][t 2][1749626929.196360826][SessionProxy.cpp:262][#6261664674][!SessionProxy:2:main#12] Have tmp_auth_key ***: NoAuth
[ 2][t 2][1749626929.196448326][Session.cpp:1415][#6261664674][!Session:2:main#12] Update auth key in session_id *** to ***
[ 2][t 1][1749626929.297143220][TempAuthKeyWatchdog.h:128][#6261664674][!TempAuthKeyWatchdog] Start auth_dropTempAuthKeys except keys {2120638941273776706, 2529706587145041496, 3761084336909002063, 4468650837720633601, 5899518542445107908, 6382811497072953036, 7412167887884394781, 7669687816575455754, 7695925836112344102, 7777511097595532702, 9092311080322624390, -9153604947895617361, -8502854612054793113, -7948194719740557722, -7270550429935802449, -6671170777273487833, -6025753948814126599, -4108621568712152407, -1319130040892238128, -817045633012706906}
[ 2][t12][1749626929.569143772][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.703464%
[ 2][t 2][1749626929.750020503][SessionProxy.cpp:262][#6261664674][!SessionProxy:2:main#12] Have tmp_auth_key ***: OK
[ 2][t12][1749626930.569681406][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 2.736949%
[ 2][t12][1749626931.568786859][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.641882%
[ 2][t12][1749626932.569453001][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.375527%
[ 2][t12][1749626933.569127082][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.328509%
[ 2][t 1][1749626934.440516471][TempAuthKeyWatchdog.h:128][#6261664674][!TempAuthKeyWatchdog] Start auth_dropTempAuthKeys except keys {2120638941273776706, 2529706587145041496, 3761084336909002063, 4468650837720633601, 5899518542445107908, 6382811497072953036, 7412167887884394781, 7669687816575455754, 7695925836112344102, 7777511097595532702, 9092311080322624390, -9153604947895617361, -8502854612054793113, -7948194719740557722, -7270550429935802449, -6671170777273487833, -6025753948814126599, -4108621568712152407, -1319130040892238128, -817045633012706906}
[ 2][t12][1749626934.569037199][Stats.cpp:37][!RunOnSchedulerWorker] CPU usage: 3.164557%
There were no issues with the Bot API server at that time. You can increase request timeout to 100 seconds to avoid the error.