teltek
Retry a recording status update if it fails
Galicaster will try once and only once to update the recording status of an event when it starts recording. (example below). If this fails, it doesn't retry.
Galicaster should retry the recording status update periodically (for example on the long counter) until it succeeds, or the recording ends.
galicaster 2019-03-05 08:59:59,946 INFO scheduler/scheduler Start time for recording e9c05e74-05e4-4773-934f-dcff729fef85, duration 3300000 ms
galicaster 2019-03-05 08:59:59,969 INFO recorder/service Recording (current status: preview)
galicaster 2019-03-05 08:59:59,970 INFO recorder/service Recording to MP e9c05e74-05e4-4773-934f-dcff729fef85
galicaster 2019-03-05 08:59:59,984 INFO opencast/service Sending state capturing for the scheduled MP (u'e9c05e74-05e4-4773-934f-dcff729fef85', u'EEE3091F LG01 09:00 14', datetime.datetime(
2019, 3, 5, 7, 0))
galicaster 2019-03-05 09:00:00,026 INFO mediapackage/repository Temporary data saved to /usr/share/galicaster-repository/rectemp/info.json
galicaster 2019-03-05 09:00:02,947 DEBUG scheduler/heartbeat galicaster-notify-short in 15
galicaster 2019-03-05 09:00:16,950 ERROR opencast/client call error in http://media.uct.ac.za/capture-admin/recordings/e9c05e74-05e4-4773-934f-dcff729fef85, status code {502}: 502 Proxy Error
galicaster 2019-03-05 09:00:16,950 WARNING opencast/service Problems to connect to opencast server: Error in Opencast client
galicaster 2019-03-05 09:00:17,946 DEBUG scheduler/heartbeat galicaster-notify-short in 15
galicaster 2019-03-05 09:00:17,947 INFO opencast/service Init opencast client
galicaster 2019-03-05 09:00:17,962 INFO opencast/service Be careful using profiles and opencast scheduler
galicaster 2019-03-05 09:00:17,962 INFO opencast/service Updating series from server
galicaster 2019-03-05 09:00:32,947 DEBUG scheduler/heartbeat galicaster-notify-short in 15
galicaster 2019-03-05 09:00:32,947 INFO opencast/service Set status capturing to server
galicaster 2019-03-05 09:00:32,947 INFO opencast/client Sending state capturing
The #614 improvement is good. We do see cases though where Opencast is up but a status update fails for some transient problem (e.g. network glitch, timeout on the request because Opencast is very busy or something).
So Galicaster needs some code that checks periodically if opencast is up, and if a recording is in progress, and if a status update for the recording has not been successfully sent, then send a recording status update.
Hi @smarquard This improvement should work as you specify.
Even if only that particular endpoint is failing, Opencast is set to "down" and, after the current changes, we will retry to set the state, which will set Opencast as "down" if it fails. As there is already a periodic check for whether Opencast is up, Galicaster will retry to set the state.
The only issue I can see here with the current change is that if Opencast is working, but that endpoint is failing (maybe because of a high load), it may not help if all Galicasters are retrying every long beat.
