video2commons icon indicating copy to clipboard operation
video2commons copied to clipboard

Allow admins to restart tasks and added worker stats to the dashboard

Open Amdrel opened this issue 2 months ago • 7 comments

Description

Phabricator Ticket: https://phabricator.wikimedia.org/T410732

  • The restart button works only for tasks created by the current logged-in user. For those like Amdrel and me who have admin privileges in the UI, we should be able to restart failed tasks created by other users
  • We don't see on which encoding instance the task was running, it's a guessing game right now, it should be displayed somewhere

For normal users, they have no idea of the current load. The frontend should display, and refresh, how many tasks are currently processed / waiting before their own, so that they can see when the system is just busy and not stuck. Sometimes users cancel theirs tasks because they think it's stuck, while there is simply someone that created many tasks before them.

Changes

  • Users in the sudoers list in Redis (maintainers) can now manage other users' tasks (relogin required for old sessions)
  • Worker statistics are now displayed on the dashboard and are updated ASAP
    • Currently displayed values are as follows:
      • Capacity: The amount of busy workers and the amount of available workers
      • Utilization: A percentage calculated from the used capacity
      • Pending: The amount of tasks that aren't being processed by any workers
    • There is a new cron job as well that updates the stats every 5 minutes in order to query the current capacity in the event workers go up or down. However, as it is currently configured in backend.pp, this currently runs on all worker instances although it doesn't need to. Ideally this job would only run on one instance.
    • Stats will not display until after the first cron job run is done
    • If stat writes fail for any reason then they shouldn't impact the workers
    • A distributed lock is used to ensure updates to stats by workers are atomic
  • Running tasks now display the instance that jobs are running on. Old tasks created prior to this change will display 'N/A'.

Amdrel avatar Dec 04 '25 19:12 Amdrel