apluslms
Teacher's email notification on submission errors should be sent only when the grading fails to complete after automatic retries
When a student's submission can not be sent to the grader due to some error, the teacher receives an error email notification. However, with the automatic retries of incomplete grading jobs (#1035 and #1060), the notification should be only sent if the grading can not be automatically completed after retries. The teacher should only receive email if he/she should really manually check the situation of the submission.
Old description in Finnish:
Juha S. muistutti, että A+ lähettää (tai pitäisi lähettää) emailia opettajalle, kun tehtävän lataus tai arviointiin lähettäminen epäonnistuvat (kun yhteys pätkii tms.). Jos palautus arvioidaan automaattisesti uudelleen hetken kuluttua (#1035), niin Juha opettajana ei halua turhaan email-ilmoituksia, jos hänen ei tarvitse käsin itse reagoida mitenkään.
Huomioita:
- automaattinen regrade saisi reagoida nopeasti, kun peruskurssilla normaali arviointi valmistuu n. 2 minuutissa. Joillain kursseilla arviointi kestää kauemmin -> pitääkö tehtävän tai kurssin asetuksissa määritellä, kauanko arvioinnin pitäisi kestää? (oletuksena yleinen peruskurssien lyhyt aika, 5 min?)
- jos automaattinen uudelleenarviointi ei myöskään toimi, Juha haluaisi ilmoituksen, jotta osaa tutkia tarkemmin tehtävää.
- Juha haluaa keskustella sposti-ilmoituksen yksityiskohdista joskus sopivaan aikaan
@PasiSa was pondering:
Tuota pitää vähän miettiä. Kuinka pitkän yrittelyn jälkeen halutaan lähettää mailia? Ja koska jumiutuneita tehtäviä voi siinä vaiheessa olla useita, niin varmaan halutaan yksi maili per kurssi, tms.?
When could the email be sent? After how long time? Should multiple submission errors be combined in one email?
Note: Juha S. wanted to discuss such details before anything is decided.
I guess the main thing is that I’d love to hear the details once there is some kind of plan of what will get implemented. Details such as these:
- Does exercise type (and expected grading speed) affect how/when/whether auto-resubmits are made?
- Does it affect how/when/whether notifications are sent?
- Can teachers customize the above on a course-by-course basis and/or by exercise type if they don’t like the defaults?
Above, it says that "peruskurssilla normaali arviointi valmistuu n. 2 minuutissa". That’s a pretty pessimistic estimate, especially now that our graders (for Scala) are faster than a couple years ago. But yeah, I suppose 2 mins might be something like an upper limit for how long a regular O1 exercise should take in grading.
Above, it says that "peruskurssilla normaali arviointi valmistuu n. 2 minuutissa". That’s a pretty pessimistic estimate, especially now that our graders (for Scala) are faster than a couple years ago. But yeah, I suppose 2 mins might be something like an upper limit for how long a regular O1 exercise should take in grading.
Reflecting on this quickly, it seems to me that there are at least five different kinds of assignments that might(?) be treated differently.
- Programming assignments that are auto-graded "quickly" (under 1 min nearly 100% of the time). This would be most assignments in O1 and Y1, for example.
- Programming assignments that are expected to take a long time in resource-intensive grading. (I don’t have any of these myself.)
- Manually graded assignments (where the only point is to get the submission in).
- Instantaneously auto-gradable assignments, such as multiple-choice questionnaires.
- ”Assignments” that are feedback forms. (E.g. Jutut forms).
This is so closely related to the retry mechanics that it should be checked by Seppo. (assigned)
At this time, the expected grading time for assignments does not affect how soon/often it will be retried. I tried to implement a solution which would create an average grading time per assignment, and retry only (but soonish) after that time has passed. However, this proved to be harder than I initially thought to implement efficiently (due to Django running on multiple threads, etc.), so it was postponed to a later date as we wanted to fix the urgent issue with the same assignment being tried over an over, blocking others from being retried.
As discussed in the last A+ weekly, this could be implemented using a periodic (celery) task in the same manner as we do with pending submissions. Instead of sending emails immediately when a submission fails, we would just save the necessary details into database. Then the task would periodically check if there are jobs failed due to errors, and send an error email for teachers as a "digest" instead of spamming constantly in case of issues. We might not even need a separate model for this, if we can get the necessary data from submissions (those already marked as failed) and pending submissions models (db tables).
Some things to consider:
- How often to check for failed submissions? Right after retrying submissions would probably be ideal (could these even be combined into one task?).
- Should we also react to a growing number of pending submissions by sending a "heads up digest" even before the first of these actually fail after the max number of retries (this may take at least an hour or two, depending on timeout settings)?
- What is the data (essential for the teacher) that should be added to the digest?
- Should we also send the error digests to A+ administrators (in addition to normal grading service monitoring)?