Raising error in unittest's setUpClass causes autograder breaking exception

[alexadkins]

Describe the bug Raising an Assertion Error in unittest's setUpClass method causes exception in jsontestrunner's getLeaderboardData

To Reproduce

import unittest
from gradescope_utils.autograder_utils.decorators import (
    weight,
    number,
    leaderboard,
)

class TestAutograder(unittest.TestCase):
    @classmethod
    def setUpClass(cls) -> None:
        raise AssertionError("compile for all tests failed")
        return super().setUpClass()
    
    @number(1.1)
    @weight(1)
    def test_01(self):
        "Test 01"
        pass

Expected behavior Raised error would have populated results.json for all test cases of that class.

Instead, received following error:

The autograder produced the following output:

Traceback (most recent call last):
  File "/usr/lib/python3.10/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
  File "/autograder/source/tests/test.py", line 14, in setUpClass
    raise AssertionError("compile for all tests failed")
AssertionError: compile for all tests failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/autograder/source/run_tests.py", line 8, in <module>
    JSONTestRunner(visibility='visible',stdout_visibility='visible').run(suite)
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 196, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.10/unittest/suite.py", line 114, in run
    self._handleClassSetUp(test, result)
  File "/usr/lib/python3.10/unittest/suite.py", line 176, in _handleClassSetUp
    self._createClassOrModuleLevelException(result, e,
  File "/usr/lib/python3.10/unittest/suite.py", line 236, in _createClassOrModuleLevelException
    self._addClassOrModuleLevelException(result, exc, errorName, info)
  File "/usr/lib/python3.10/unittest/suite.py", line 246, in _addClassOrModuleLevelException
    result.addError(error, sys.exc_info())
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 136, in addError
    self.processResult(test, err)
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 123, in processResult
    if self.getLeaderboardData(test)[0]:
  File "/usr/local/lib/python3.10/dist-packages/gradescope_utils/autograder_utils/json_test_runner.py", line 51, in getLeaderboardData
    column_name = getattr(getattr(test, test._testMethodName), '__leaderboard_column__', None)
AttributeError: '_ErrorHolder' object has no attribute '_testMethodName'

[gregbell26]

I fixed this issue in my fork (see #38).

GS doesn't seem to be responsive here for some goofy reason, but if you want to use my fork with your autograder you can add git+https://github.com/gregbell26/gradescope-utils to your requirements.txt instead of the official package.

Im also working on a complete rewrite of this package that will support the new features that the GS team has added to the platform.

[gregbell26]

I now have an official package that fixes this. https://pypi.org/project/HybridJSONTestRunner/