Fix bug with usage of getattr to get the test name

[gregbell26]

In the event that an exception is raised during the class setup (setUpClass) using the JSONTestRunner with the built-in unittest module, the test runner crashes and fails to write to results.json.

Minimal example:

import unittest
from gradescope_utils.autograder_utils.json_test_runner import JSONTestRunner

class Test(unittest.TestCase):
    @classmethod
    def setUpClass(cls) -> None:
        raise AssertionError("This shouldnt crash the entire platform")

    def test1(self):
        pass


if __name__ == "__main__":
    tests = unittest.defaultTestLoader.loadTestsFromTestCase(Test)
    resultsPath = "./results.json"

    with open(resultsPath, 'w') as w:
        runner = JSONTestRunner(w)
        runner.run(tests)

Stack trace

  Traceback (most recent call last):
    File "/usr/lib/python3.10/unittest/suite.py", line 166, in _handleClassSetUp
      setUpClass()
    File "/home/gjbell/git/gradescope-utils/main.py", line 7, in setUpClass
      raise AssertionError("This shouldnt crash the entire platform")
  AssertionError: This shouldnt crash the entire platform
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "/home/gjbell/git/gradescope-utils/main.py", line 19, in <module>
      runner.run(tests)
    File "/home/gjbell/git/gradescope-utils/gradescope_utils/autograder_utils/json_test_runner.py", line 196, in run
      test(result)
    File "/usr/lib/python3.10/unittest/suite.py", line 84, in __call__
      return self.run(*args, **kwds)
    File "/usr/lib/python3.10/unittest/suite.py", line 114, in run
      self._handleClassSetUp(test, result)
    File "/usr/lib/python3.10/unittest/suite.py", line 176, in _handleClassSetUp
      self._createClassOrModuleLevelException(result, e,
    File "/usr/lib/python3.10/unittest/suite.py", line 236, in _createClassOrModuleLevelException
      self._addClassOrModuleLevelException(result, exc, errorName, info)
    File "/usr/lib/python3.10/unittest/suite.py", line 246, in _addClassOrModuleLevelException
      result.addError(error, sys.exc_info())
    File "/home/gjbell/git/gradescope-utils/gradescope_utils/autograder_utils/json_test_runner.py", line 136, in addError
      self.processResult(test, err)
    File "/home/gjbell/git/gradescope-utils/gradescope_utils/autograder_utils/json_test_runner.py", line 123, in processResult
      if self.getLeaderboardData(test)[0]:
    File "/home/gjbell/git/gradescope-utils/gradescope_utils/autograder_utils/json_test_runner.py", line 51, in getLeaderboardData
      column_name = getattr(getattr(test, test._testMethodName), '__leaderboard_column__', None)
  AttributeError: '_ErrorHolder' object has no attribute '_testMethodName'

results.json is not created, which causes the Gradescope autograder to fail when deployed.

Ideally, this should show the test failure in results.json similarly to how the built in TextTestRunner behaves:

E
======================================================================
ERROR: setUpClass (__main__.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/gjbell/git/gradescope-utils/main.py", line 7, in setUpClass
    raise AssertionError("This shouldnt crash the entire platform")
AssertionError: This shouldnt crash the entire platform

----------------------------------------------------------------------
Ran 0 tests in 0.000s

FAILED (errors=1)

The reason that this is occurring is because the how the getters called from buildResult and buildLeaderboardEntry are assuming that test will always has _testMethodName. And that is true, when test is an instance of unittest.TestCase. However, that is not the case in the error condition that I explained above (when test is actually an instance of _ErrorHolder).

This pull request fixes that error by checking to see if _testMethodName actually exists in test before trying to access it.

After those fixes, we see that we get similar output to TextTestRunner in results.json.

{
    "tests": [
        {
            "name": "setUpClass (__main__.Test)",
            "status": "failed",
            "output": "Test Failed: This shouldnt crash the entire platform\n"
        }
    ],
    "leaderboard": [],
    "execution_time": "0.00",
    "score": 0.0
}