nedbat
Branch coverage of try/except
Is your feature request related to a problem? Please describe. Consider the following code:
def f(x):
try:
y = 1/x
except ZeroDivisionError:
y = 0
return y
def test_exception_branch_coverage():
f(0)
test_exception_branch_coverage()
Here f(0) clearly misses a path: the case where 1/x did not raise an exception isn't covered. However Coverage.py (version 4.5.4) reports 100% statement and branch coverage.
Describe the solution you'd like A try/except is like a branch. Consider the following:
def g(x):
try:
y = 1/x
except ZeroDivisionError:
y = 0
else:
pass # this line wasn't executed, but coverage doesn't report it as missed
return y
g(0)
The else represents a branch target that was never taken. Coverage could know about the branch intrinsic in the try/except/else itself: whether an exception was raised or not.
Describe alternatives you've considered
The else block does not always represent a missed branch:
def h(x):
try:
return 1/x
except ZeroDivisionError:
return 0
else:
pass # never executed, but this does not imply a missed branch ...
h(3)
h(0)
Instead of just looking at whether the whole try threw an exception, one could consider each line individually:
def i(x):
try:
a = 1/(x+1)
b = 1/(x+2)
return a + b
except ZeroDivisionError:
return 0
i(0)
i(-1)
At first, it appears that all branches were covered by the above. i(0) executes the entire try block and throws no exception. i(-1) throws an exception and executes the except block.
However, both a = ... and b = ... could raise exceptions and only one of them was tested.
Keeping track of each line like this is tracking too much: in real code, not every line can raise interesting exceptions and those that don't shouldn't be considered missed branches:
def j(x):
try:
y = x[0]
return y + 1 # this line never raised an IndexError, but that's ok
except IndexError:
return None
j([])
j([1])
The return y + 1 line cannot raise an IndexError and therefore isn't a missed branch.
Additional context
It's not clear whether coverage tools should consider these "branches", or whether they should even report this kind of thing. One would expect that most of the time, statement coverage will catch a try/except that misses one of these cases, and it's only in the situation where there's a one-line try where later code is not conditional on the exception happening/not happening.
Maybe if added behaviours like this should be hidden behind flags for those that want more pedantic coverage metrics.
With thanks to @edk0 who originally pointed this out and for comments.
I've been using the following hack to experiment with this feature without having to implement it:
import sys
try_stack = []
with open(sys.argv[1], 'r') as f:
lines = [*f, '']
sys.stdout = open(sys.argv[1], 'w')
last_block_stmt = None
jump = False
for i, line in enumerate(lines):
line = line.rstrip('\n')
bare_line = line.lstrip(' ')
indent_level = len(line) - len(bare_line)
append = []
if ':' in bare_line:
block_stmt = bare_line.partition(':')[0].split(' ')[0]
else:
block_stmt = None
if block_stmt == 'try':
jump = False
elif block_stmt == 'except':
if not try_stack or try_stack[-1] < indent_level:
try_stack.append(indent_level)
if bare_line and block_stmt != 'except':
# we might need to close things
while try_stack and indent_level <= try_stack[-1]:
try_level = try_stack.pop(-1)
if jump:
continue
if block_stmt == 'else' and indent_level == try_level:
append.append(f"{' ' * indent_level} assert True")
break
print(f"{' ' * try_level}else:")
print(f"{' ' * try_level} assert True")
if block_stmt is None and last_block_stmt == 'try':
jump = bare_line.split(' ')[0] in ('return', 'continue', 'break')
if i < len(lines) - 1:
print(line)
for a in append:
print(a)
last_block_stmt = block_stmt or last_block_stmt
While obviously far from perfect, so far it's at least told me that the feature as requested doesn't have false positives on my code.
Thanks for the thought-provoking issue, and @edk0 for the impressive hack to try it out :)
I think just as branch coverage can shine light on the "missing implicit else" in if-statements, we can try doing the same for try-blocks. I have to get 5.0 out the door before I can experiment with it, though.
I have a case where the lines are reported as "missing" but are indeed covered, with branching on. Line 11 to 14 is from the if statement to print('3'). As you can see, all 1, 2, and 3 were printed.
tests/module.py
class Creds:
def __init__(self, store):
self.store = store
def func(self):
try:
1/0
except:
print('1')
if self.store:
print('2')
self.store.unlock()
print('3')
raise
tests/test_coverage.py
from . import module
import mock
import unittest
class TestCoverage(unittest.TestCase):
def test_coverage(self):
store = mock.Mock()
creds = module.Creds(store)
with self.assertRaises(Exception):
creds.func()
py.test tests/test_coverage.py --cov=tests.module -s --cov-branch
tests/test_coverage.py 1
2
3
.
----------- coverage: platform linux, python 3.8.5-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
-------------------------------------------------------------
tests/module.py 13 0 2 1 93% 11->14
py.test tests/test_coverage.py --cov=tests.module -s --cov-branch
tests/test_coverage.py 1
2
3
.
----------- coverage: platform linux, python 3.8.5-final-0 -----------
Name Stmts Miss Cover Missing
-----------------------------------------------
tests/module.py 13 0 100%
py.test tests/test_coverage.py --cov=tests.module -s --cov-branch
tests/test_coverage.py 1 2 3 . ----------- coverage: platform linux, python 3.8.5-final-0 ----------- Name Stmts Miss Branch BrPart Cover Missing ------------------------------------------------------------- tests/module.py 13 0 2 1 93% 11->14py.test tests/test_coverage.py --cov=tests.module -s --cov-branch
tests/test_coverage.py 1 2 3 . ----------- coverage: platform linux, python 3.8.5-final-0 ----------- Name Stmts Miss Cover Missing ----------------------------------------------- tests/module.py 13 0 100%
@postmasters Is that an intermittent problem, or do you know what caused the two different outputs?
