Leakage of NUMBER option out of docstring scope

[MatthewFlamm]

• [x] a detailed description of the bug or problem you are having
• [x] output of pip list from the virtual environment you are using
• [x] pytest and operating system versions
• [x] minimal example if possible

This was originally noted in #9892, but there is a more targeted MRE here.

MRE

Running pytest --doctest-modules test_x.py with x=1, 2 or 3 results in the following pattern of failing and passing tests. The test inside test_wo_number incorrectly passes only on the cases (test_1.py and test_2.py) where a failing test that uses NUMBER (test_w_off_by_2_number) also exists in the same module. This leakage does not occur when there are tests that pass that utilize NUMBER (test_3.py)

# test_1.py
def test_w_number():
    """
    Correctly Passes

    Examples
    --------
    >>> 1.  # doctest: +NUMBER
    0.
    """

def test_w_off_by_2_number():
    """
    Correctly Fails

    Examples
    --------
    >>> 0.  # doctest: +NUMBER
    2.
    """

def test_wo_number():
    """
    Incorrectly Passes

    Examples
    --------
    >>> 1.
    0.
    """

# test_2.py
def test_w_off_by_2_number():
    """
    Correctly Fails

    Examples
    --------
    >>> 0.  # doctest: +NUMBER
    2.
    """


def test_wo_number():
    """
    Incorrectly Passes

    Examples
    --------
    >>> 1.
    0.
    """

# test_3.py
def test_w_number():
    """
    Correctly Passes

    Examples
    --------
    >>> 1.  # doctest: +NUMBER
    0.
    """

def test_wo_number():
    """
    Correctly Fails

    Examples
    --------
    >>> 1.
    0.
    """

pip list

Package    Version
---------- -------
attrs      21.4.0
iniconfig  1.1.1
numpy      1.22.3
packaging  21.3
pip        22.0.4
pluggy     1.0.0
py         1.11.0
pyparsing  3.0.8
pytest     7.1.2
setuptools 41.2.0
tomli      2.0.1

system version

CentOS 7.9