coveragepy
coveragepy copied to clipboard
nedbat
Performance drop with using coverage testing starlette websocket
Describe the bug Performance drop with using coverage testing starlette websocket.
To Reproduce How can we reproduce the problem? Please be specific. Don't link to a failing CI job. Answer the questions below:
-
platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0 - Package versions
>>> coverage.__version__
'6.4.1'
>>> fastapi.__version__
'0.78.0'
>>> starlette.__version__
'0.19.1'
- code: https://pastebin.com/F2iXUM4E.
- just running this code with
pytestwill in average less than 1s. But withpytest --covorcoverage run -m pytestincreases its execution time 10 times (10s+ avg)
Expected behavior Coverage do not affect execution time.
Additional context
As far as I can see problem with async to sync in starlette.testclient.TestClient and coverage.
Also coverage do not affect execution time with concurrency=greenlet (because it wont check any code that way) but execution time raises even more with concurrency=thread.greenlet
I don't see the same results you do. I had to add a line to your file: from starlette.testclient import TestClient
Then:
% pytest bug1411.py
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: /private/tmp
plugins: anyio-3.6.1, cov-3.0.0
collected 1 item
bug1411.py . [100%]
======================================================================= 1 passed in 0.23s ========================================================================
% coverage run -m pytest bug1411.py
====================================================================== test session starts =======================================================================
platform darwin -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: /private/tmp
plugins: anyio-3.6.1, cov-3.0.0
collected 1 item
bug1411.py . [100%]
======================================================================= 1 passed in 0.30s ========================================================================
The times are not the same, but they are similar, and within reason. Can you show the results of coverage debug sys? Are you using the C tracer?
% coverage debug sys
-- sys -------------------------------------------------------
coverage_version: 6.4.1
coverage_module: /home/randomneo/.virtualenvs/default/lib/python3.10/site-packages/coverage/__init__.py
tracer: -none-
CTracer: available
plugins.file_tracers: -none-
plugins.configurers: -none-
plugins.context_switchers: -none-
configs_attempted: .coveragerc
setup.cfg
tox.ini
pyproject.toml
configs_read: -none-
config_file: None
config_contents: -none-
data_file: -none-
python: 3.10.5 (main, Jun 6 2022, 18:49:26) [GCC 12.1.0]
platform: Linux-5.18.7-arch1-1-x86_64-with-glibc2.35
implementation: CPython
executable: /home/randomneo/.virtualenvs/default/bin/python
def_encoding: utf-8
fs_encoding: utf-8
pid: 2569
cwd: /home/randomneo/projects/personal/fastapi_websocket_slow
path: /home/randomneo/.virtualenvs/default/bin
/usr/lib/python310.zip
/usr/lib/python3.10
/usr/lib/python3.10/lib-dynload
/home/randomneo/.virtualenvs/default/lib/python3.10/site-packages
environment: HOME = /home/randomneo
command_line: /home/randomneo/.virtualenvs/default/bin/coverage debug sys
sqlite3_version: 2.6.0
sqlite3_sqlite_version: 3.38.5
sqlite3_temp_store: 0
sqlite3_compile_options: ATOMIC_INTRINSICS=1, COMPILER=gcc-12.1.0, DEFAULT_AUTOVACUUM,
DEFAULT_CACHE_SIZE=-2000, DEFAULT_FILE_FORMAT=4,
DEFAULT_JOURNAL_SIZE_LIMIT=-1, DEFAULT_MMAP_SIZE=0, DEFAULT_PAGE_SIZE=4096,
DEFAULT_PCACHE_INITSZ=20, DEFAULT_RECURSIVE_TRIGGERS,
DEFAULT_SECTOR_SIZE=4096, DEFAULT_SYNCHRONOUS=2,
DEFAULT_WAL_AUTOCHECKPOINT=1000, DEFAULT_WAL_SYNCHRONOUS=2,
DEFAULT_WORKER_THREADS=0, ENABLE_COLUMN_METADATA, ENABLE_DBSTAT_VTAB,
ENABLE_FTS3, ENABLE_FTS3_TOKENIZER, ENABLE_FTS4, ENABLE_FTS5,
ENABLE_MATH_FUNCTIONS, ENABLE_RTREE, ENABLE_STMTVTAB, ENABLE_UNLOCK_NOTIFY,
HAVE_ISNAN, MALLOC_SOFT_LIMIT=1024, MAX_ATTACHED=10, MAX_COLUMN=2000,
MAX_COMPOUND_SELECT=500, MAX_DEFAULT_PAGE_SIZE=8192, MAX_EXPR_DEPTH=10000,
MAX_FUNCTION_ARG=127, MAX_LENGTH=1000000000, MAX_LIKE_PATTERN_LENGTH=50000,
MAX_MMAP_SIZE=0x7fff0000, MAX_PAGE_COUNT=1073741823, MAX_PAGE_SIZE=65536,
MAX_SQL_LENGTH=1000000000, MAX_TRIGGER_DEPTH=1000,
MAX_VARIABLE_NUMBER=250000, MAX_VDBE_OP=250000000, MAX_WORKER_THREADS=8,
MUTEX_PTHREADS, SECURE_DELETE, SYSTEM_MALLOC, TEMP_STORE=1, THREADSAFE=1
➜ fastapi_websocket_slow pytest main.py (env: default)
========================== test session starts =============================
platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/randomneo/projects/personal/fastapi_websocket_slow, configfile: pytest.ini
plugins: cov-3.0.0, anyio-3.6.1
collected 1 item
main.py . [100%]
========================= 1 passed in 1.54s ========================================== =
➜ fastapi_websocket_slow pytest main.py --cov (env: default)
========================= test session starts ===========================================
platform linux -- Python 3.10.5, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/randomneo/projects/personal/fastapi_websocket_slow, configfile: pytest.ini
plugins: cov-3.0.0, anyio-3.6.1
collected 1 item
main.py . [100%]
---------- coverage: platform linux, python 3.10.5-final-0 -----------
Name Stmts Miss Cover
-----------------------------
main.py 23 0 100%
-----------------------------
TOTAL 23 0 100%
================================= 1 passed in 22.26s ===================================