PyTd
PyTd copied to clipboard
Expensive logging calls during `convertValue` degrade large query performance
The method convertValue of DefaultDataTypeConverter contains the following line (L230-L231):
logger.trace(
"Converting \"%s\" to (%s, %s).", value, dataType, typeCode)
This results in many calls to logger.isEnabledFor, even when configureLogging=False in the application configuration.
For context: I am fetching ~80,000 rows with 30 columns each for a total of 2,400,000 type conversions. Even if each of these logging calls is very fast the time becomes significant given the volume of data. cProfile shows that ~36% of the time in convertValue was spent in util.py:41(trace).
Maybe a fix similar to the one used for issue #37 could work?