capitalone
Presets for ProfilerOptions/Profiler
Is your feature request related to a problem? Please describe.
The Profiler can be slow with all options enabled. Most users won't need all of the settings to be enabled. Changing the settings one by one takes quite a bit of time and you need to study the docs quite a bit.
Describe the outcome you'd like:
It would be nice if ProfilerOptions or the Profiler itself had the possibility of quickly adjusting multiple settings by setting a preset. The current default behavior could have the option "all" or "complete". It might look like this:
profiler = dp.Profiler(data, preset="complete")
or
opts = dp.ProfilerOptions(preset="complete")
Some other presets could be:
-
"standard"- where some niche features that are performance intensive are deactivated -
"numeric_stats_disabled"- self explainatory -
"data_types"- where only data types of columns are infered (this is actually what I want right now)
I'm sure you can think of others. The down side of this feature is that many presets would be opinionated.
The implementation would be straight forward and the source code e.g. profiler_presets.py would nicely list all the settings that are changed by the preset.
Thanks, @pietz, for the well-documented idea -- thanks for putting in the issue. Will be in touch if we have any questions. Cheers!
I think we can focus on getting the data_types done first as that is the most desired.
@lovleen3112 I think we can focus on the options preset:
opts = dp.ProfilerOptions(preset="complete") as that could lead to easy add of an option preset during Profiling subsequently
Once this is merged, we will have to do a validate on running the Profiler with the options to ensure there aren't any report failures since many options have been disabled.
Tested all three preset options (i.e. complete, data_types, and numeric_stats_disabled) and no failures to generate reports using profiler.report()
@pietz thoughts?
If no additional thoughts regarding this, @pietz, I'll go ahead and close. Thx!