modern-data-warehouse-dataops icon indicating copy to clipboard operation
modern-data-warehouse-dataops copied to clipboard

Review/Increase the parameterization of the notebooks

Open LiliamLeme opened this issue 1 year ago • 3 comments

Review notebooks and increase the parameterization. Follow overview of the NB made by Sreedhar ( https://github.com/Azure-Samples/modern-data-warehouse-dataops/pull/862 ).

This task most likely will be break in more tasks:

  1. investigate if there a point where it could use better the parametrization of the NBs Example: DDL table definition could be done in separate file.

2)Add the log library to the notebooks. 2.1 Investigate and define how the logs will be stored for this repo and which libraries we should use for this purpose. LIke will be using python telemetry using Open Telemetry or something different/ 2.2 Implement the changes in the NBs.

  1. Evaluate if any performance improvements/techniques can be applied.

  2. If necessary, alter the documentation accordingly.

LiliamLeme avatar Nov 22 '24 09:11 LiliamLeme

@LiliamLeme For your review/refrence, here is current implementation #862 . These use multiple ways of parameterization: %%configure, notebook parameters tag and an external config file application.cfg which uses Python's configparser module to read it.

sreedhar-guda avatar Nov 22 '24 14:11 sreedhar-guda

thanks @sreedhar-guda I was thinking in your notebook implementation when I created. So I appreciate

LiliamLeme avatar Nov 27 '24 17:11 LiliamLeme

FYI @ydaponte I changed the description.

LiliamLeme avatar Dec 18 '24 12:12 LiliamLeme