Jayant Parashar

Data processing

3

Which script is creating the processed data in pkl format? as used in data_utils.py file: CODEX_PREDICTIONS_PATH = 'processed_data/codex_predictions.pkl' I am trying to use your code as a baseline, please help.

In both the Trainers, Basic, and FSDP, there is an underlying pattern of GPU memory not being freed. Allocation keeps increasing in steps while utilization remains roughly constant.  Does...