deepethogram icon indicating copy to clipboard operation
deepethogram copied to clipboard
verified publisher icon jbohnslav

Problem when training the flow generator

Open Specom opened this issue 1 year ago • 0 comments

Hey there, I am currently trying to train the flow generator in my DeepEthogram project (windows, anaconda, installed with pip, cudatoolkit=11.2 cudnn=8.1.0 , and tensorflow-gpu==2.10.0 ):

"(deepethogramg) C:\Users\Sander>python -m deepethogram [2024-05-22 11:50:20,333] INFO [deepethogram.gui.main.setup_gui_cfg:1268] CWD: C:\Users\Sander [2024-05-22 11:50:20,336] INFO [deepethogram.gui.main.setup_gui_cfg:1269] Configuration used: split: reload: true file: null train_val_test:

  • 0.8
  • 0.2
  • 0.0 compute: fp16: false num_workers: 8 batch_size: auto min_batch_size: 8 max_batch_size: 512 distributed: false gpu_id: 0 dali: false metrics_workers: 0 reload: overwrite_cfg: false latest: false notes: null log: level: info run: type: gui label_view_width: 31 control_arrow_jump: 31 vertical_arrow_jump: 3 cmap: deepethogram unlabeled_alpha: 0.1 prediction_opacity: 0.2

[2024-05-22 11:50:33,212] WARNING [deepethogram.projects.fix_config_paths:1264] Erroneous name of config file in the config file itself, changing... [2024-05-22 11:50:33,249] INFO [deepethogram.gui.main.initialize_project:1017] cwd: C:\Users\Sander [2024-05-22 11:50:33,250] INFO [deepethogram.projects.convert_config_paths_to_absolute:1135] cwd in absolute: C:\Users\Sander [2024-05-22 11:50:33,251] INFO [deepethogram.projects.convert_config_paths_to_absolute:1178] after absolute: {'class_names': ['background', 'walking', 'ghead', 'gfront', 'gabdomen', 'gback', 'gwing', 'flight', 'pushing'], 'config_file': 'C:\DeepEthogram\test_deepethogram\project_config.yaml', 'data_path': 'C:\DeepEthogram\test_deepethogram\DATA', 'labeler': None, 'model_path': 'C:\DeepEthogram\test_deepethogram\models', 'name': 'test', 'path': 'C:\DeepEthogram\test_deepethogram', 'pretrained_path': 'C:\DeepEthogram\test_deepethogram\models\pretrained_models'} [2024-05-22 11:50:33,251] INFO [deepethogram.gui.main.initialize_project:1019] cwd: C:\Users\Sander [2024-05-22 11:50:33,266] INFO [deepethogram.gui.main.initialize_project:1021] loaded project configuration: split: reload: true file: null train_val_test:

  • 0.8
  • 0.2
  • 0.0 compute: fp16: false num_workers: 8 batch_size: 32 min_batch_size: 8 max_batch_size: 512 distributed: false gpu_id: 0 dali: false metrics_workers: 0 reload: overwrite_cfg: false latest: false notes: null log: level: info run: type: gui model: null dir: C:\DeepEthogram\test_deepethogram\gui_logs\240522_115033 label_view_width: 31 control_arrow_jump: 31 vertical_arrow_jump: 3 cmap: deepethogram unlabeled_alpha: 0.1 prediction_opacity: 0.2 postprocessor: type: min_bout_per_behavior min_bout_length: 1 augs: LR: 0.5 UD: 0.0 brightness: 0.25 contrast: 0.1 crop_size: null degrees: 10 grayscale: 0.5 hue: 0.1 normalization: 'N': 324172800 mean:
    • 0.26870773349589994
    • 0.26870773349589994
    • 0.26870773349589994 std:
    • 0.23503533203484067
    • 0.23503533203484067
    • 0.23503533203484067 pad: null random_resize: false resize:
  • 224
  • 224 saturation: 0.1 project: class_names:
  • background
  • walking
  • ghead
  • gfront
  • gabdomen
  • gback
  • gwing
  • flight
  • pushing config_file: C:\DeepEthogram\test_deepethogram\project_config.yaml data_path: C:\DeepEthogram\test_deepethogram\DATA labeler: null model_path: C:\DeepEthogram\test_deepethogram\models name: test path: C:\DeepEthogram\test_deepethogram pretrained_path: C:\DeepEthogram\test_deepethogram\models\pretrained_models sequence: filter_length: 15 train: loss_weight_exp: 1.0

[2024-05-22 11:50:33,268] INFO [deepethogram.gui.main.initialize_project:1022] cwd: C:\DeepEthogram\test_deepethogram\gui_logs\240522_115033 [2024-05-22 11:50:33,281] INFO [deepethogram.gui.main.project_loaded_buttons:175] Number finalized labels: 0 [2024-05-22 11:50:33,588] INFO [deepethogram.gui.main.initialize_video:226] Record for loaded video: {'flow': None, 'label': None, 'output': None, 'rgb': 'C:\DeepEthogram\test_deepethogram\DATA\2024-05-14_A02_005_camera_01\2024-05-14_A02_005_camera_01.mp4', 'keypoint': None, 'key': '2024-05-14_A02_005_camera_01'} [2024-05-22 11:50:48,936] INFO [deepethogram.gui.main.get_selected_models:1136] {'flow_generator': {'no pretrained weights': None, '200221_115158_TinyMotionNet': 'C:\DeepEthogram\test_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt'}, 'feature_extractor': {'no pretrained weights': None, '200415_125824_hidden_two_stream_kinetics_degf': 'C:\DeepEthogram\test_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt'}, 'sequence': {'': None}} [2024-05-22 11:50:48,936] INFO [deepethogram.gui.main.flow_train:343] flow_train called with args: ['python', '-m', 'deepethogram.flow_generator.train', 'project.path=C:\DeepEthogram\test_deepethogram', 'reload.weights=C:\DeepEthogram\test_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt'] [2024-05-22 11:50:52,987] INFO [deepethogram.projects.convert_config_paths_to_absolute:1135] cwd in absolute: C:\DeepEthogram\test_deepethogram\gui_logs\240522_115033 [2024-05-22 11:50:52,988] INFO [deepethogram.projects.convert_config_paths_to_absolute:1178] after absolute: {'class_names': ['background', 'walking', 'ghead', 'gfront', 'gabdomen', 'gback', 'gwing', 'flight', 'pushing'], 'config_file': 'C:\DeepEthogram\test_deepethogram\project_config.yaml', 'data_path': 'C:\DeepEthogram\test_deepethogram\DATA', 'labeler': None, 'model_path': 'C:\DeepEthogram\test_deepethogram\models', 'name': 'test', 'path': 'C:\DeepEthogram\test_deepethogram', 'pretrained_path': 'C:\DeepEthogram\test_deepethogram\models\pretrained_models'} [2024-05-22 11:50:53,000] INFO [main.flow_generator_train:54] args: C:\Users\Sander\anaconda3\envs\deepethogramg\lib\site-packages\deepethogram\flow_generator\train.py project.path=C:\DeepEthogram\test_deepethogram reload.weights=C:\DeepEthogram\test_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt [2024-05-22 11:50:53,000] INFO [main.flow_generator_train:62] configuration used ~~~~~ [2024-05-22 11:50:53,010] INFO [main.flow_generator_train:63] split: reload: true file: null train_val_test:

  • 0.8
  • 0.2
  • 0.0 compute: fp16: false num_workers: 8 batch_size: 32 min_batch_size: 8 max_batch_size: 512 distributed: false gpu_id: 0 dali: false metrics_workers: 0 reload: overwrite_cfg: false latest: false weights: C:\DeepEthogram\test_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt notes: null log: level: info augs: brightness: 0.25 contrast: 0.1 hue: 0.1 saturation: 0.1 color_p: 0.5 grayscale: 0.5 crop_size: null resize:
  • 224
  • 224 dali: false random_resize: false pad: null LR: 0.5 UD: 0.0 degrees: 10 normalization: 'N': 324172800 mean:
    • 0.26870773349589994
    • 0.26870773349589994
    • 0.26870773349589994 std:
    • 0.23503533203484067
    • 0.23503533203484067
    • 0.23503533203484067 train: lr: 0.0001 scheduler: plateau num_epochs: 10 steps_per_epoch: train: 1000 val: 200 test: 20 min_lr: 5.0e-07 stopping_type: learning_rate milestones:
  • 50
  • 100
  • 150
  • 200
  • 250
  • 300 weight_loss: true patience: 3 early_stopping_begins: 0 viz_metrics: true viz_examples: 10 reduction_factor: 0.1 loss_weight_exp: 1.0 loss_gamma: 1.0 label_smoothing: 0.05 oversampling_exp: 0.0 regularization: style: l2_sp alpha: 1.0e-05 beta: 0.001 flow_generator: type: flow_generator flow_loss: MotionNet flow_max: 10 input_images: 11 flow_sparsity: false smooth_weight_multiplier: 1.0 sparsity_weight: 0.0 loss: MotionNet max: 5 n_rgb: 11 arch: TinyMotionNet weights: pretrained cmap: deepethogram control_arrow_jump: 31 label_view_width: 31 postprocessor: min_bout_length: 1 type: min_bout_per_behavior prediction_opacity: 0.2 project: class_names:
  • background
  • walking
  • ghead
  • gfront
  • gabdomen
  • gback
  • gwing
  • flight
  • pushing config_file: C:\DeepEthogram\test_deepethogram\project_config.yaml data_path: C:\DeepEthogram\test_deepethogram\DATA labeler: null model_path: C:\DeepEthogram\test_deepethogram\models name: test path: C:\DeepEthogram\test_deepethogram pretrained_path: C:\DeepEthogram\test_deepethogram\models\pretrained_models run: type: train model: flow_generator dir: C:\DeepEthogram\test_deepethogram\models\240522_115052_flow_generator_train sequence: filter_length: 15 unlabeled_alpha: 0.1 vertical_arrow_jump: 3

[2024-05-22 11:50:53,123] INFO [main.flow_generator_train:67] Total trainable params: 1,951,784 [2024-05-22 11:50:53,286] INFO [deepethogram.projects.get_weightfile_from_cfg:1068] loading pretrained weights: C:\DeepEthogram\test_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt reloading weights... [2024-05-22 11:50:53,287] INFO [deepethogram.utils.load_state:341] loading from checkpoint file C:\DeepEthogram\test_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt... [2024-05-22 11:50:53,314] INFO [main.get_metrics:364] key metric is SSIM [2024-05-22 11:50:53,323] INFO [deepethogram.data.augs.get_gpu_transforms:246] GPU transforms: {'train': Sequential( (0): ToFloat() (1): VideoSequential( (RandomHorizontalFlip_0): RandomHorizontalFlip(p=0.5, p_batch=1.0, same_on_batch=False) (RandomRotation_1): RandomRotation(degrees=10, p=0.5, p_batch=1.0, same_on_batch=False, resample=bilinear, align_corners=True) (ColorJitter_2): ColorJitter(brightness=0.25, contrast=0.1, saturation=0.1, hue=0.1, p=0.5, p_batch=1.0, same_on_batch=False) (RandomGrayscale_3): RandomGrayscale(p=0.5, p_batch=1.0, same_on_batch=False) ) (2): NormalizeVideo() (3): StackClipInChannels() ), 'val': Sequential( (0): ToFloat() (1): NormalizeVideo() (2): StackClipInChannels() ), 'test': Sequential( (0): ToFloat() (1): NormalizeVideo() (2): StackClipInChannels() ), 'denormalize': Sequential( (0): UnstackClip() (1): DenormalizeVideo() )} [2024-05-22 11:50:53,323] INFO [deepethogram.base.init:95] scheduler mode: min [2024-05-22 11:50:53,393] INFO [deepethogram.losses.get_regularization_loss:205] Regularization: L2_SP. Pretrained file: C:\DeepEthogram\test_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt alpha: 1e-05 beta: 0.001 [2024-05-22 11:50:53,412] INFO [deepethogram.flow_generator.losses.init:179] Using MotionNet Loss with settings: smooth_weights: [0.01, 0.02, 0.04, 0.08, 0.16] flow_sparsity: False sparsity_weight: 0.0 Traceback (most recent call last): File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\site-packages\deepethogram\base.py", line 398, in get_trainer_from_cfg log_every_n_steps=1) File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\site-packages\pytorch_lightning\utilities\argparse.py", line 348, in insert_env_defaults return fn(self, **kwargs) TypeError: init() got an unexpected keyword argument 'reload_dataloaders_every_epoch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\site-packages\deepethogram\flow_generator\train.py", line 374, in flow_generator_train(cfg) File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\site-packages\deepethogram\flow_generator\train.py", line 78, in flow_generator_train trainer = get_trainer_from_cfg(cfg, lightning_module, stopper) File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\site-packages\deepethogram\base.py", line 413, in get_trainer_from_cfg log_every_n_steps=1) File "C:\Users\Sander\anaconda3\envs\deepethogramg\lib\site-packages\pytorch_lightning\utilities\argparse.py", line 348, in insert_env_defaults return fn(self, **kwargs) TypeError: init() got an unexpected keyword argument 'progress_bar_refresh_rate' [2024-05-22 11:50:53,974] INFO [deepethogram.gui.main.flow_train:353] Training finished. If you see error messages above, training did not complete successfully. [2024-05-22 11:50:53,974] INFO [deepethogram.gui.main.flow_train:359] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [2024-05-22 11:50:53,982] INFO [deepethogram.gui.main.project_loaded_buttons:175] Number finalized labels: 0"

Could you please help me on this?

Specom avatar May 22 '24 09:05 Specom