Jan Profant

icon indicating an email [email protected]

Results 11 comments of Jan Profant

Training on a large set is much slower than on a smaller set - proportionally

Thank you for the response. I am running a simple profiler in pytorch_lightning and this is the output for `dev` (discarded small percentages): ``` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | Action | Mean duration...

Training on a large set is much slower than on a smaller set - proportionally

So I ran the same code on 2000 hours of training audios. There is a huge shift in `[_TrainingEpochLoop].train_dataloader_next` from 5.7% with 545 hours to 61.26% with 2000 hours, even...

Training on a large set is much slower than on a smaller set - proportionally

Not sure how to profile this in some advanced way, but with some manual profiling I believe that one of the issues is on this line https://github.com/pyannote/pyannote-audio/blob/f45da7161e22a3f16839b9b7177c4ba98097296b/pyannote/audio/tasks/segmentation/speaker_diarization.py#L331 (please note that...

Training on a large set is much slower than on a smaller set - proportionally

This is a version with the current `develop` and `np.searchsorted` implementation - I was able to make it work after some minor changes. I was expecting more, I am gonna...

Training on a large set is much slower than on a smaller set - proportionally

I ran the current `develop`, and this is the result ``` -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Action | Mean duration (s) | Num calls | Total time (s) | Percentage % | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------...

Training on a large set is much slower than on a smaller set - proportionally

So I ran all three on a local volume with the highest possible disk setup on AWS `develop` ``` -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | Action | Mean duration (s) | Num calls |...

Training on a large set is much slower than on a smaller set - proportionally

But what is strange, when I change the number of workers from 2 to 4 (I have 4 physical cores and 8 threads per GPU), numbers change dramatically. This is...

Training on a large set is much slower than on a smaller set - proportionally

I was starting on g4dn.xlarge which has half of the resources of p3.2xlarge, but that drop when increasing the number of workers is huge, I was not expecting that. So...

Training on a large set is much slower than on a smaller set - proportionally

I am trying to push my changes to a branch named `segments_dict`. But when doing ```bash git push --set-upstream origin segments_dict ``` I got ``` ERROR: Permission to pyannote/pyannote-audio.git denied...

Training on a large set is much slower than on a smaller set - proportionally

Right, makes sense. Thank you. https://github.com/pyannote/pyannote-audio/pull/1633