jbohnslav
Prediction on new videos
Dear deepethogram devs,
I was wondering what would be the minimum steps required for, given a set of flow generator + feature extractor + sequence models already trained on my specific dataset, getting new predictions for new videos. It is not clear to me from the Colab notebook and the documentation. I only got the workflows there to work if the video is labeled and I train all 3 models on the video.
For example, I added a new video, and if I open it in the GUI and click on sequence > infer (which is the last step of the above workflows), I can only choose among the videos I used for training, but not the new one. Ideally, I would do this with the CLI though, with something like the one-liner you provide in the documentation:
python -m deepethogram.flow_generator.train project.config_file=path/to/config/file.yaml reload.weights=path/to/flow/weights.pt split.file=path/to/split.yaml preset=deg_s
Maybe I only need to train the flow generator (which needs no user labels) on the new video, but not the feature extractor? What about the sequence model?
Thank you!
Antonio
Answering my own question:
when training deepethogram on any video, this is the sequence of steps that need to be taken (as explained in the Colab notebook )
flow_generator.train feature_extractor.train feature_extractor.inference sequence.train
i.e. the sequence model cannot be trained before the videos used for training have been passed through the feature extractor inference module.
Once all 3 models (flow generator, feature extractor and sequence) are trained, then one needs to do the following to predict on unseen videos
feature_extractor.inference sequence.inference
Moreover, when calling the inference modules from the CLI, one can pass the parameter inference.directory_list like so
inference.directory_list=[${PROJECT_PATH}/DATA/foo,${PROJECT_PATH}/DATA/bar]
to select which videos will be run through inference (vs all)
Finally, I have made a Github repository where the .pbs scripts I wrote to run deepethogram in a cluster are available for anyone to consult
@antortjim I am attempting to reuse my trained models on new, unseen videos, however, I keep encountering problems along the way. Were you able to successfully run feature_extractor.inference on new videos or did you have to retrain the feature_extractor first with the unseen videos? Could you please briefly describe your workflow (e.g. do you start a new project or add video to existing project - do you need to retrain the flow generator and feature extractor before running inference) for using pretrained models to create inferences on new video? Thank you in advance!
Hi @jbwallace123 when I reuse the trained models on new data I only run deepethogram.feature_extractor.inference and deepethogram.sequence.inference. So no retraining is needed!
I add the videos to the existing project and specify which videos I want to infer using the option inference.directory_list in the config.
I may be able to help if you share a log of the error, and also make sure you are using the last version of deepethogram
@antortjim Thank you so much for your help! Updating deepethogram completely remedied the issue and now it's working like a dream. Thank you for the advice and the quick reply!