nlopez94 comments

Results 4 comments of


                                            nlopez94

Issues with Incompatible TensorRT libraries in docker image google/deepvariant:latest-gpu and google/deepvariant:1.6.1-gpu

Hi @kishwarshafin, The training process starts as expected with GPU activity visible, but it abruptly stops without any error message while processing the first epoch and determining the best checkpoint...

Issues with Incompatible TensorRT libraries in docker image google/deepvariant:latest-gpu and google/deepvariant:1.6.1-gpu

@kishwarshafin As I mentioned before, the same dataset and parameters were used when I ran this on CPU, and as I indicated earlier, this process continued without abruptly ending as...

Issues with Incompatible TensorRT libraries in docker image google/deepvariant:latest-gpu and google/deepvariant:1.6.1-gpu

@kishwarshafin I will try this and update you on the results I get. Thank you so much for the support!

Issues with Incompatible TensorRT libraries in docker image google/deepvariant:latest-gpu and google/deepvariant:1.6.1-gpu

@kishwarshafin I just found the error that was causing this to abruptly exit without warning. I was running the script with insufficient memory, and after changing my instance type, everything...