Josh Williams
Josh Williams
@pplonski Interesting, I will look more into my code. Were you testing using the mnist example?
Yeah I think that would be reasonable. I have rarely seen a package give an exact compilation flag for a dependency.
Hi @pplonski I just got around to looking into this further. It seems keras do not use dropout in prediction (https://github.com/keras-team/keras/blob/dc95ceca57cbfada596a10a72f0cb30e1f2ed53b/keras/layers/core.py#L116). I guess this is consistent then, but strange that...
HI @CBqk3762 An easy check to first see if VTK is installed correctly would be to try run a MOOSE example with the `vtk = true` in `[Outputs]`. Could you...
Yes seems your libmesh is not picking up VTK. I had that problem before a while ago, and just ended up moving on (I do need to go back and...
Hi @GiudGiud Thanks for the helpful comments. I was under the impression that `greedy_search=true` would handle the issue of nodes on different processes. But you are right, I made a...
From the log file, it seems we have vanishing gradients, based on the fact that the loss is barely changing. I should mention including `lbann.Scale(x, constant=1)` in my StyleGAN implementation...
Ok thanks, I added `os.environ['LBANN_NO_INPLACE'] = "1"` to my python script. It seems to be working well.
Quick question, how much extra overhead do you expect the environment variable `LBANN_NO_INPLACE=1` to create? I am seeing a significantly lower performance compared to pytorch (approx 3.5 days compared to...
Will check the fix tomorrow morning (UK time).. Just a quick comment regarding the performance issue I mentioned, I don't think it is the in-place. I profiled my job, see...