Michael Carilli

There may still be some python import weirdness going on. Try moving apex out of your training directory hierarchy, to a completely different location.

This might be a cudnn issue, especially if you're using cudnn 7.2. Try ``` >>> import torch >>> torch.backends.cudnn.version() ``` Upgrading your cudnn version may fix it: https://github.com/NVIDIA/apex/issues/78#issuecomment-440301134 Container options...

> I tried updating and unfortunately the error persists. The command you mentioned outputs 7401. @pancho111203 Since you've got cuda 10 on bare metal (meaning your system has the cuda...

Yes. `torch.cuda.amp.autocast` can be enabled wherever you want and affects only ops invoked within enabled regions. `autocast` and `torch.cuda.amp.GradScaler` are modular codewise. During training, you should use both (`autocast` selects...

@Damiox `torch.cuda.amp.autocast` is similar to O1 in that it casts function inputs on the fly without touching model weights. However, unlike apex O1, `autocast` only causes casting behavior in regions...

@vince62s `apex.optimizers.FusedAdam` and `torch.optim.Adam` should both work out of the box with native Amp following the [documented control flow](https://pytorch.org/docs/master/notes/amp_examples.html#typical-mixed-precision-training) (create model in default precision aka fp32). If you also need...

@ysystudio Autocast does not touch the model object itself, so its dtype (param type) remains as you created it (leaving it to default FP32 is recommended). Save the trained model...

@SeungjunNah The options available in native Amp are a better representation of what users should control. Some apex options, like opt-level O2, are unsafe for general use. If an option...

>amp would skip a gradient-overflowed batch every N intervals That's true, but N is a large value (2000 by default). After the initial few iterations where GradScaler calibrates, it settles...

It really depends on your model. It's hard to tell the impact of non-overlapped dataloading, and if prefetching successfully overlaps, without looking at a visual profile. For data prefetching to...