Tensorflow-Neuroevolution
Tensorflow-Neuroevolution copied to clipboard
RAM consumption on GPU Grows Infinitely
https://github.com/PaulPauls/Tensorflow-Neuroevolution/blob/55c76f08ee4e4206d842565902b3d11c517c3756/tfne/environments/cifar10_environment.py#L64
This isn't about the VRAM, which seems to be cleared properly. The problem lies with the system RAM when running the _eval_genome_fitness_weight_training method when using a GPU. The RAM usage balloons over time.
This can be fixed by implementing a clear session before or after the evaluation method.
def _eval_genome_fitness_weight_training(self, genome) -> float:
K.clear_session()
model = genome.get_model()
optimizer = genome.get_optimizer()
do stuff