Tensorflow-Neuroevolution icon indicating copy to clipboard operation
Tensorflow-Neuroevolution copied to clipboard

RAM consumption on GPU Grows Infinitely

Open hyang0129 opened this issue 5 years ago • 0 comments

https://github.com/PaulPauls/Tensorflow-Neuroevolution/blob/55c76f08ee4e4206d842565902b3d11c517c3756/tfne/environments/cifar10_environment.py#L64

This isn't about the VRAM, which seems to be cleared properly. The problem lies with the system RAM when running the _eval_genome_fitness_weight_training method when using a GPU. The RAM usage balloons over time.

This can be fixed by implementing a clear session before or after the evaluation method.

def _eval_genome_fitness_weight_training(self, genome) -> float:
    K.clear_session()
    
    model = genome.get_model()
    optimizer = genome.get_optimizer()

    do stuff 

hyang0129 avatar Apr 09 '21 15:04 hyang0129