NUMWORKER ESSENTIAL

[Thiswycf]

In given file NAS-Bench-201/datasets_nas.py, the loaders in the get_dataloader interface need to be changed for efficiency. For an example, to train_loader:

# NAS-Bench-201/datasets_nas.py
def get_dataloader(config, train_dataset, eval_dataset, test_dataset):
	train_loader = DataLoader(dataset=train_dataset,
	batch_size=config.training.batch_size,
        shuffle=True,
        collate_fn=None,
        num_workers=32) # Need to be added

So as eval_loader and test_loader. Otherwise, it costs me average 18ms to load an item, and it comes to 4.8s when batch size is 256 !! That's costly. And advised num_workers as cpu_count * 4 when IO_intensive, cpu_count * 1 when memory_intensive and cpu_count * 2 when compute_intensive (my cpu count is 32).

[Thiswycf]

Linux check cpu count

cat /proc/cpuinfo | grep "cpu cores" | uniq