automl
Recreate a subset of benchmarks
Hi, what would be the best way to recreate a subset of benchmarking accuracy data from the ZC json? For ex: I would want to replicate the val accuracy obtained for the architecture 1-100 indexed from NASBench201. I understand that the val accuracies and corresponding ZC scores were open-sourced to remove the need for training, but I need to perform training for an internal evaluation. How do you suggest to go about this? Thank you.
Would you like to create a benchmark from NASBench201 Search Space's arches or the ZC scores? You can use Trainer in naslib/defaults to run and create benchmarks. If you want to create ZeroCost benchmarks, you can simply use ZeroCost available in predictors.
from naslib.predictors import ZeroCost
pred_type = "jacov"
zc_predictor = ZeroCost(method_type=zc_pred)
score = zc_predictor.query(graph=graph, dataloader=data_loader)
# The graph is of type NASBench201 type in your case and dataloader would be a subset of CIFAR10 data.
Let me know, if that answers your issue.
