NAS-Bench-360

A NAS Benchmark for Diverse Tasks


Neural architecture search (NAS) benchmarks and methods prioritize performance on well-studied tasks, e.g., image classification on CIFAR and ImageNet. To mitigate this bias, NAS-Bench-360 is a benchmark suite for evaluating state-of-the-art NAS methods on a diverse set of tasks. The selection spans different application domains, dataset sizes, problem dimensionalities, and learning objectives. Please contact Renbo Tu (renbo@cmu.edu) if you have questions or wish to contribute to the benchmark.

GitHub

Datasets




Information
Number of samples: 60K
Data split: 40,000/10,000/10,000
Task type: Point
License: CC BY 4.0
Download link: all data

Information
Number of samples: 60K
Data split: 40,000/10,000/10,000
Task type: Point
License: CC BY-SA
Metadata: pdf
Download link: all data

Information
Number of samples: 3,916
Data split: 2,638/659/659
Task type: Point
License: CC BY-ND
Download links:
Train data
Train labels
Validation data
Validation labels
Test data
Test labels

Information
Number of samples: 51K
Data split: 37K/4K/10K
Task type: Point (multi-label)
License: CC BY 4.0
Download link: all data(24 GB)

Information
Number of samples: 1,100
Data split: 900/100/100
Task type: Dense
License: MIT
Download links:
Train + Validation data
Test data

Information
Number of samples: 3,606
Data split: 3,356/100/150
Task type: Dense
License: GPL
Download link: all data

Information
Number of samples: 5,250
Data split: 4,347/483/420
Task type: Dense
License: Open data
Download links:
Train+Validation data (6.5 GB)
Test data (2.0 GB)

Information
Number of samples: 330K
Data split: 260K/33K/33K
Task type: 1D Point
License: ODC-BY 1.0
Download links:
all data

Information
Number of samples: 1M
Data split: 800K/100K/100K
Task type: 1D Point
License: GPL
Download links:
Train+Validation data
Test data

Information
Number of samples: 250K
Data split: 71K/2.5K/150K
Task type: 1D Point (multi-label)
License: CC BY 4.0
Download links:
all data

Citation

Please cite the following if you use NAS-Bench-360:

NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search
Renbo Tu, Mikhail Khodak, Nicholas Roberts, Ameet Talwalkar 

			@misc{tu2021nasbench360 
			      title={NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search}, 
			      author={Renbo Tu and Mikhail Khodak and Nicholas Roberts and Ameet Talwalkar},
			      year={2021},    
			      eprint={2110.05668},
			      archivePrefix={arXiv},
			      primaryClass={cs.CV}
			      
			}

License

NAS-Bench-360 is released under the MIT license.