NAS-Bench-360

A NAS Benchmark for Diverse Tasks


Neural architecture search (NAS) benchmarks and methods prioritize performance on well-studied tasks, e.g., image classification on CIFAR and ImageNet. To mitigate this bias, NAS-Bench-360 is a benchmark suite for evaluating state-of-the-art NAS methods on a diverse set of tasks. The selection spans different application domains, dataset sizes, problem dimensionalities, and learning objectives. Please contact Renbo Tu (renbo.tu@mail.utoronto.ca) or Nicholas Roberts (nick11roberts@cs.wisc.edu) if you have questions or wish to contribute to the benchmark.

GitHub

Datasets




Information
Number of samples: 60K
Data split: 40,000/10,000/10,000
Task type: Point
License: CC BY 4.0
Download link: all data

Information
Number of samples: 60K
Data split: 40,000/10,000/10,000
Task type: Point
License: CC BY-SA
Metadata: pdf
Download link: all data

Information
Number of samples: 3,916
Data split: 2,638/659/659
Task type: Point
License: CC BY-ND
Download links:
Train data
Train labels
Validation data
Validation labels
Test data
Test labels

Information
Number of samples: 51K
Data split: 37K/4K/10K
Task type: Point (multi-label)
License: CC BY 4.0
Download link: all data(24 GB)

Information
Number of samples: 1,100
Data split: 900/100/100
Task type: Dense
License: MIT
Download links:
Train + Validation data
Test data

Information
Number of samples: 3,606
Data split: 3,356/100/150
Task type: Dense
License: GPL
Download link: all data

Information
Number of samples: 5,250
Data split: 4,347/483/420
Task type: Dense
License: Open data
Download links:
Train+Validation data (6.5 GB)
Test data (2.0 GB)

Information
Number of samples: 330K
Data split: 260K/33K/33K
Task type: 1D Point
License: ODC-BY 1.0
Download links:
all data

Information
Number of samples: 1M
Data split: 800K/100K/100K
Task type: 1D Point
License: GPL
Download links:
Train+Validation data
Test data

Information
Number of samples: 250K
Data split: 71K/2.5K/150K
Task type: 1D Point (multi-label)
License: CC BY 4.0
Download links:
all data

Citation

Please cite the following if you use NAS-Bench-360:

NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
Renbo Tu*, Nicholas Roberts*, Mikhail Khodak, Junhong Shen, Frederic Sala, Ameet Talwalkar 
	@misc{tu2021benchmark,,
		doi = {10.48550/ARXIV.2110.05668},
		url = {https://arxiv.org/abs/2110.05668},
		author = {Tu, Renbo and Roberts, Nicholas and Khodak, Mikhail and Shen, Junhong and Sala, Frederic and Talwalkar, Ameet},
		title = {NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks},
		publisher = {arXiv},
		year = {2021},
		}


License

NAS-Bench-360 is released under the MIT license.