Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
NAS evaluation is frustratingly hard
Authors: Antoine Yang, Pedro M. Esperança, Fabio M. Carlucci
ICLR 2020 | Venue PDF | LLM Run Details | Input Tokens: 18,489 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,602 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our first contribution is a benchmark of 8 NAS methods on 5 datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method s relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols. Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. |
| Researcher Affiliation | Collaboration | Antoine Yang Ecole Polytechnique France Pedro M Esperanc a Huawei Noah s Ark Lab London, UK Fabio Maria Carlucci Huawei Noah s Ark Lab London, UK This work was done when the first author was an intern at Huawei Noah s Ark Lab, London, United Kingdom. |
| Pseudocode | No | The paper describes methods and protocols in prose, but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code used is available at https://github.com/antoyang/NAS-Benchmark. |
| Open Datasets | Yes | The datasets used are CIFAR10, CIFAR100, SPORT8, MIT67 and FLOWERS102. The CIFAR10 dataset (Krizhevsky, 2009)... The CIFAR100 dataset (Krizhevsky, 2009)... SPORT8... (Li & Fei-Fei, 2007). MIT67... (Quattoni & Torralba, 2009). FLOWERS102... (Nilsback & Zisserman, 2008). |
| Dataset Splits | Yes | Each of these datasets is split into a training, validation and testing subsets of size 25, 000, 25, 000 and 10, 000 respectively. (for CIFAR10/100) Each of these datasets is split into a training, validation and testing subsets with proportions 40/40/20 (%). (for SPORT8, MIT67, FLOWERS102) |
| Hardware Specification | Yes | All experiments were run on NVIDIA Tesla V100 GPUs. |
| Software Dependencies | Yes | It notably updates the official implementation to a pytorch version posterior to 0.4. |
| Experiment Setup | Yes | Table 2: Hyperparameters for different NAS methods. Appendix A.1 METHODS AND HYPERPARAMETERS provides detailed settings for optimizers, learning rates, epochs, batch sizes, and additional enhancements like Cutout, Drop Path, Auxiliary Towers. |