Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
What Do Self-Supervised Vision Transformers Learn?
Authors: Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, Sangdoo Yun
ICLR 2023 | Venue PDF | LLM Run Details | Input Tokens: 21,541 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,201 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We present a comparative study on how and why contrastive learning (CL) and masked image modeling (MIM) differ in their representations and in their performance of downstream tasks. |
| Researcher Affiliation | Industry | Namuk Park1 Wonjae Kim2 Byeongho Heo2 Taekyung Kim2 Sangdoo Yun2 1Prescient Design, Genentech 2NAVER AI Lab EMAIL EMAIL |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | Yes | The code for analysis is available at https://github.com/naver-ai/cl-vs-mim. |
| Open Datasets | Yes | Our analyses mainly compare Vi T-B/16 pre-trained on Image Net-1K (Russakovsky et al., 2015) |
| Dataset Splits | Yes | We use the Image Net validation images for our experiments. Table A.1: Training settings. batch size 1k training epoch 50 |
| Hardware Specification | Yes | All experiments use {1, 4, 8} NVIDIA A100 Tensor Core GPU. |
| Software Dependencies | No | Neural network models are implemented in Py Torch (Paszke et al., 2019). |
| Experiment Setup | Yes | Table A.1: Training settings. optimizer sgd adamw adamw base learning rate 1.0e-0 1.25e-3 1.0e-4 weight decay 0.05 0.05 0.05 batch size 1k 2k 1k training epoch 50 100 100 learning rate schedule cosine cosine multistep warmup epoch 0 20 10 warmup schedule linear linear randaugment 9, 0.5 9, 0.5 label smoothing 0.1 0.1 mixup 0.8 0.8 cutmix 1.0 1.0 stochastic depth 0.1 0.1 layer decay 0.65 1.0 gradient clip 5.0 5.0 |