Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
On the Convergence of Encoder-only Shallow Transformers
Authors: Yongtao Wu, Fanghui Liu, Grigorios Chrysos, Volkan Cevher
NeurIPS 2023 | Venue PDF | LLM Run Details | Input Tokens: 39,168 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,441 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments are organized as follows: In Section 5.1, we conduct experiments with the model Eq. (1.2) on synthetic data and study the training dynamics. Next, we show convergence results on Vi T [Dosovitskiy et al., 2021] on the standard MNIST dataset in Section 5.2. |
| Researcher Affiliation | Academia | Yongtao Wu LIONS, EPFL EMAIL; Fanghui Liu University of Warwick EMAIL; Grigorios G Chrysos LIONS, EPFL University of Wisconsin-Madison EMAIL; Volkan Cevher LIONS, EPFL EMAIL |
| Pseudocode | Yes | Algorithm 1: Gradient descent training |
| Open Source Code | No | The paper does not provide any explicit statements or links indicating that its source code is open or publicly available. |
| Open Datasets | Yes | Here we experimentally validate this assumption under a standard language IMDB dataset [Maas et al., 2011]... Next, we show convergence results on Vi T [Dosovitskiy et al., 2021] on the standard MNIST dataset [Le Cun et al., 1998] |
| Dataset Splits | No | The paper mentions generating 100 data points for synthetic data and using the MNIST dataset, but it does not specify explicit training, validation, or test split percentages or sample counts in the main text. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., Python, PyTorch, CUDA versions). |
| Experiment Setup | Yes | We apply gradient descent on the shallow Transformer defined in Eq. (1.2) with Le Cun initialization and τ0 = d 1/2 m for 400 epochs with a fixed step size γ = 1. We test different widths of the network including dm = {10, 100, 1000, 4000}... The dimension of d is 64. We change the dimension of the query, key, and value from 16 to 1024 and 16384. The network is optimized with SGD with step size 0.1, and momentum 0.9 for 50 epochs. |