Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Ringing ReLUs: Harmonic Distortion Analysis of Nonlinear Feedforward Networks
Authors: Christian H.X. Ali Mehmeti-Göpel, David Hartmann, Michael Wand
ICLR 2021 | Venue PDF | LLM Run Details | Input Tokens: 26,957 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,412 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically verify the connection between blueshift and architectural choices, and provide evidence for a connection with trainability.Experiments confirm the theoretical predictions: We observe the predicted effects of depth, shortcuts and parallel computation on blueshift, and are able to differentiate different types of nonlinearities by the decay rate of coefficients of a polynomial approximation. |
| Researcher Affiliation | Academia | Christian H.X. Ali Mehmeti-Göpel Institute of Computer Science Johannes-Gutenberg University Mainz Staudingerweg 9, 55122 Mainz, Germany EMAIL David Hartmann Institute of Computer Science Johannes Gutenberg-University of Mainz Staudingerweg 9, 55128 Mainz, Germany EMAIL Michael Wand Institute of Computer Science Johannes Gutenberg-University of Mainz Staudingerweg 9, 55128 Mainz, Germany EMAIL |
| Pseudocode | No | The paper includes mathematical derivations and descriptions of methods but does not contain any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | The implementation for our experiments is based on Py Torch 1.5 and are provided as supplementary material. |
| Open Datasets | Yes | Dataset Cifar10 (Cifar100 for Figure 10) |
| Dataset Splits | Yes | We repeat the experiment on averaging-networks for the Cifar100 dataset, holding out 1% of the training data for validation. |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU or CPU models used for the experiments. |
| Software Dependencies | Yes | The implementation for our experiments is based on Py Torch 1.5 |
| Experiment Setup | Yes | The hyper-parameters below usually reach the standard test-accuracy of approximately 92-93% for a Res Net56 on Cifar10. Dataset Cifar10 (Cifar100 for Figure 10) Epochs 200 Scheduler Multistep (γ = 0.1) Milestones 100, 150 Learning rate 0.1 Batch size 128 Optimizer SGD + Momentum Momentum 0.9 Weight decay 0.0001 Augmentation Random Flip |