Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Position: Why We Must Rethink Empirical Research in Machine Learning
Authors: Moritz Herrmann, F. Julian D. Lange, Katharina Eggensperger, Giuseppe Casalicchio, Marcel Wever, Matthias Feurer, David Rügamer, Eyke Hüllermeier, Anne-Laure Boulesteix, Bernd Bischl
ICML 2024 | Venue PDF | LLM Run Details | Input Tokens: 35,174 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 3,518 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | This position paper warns against similar tendencies in empirical research in machine learning (ML) and calls for a mindset change to address methodological and epistemic challenges of experimentation. |
| Researcher Affiliation | Academia | 1Institute for Medical Information Processing, Biometry, and Epidemiology, Faculty of Medicine, LMU Munich, Munich, Germany 2Munich Center for Machine Learning (MCML), Munich, Germany 3University of Tübingen, Tübingen, Germany 4Department of Statistics, LMU Munich, Munich, Germany 5Institute of Informatics, LMU Munich, Munich, Germany. |
| Pseudocode | No | The paper is a position paper and does not present any algorithms or pseudocode. |
| Open Source Code | No | The paper is a position paper and does not release its own source code for a new method. |
| Open Datasets | No | The paper is a position paper and does not describe experiments conducted with a specific dataset. |
| Dataset Splits | No | The paper is a position paper and does not describe experiments conducted with dataset splits. |
| Hardware Specification | No | The paper is a position paper and does not conduct experiments, so no hardware specifications are mentioned. |
| Software Dependencies | No | The paper is a position paper and does not conduct experiments, so no specific software dependencies with version numbers are mentioned. |
| Experiment Setup | No | The paper is a position paper and does not conduct experiments, so no experimental setup details or hyperparameters are provided. |