Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers
Authors: Wanqian Yang, Polina Kirichenko, Micah Goldblum, Andrew G. Wilson
NeurIPS 2022 | Venue PDF | LLM Run Details | Input Tokens: 18,918 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,123 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness of Chroma-VAE on several benchmark datasets (Section 5). In Section 5.1, we present results on the Colored MNIST benchmark... In Section 5.2, we apply Chroma-VAE to two large-scale benchmark datasets (Celeb A and MNIST-Fashion MNIST Dominoes), as well as a real-world problem involving pneumonia prediction using chest X-ray scans. |
| Researcher Affiliation | Academia | New York University EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1 Chroma-VAE |
| Open Source Code | Yes | Our code is publicly available at https://github.com/Wanqianxn/chroma-vae-public. |
| Open Datasets | Yes | Setup. Following Arjovsky et al. [1], (i) first we binarize MNIST [26] labels... We consider two benchmark Celeb A tasks:... We also consider the MF-Dominoes dataset [35]... Likewise, we make a training set (N = 20K) with roughly equal numbers of X-rays from the National Institutes of Health Clinical Center (NIH) dataset [48] and the Chexpert dataset [18] |
| Dataset Splits | Yes | For Colored MNIST and MF-Dominoes, we follow the same experimental setup as [27]. We then perform a random split with 80% for training and 20% for testing. For CelebA, we use the splits provided by the dataset directly. For the Chest X-ray dataset, we combine NIH [48] and Chexpert [18] to make a training set (N = 20K) and validation set (N = 5K) for training, and a fixed test set (N = 1.6K) for testing. |
| Hardware Specification | No | The paper states 'We note that the computational resources we used are typical of most research using the same models and datasets' but does not provide specific details such as GPU models, CPU types, or memory specifications used for the experiments. |
| Software Dependencies | No | The paper mentions 'Adam' as an optimization method, citing Kingma and Ba [22], but does not specify any particular software, libraries, or their version numbers (e.g., PyTorch 1.x, TensorFlow 2.x, Python 3.x) used for implementation. |
| Experiment Setup | Yes | Detailed experimental setups can be found in Appendix B. For example, in Section 5.1 (Colored MNIST setup), it mentions 'pd = 0.25' and 'pc = 0.1'. In Section 5.2, it discusses using the 'harder setting from Lee et al. [27]' for Celeb A and MF-Dominoes. Algorithm 1 also outlines the training procedure, including 'initialize (θ, ϕ, φ) using Adam'. |