Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Sanity Checks for Saliency Maps
Authors: Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, Been Kim
NeurIPS 2018 | Venue PDF | LLM Run Details | Input Tokens: 15,560 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 6,087 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through extensive experiments we show that some existing saliency methods are independent both of the model and of the data generating process. |
| Researcher Affiliation | Collaboration | Julius Adebayo , Justin Gilmer , Michael Muelly , Ian Goodfellow , Moritz Hardt , Been Kim EMAIL, EMAIL Google Brain University of California Berkeley |
| Pseudocode | No | The paper describes methods in prose and through mathematical formulations but does not include any explicit pseudocode or algorithm blocks. |
| Open Source Code | Yes | All code to replicate our ๏ฌndings will be available here: https://goo.gl/h Bmh Dt |
| Open Datasets | Yes | Inception v3 model trained on Image Net.; CNN Fashion MNIST MLPMNIST Inception v3 Image Net; MNIST test set for a CNN. |
| Dataset Splits | No | The paper mentions training models on datasets like ImageNet, MNIST, and Fashion MNIST, and refers to a โtest set,โ but does not provide specific training, validation, or test split percentages, sample counts, or explicit splitting methodologies for reproducibility. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory amounts, or exact cloud instance types) used for running its experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies, such as library names with version numbers (e.g., TensorFlow 2.x, PyTorch 1.x), required to replicate the experiments. |
| Experiment Setup | No | The paper describes the general training process for models (e.g., training to >95% accuracy) but does not provide specific experimental setup details such as hyperparameters (learning rate, batch size, number of epochs, optimizer settings) or detailed training configurations. |