Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Compositional Explanations of Neurons
Authors: Jesse Mu, Jacob Andreas
NeurIPS 2020 | Venue PDF | LLM Run Details | Input Tokens: 17,829 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,545 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but semantically coherent visual concepts, while other polysemantic neurons detect multiple unrelated features; in natural language inference (NLI), neurons learn shallow lexical heuristics from dataset biases. Second, we see whether compositional explanations give us insight into model performance: vision neurons that detect human-interpretable concepts are positively correlated with task performance, while NLI neurons that fire for shallow heuristics are negatively correlated with task performance. |
| Researcher Affiliation | Academia | Jesse Mu Stanford University EMAIL Jacob Andreas MIT CSAIL EMAIL |
| Pseudocode | No | The paper describes the explanation generation procedure verbally and visually in Figure 1, but it does not include a formal pseudocode block or algorithm. |
| Open Source Code | Yes | Code and data are available at github.com/jayelm/compexp. |
| Open Datasets | Yes | We take the final 512unit convolutional layer of a Res Net-18 [15] trained on the Places365 dataset [40], probing for concepts in the ADE20k scenes dataset [41] with atomic concepts C defined by annotations in the Broden dataset [5]. |
| Dataset Splits | Yes | We use the SNLI validation set as our probing dataset (10K examples). |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU or CPU models, or cloud computing specifications used for running the experiments. |
| Software Dependencies | No | The paper mentions 'Spa Cy1' and 'GloVe embedding space [28]' but does not provide specific version numbers for software dependencies or libraries used in the experiments. |
| Experiment Setup | Yes | Since we cannot exhaustively search L(C), in practice we limit ourselves to formulas of maximum length N, by iteratively constructing formulas from primitives via beam search with beam size B = 10. At each step of beam search, we take the formulas already present in our beam, compose them with new primitives, measure Io U of these new formulas, and keep the top B new formulas by Io U, as shown in Figure 1e. |