Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
SPECTRE: defending against backdoor attacks using robust statistics
Authors: Jonathan Hayase, Weihao Kong, Raghav Somani, Sewoong Oh
ICML 2021 | Venue PDF | LLM Run Details | Input Tokens: 20,065 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 6,349 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirical evidence in these works suggest that a small fraction of poisoned data is sufficient to successfully create backdoors in trained neural networks. For example, CIFAR-10 data has 5,000 training examples for each of the ten classes. When the pixel attack (Gu et al., 2017) is launched with only 125 poisoned samples injected during training, the pixel attack succeeds in planting a backdoor in the trained model, achieving an attack accuracy of 63% (shown in Fig. 1 in blue triangles). |
| Researcher Affiliation | Academia | 1Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, United States. |
| Pseudocode | Yes | Algorithm 1: SPECTRE... Algorithm 2: QUantum Entropy scoring (QUESCORE)... Algorithm 3: k-IDENTIFIER... Algorithm 4: Target label identifier |
| Open Source Code | Yes | Code and pre-trained models are available at https:// github.com/Sewoong Lab/spectre-defense. |
| Open Datasets | Yes | On the CIFAR-10 dataset, each label has 5,000 clean examples. ... We use a poisoned CIFAR-10 dataset to train a 32-layer Res Net3 model... |
| Dataset Splits | No | The paper uses CIFAR-10, a dataset with standard splits, and mentions 'poisoned training data' and 'test examples'. However, it does not explicitly provide specific percentages, counts, or a clear reference to how training, validation, and test splits were defined or used in the experiments. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments, such as CPU or GPU models, or memory specifications. |
| Software Dependencies | No | The paper refers to modifying an implementation from a GitHub repository ('We modified the implementation at https://github.com/akamaster/pytorch_resnet_cifar10'), implying the use of PyTorch, but it does not specify version numbers for any software dependencies. |
| Experiment Setup | Yes | We use a poisoned CIFAR-10 dataset to train a 32-layer Res Net3 model composed of three groups of residual blocks with 16, 32, and 64 filters respectively and 5 residual blocks per group. Details of the training are provided in Appendix E. |