Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Formalizing the Generalization-Forgetting Trade-off in Continual Learning
Authors: Krishnan Raghavan, Prasanna Balaprakash
NeurIPS 2021 | Venue PDF | LLM Run Details | Input Tokens: 21,674 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,662 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We use the CL benchmark [21] for our experiments and retain the experimental settings (hyperparameters) from [21, 44]. For comparison, we use the split-MNIST, permuted-MNIST, and split-Ci FAR100 data sets while considering three scenarios: incremental domain learning (IDL), incremental task learning (ITL), and incremental class learning (ICL). ... All experiments are conducted in Python 3.4 using the pytorch 1.7.1 library with the NVIDIA-A100 GPU for our simulations. |
| Researcher Affiliation | Academia | R. Krishnan1 and Prasanna Balaprakash1,2 1Mathematics and Computer Science Division 2Leadership Computing Facility Argonne National Laboratory kraghavan,EMAIL |
| Pseudocode | Yes | The pseudo code of the BCL is shown in Algorithm 1. |
| Open Source Code | Yes | Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Refer to supplementary materials |
| Open Datasets | Yes | We use the CL benchmark [21] for our experiments and retain the experimental settings (hyperparameters) from [21, 44]. For comparison, we use the split-MNIST, permuted-MNIST, and split-Ci FAR100 data sets while considering three scenarios: incremental domain learning (IDL), incremental task learning (ITL), and incremental class learning (ICL). |
| Dataset Splits | No | The main paper body does not explicitly detail the training/validation/test dataset splits. It refers to Appendix C in supplementary materials and [21] for these details in the author checklist. |
| Hardware Specification | Yes | All experiments are conducted in Python 3.4 using the pytorch 1.7.1 library with the NVIDIA-A100 GPU for our simulations. |
| Software Dependencies | Yes | All experiments are conducted in Python 3.4 using the pytorch 1.7.1 library with the NVIDIA-A100 GPU for our simulations. |
| Experiment Setup | No | The paper states that experimental settings (hyperparameters) are retained from [21, 44] and refers to Appendix C in supplementary materials for training details, but does not explicitly provide them within the main text. |