Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
BOBCAT: Bilevel Optimization-Based Computerized Adaptive Testing
Authors: Aritra Ghosh, Andrew Lan
IJCAI 2021 | Venue PDF | LLM Run Details | Input Tokens: 18,352 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 3,884 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through extensive experiments on five real-world student response datasets, we show that BOBCAT outperforms existing CAT methods (sometimes significantly) at reducing test length. |
| Researcher Affiliation | Academia | Aritra Ghosh and Andrew Lan University of Massachusetts Amherst EMAIL |
| Pseudocode | Yes | Algorithm 1 BOBCAT training process |
| Open Source Code | Yes | Our implementation will be publicly available at https://github.com/arghosh/BOBCAT. |
| Open Datasets | Yes | We use five publicly available benchmark datasets: Ed Net2, Junyi3, Eedi-1, Eedi-24, and ASSISTments5. ... 2https://github.com/riiid/ednet 3https://www.kaggle.com/junyiacademy/learning-activitypublic-dataset-by-junyi-academy 4https://eedi.com/projects/neurips-education-challenge 5https://sites.google.com/site/assistmentsdata/home/assistment2009-2010-data |
| Dataset Splits | Yes | We perform 5-fold cross validation for all datasets; for each fold, we use 60%-20%-20% students for training, validation, and testing, respectively. |
| Hardware Specification | Yes | We implement all methods in Py Torch and run our experiments in a NVIDIA Titan X/1080Ti GPU. |
| Software Dependencies | No | The paper mentions 'Py Torch' but does not specify a version number for it or other software dependencies. |
| Experiment Setup | Yes | For Bi NN, we use a two-layer, fully-connected network (with 256 hidden nodes, Re LU nonlinearity, 20% dropout rate, and a final sigmoid output layer)... We use another fully-connected network (with two hidden layers, 256 hidden nodes, Tanh nonlinearity, and a final softmax output layer) as the question selection algorithm. |