Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The Shift Toward Open and Reproducible AI Research," Under Review, 2026. doi: 10.48550/arXiv.2606.16974..
Efficient Online Estimation of Causal Effects by Deciding What to Observe
Authors: Shantanu Gupta, Zachary Lipton, David Childers
NeurIPS 2021 | Venue PDF | LLM Run Details | Input Tokens: 19,362 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,878 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we validate our ๏ฌndings experimentally 1 (Section 6). We validate our methods on synthetic data generated from known causal graphs (see Appendix D for parameter values and moment conditions used). |
| Researcher Affiliation | Academia | Shantanu Gupta, Zachary C. Lipton, David Childers Carnegie Mellon University EMAIL |
| Pseudocode | Yes | Figure 1: Algorithms for OMS-ETC and OMS-ETG. |
| Open Source Code | Yes | The code and data are available at https://www.github.com/acmi-lab/ online-moment-selection. |
| Open Datasets | Yes | Finally, we demonstrate the effectiveness of our methods on two semi-synthetic datasets: the Infant Health Development Program (IHDP) dataset [19] and a Vietnam era draft lottery dataset [2]. Replication data for: Lifetime Earnings and the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records, 2009. URL https://doi.org/10. 7910/DVN/PLF0YL. |
| Dataset Splits | No | No specific train/validation/test dataset splits (e.g., percentages, sample counts, or explicit standard splits) are mentioned. The paper describes generating semi-synthetic data and states 'The MSE values are computed across 12, 000 runs.' |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, processor types, memory amounts) used for running experiments are mentioned in the paper. |
| Software Dependencies | No | No specific software dependencies with version numbers (e.g., library names like 'PyTorch 1.9' or solver versions like 'CPLEX 12.4') are provided. The paper only mentions using a 'regularized GMM weight matrix' and implies Python use given the GitHub link. |
| Experiment Setup | No | The paper mentions 'For OMS-ETG, we use the regularized GMM weight matrix with ฮปW := 0.01' and 'The MSE values are computed across 12, 000 runs'. However, it does not provide specific hyperparameter values for model training (e.g., learning rates, batch sizes, epochs) or detailed system-level configurations. |