Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Sampling-Based Accuracy Testing of Posterior Estimators for General Inference
Authors: Pablo Lemos, Adam Coogan, Yashar Hezaveh, Laurence Perreault-Levasseur
ICML 2023 | Venue PDF | LLM Run Details | Input Tokens: 24,078 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 3,012 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We apply our algorithm, described in Algorithm 2 to three different experiments. |
| Researcher Affiliation | Academia | 1Mila Quebec AI Institute, Montreal, Quebec, Canada 2Universit e de Montr eal, Montreal, Quebec, Canada 3CIELA Institute, Montreal, Quebec, Canada 4Flatiron Institute Center for Computational Mathematics, 162 5th Ave, 3rd floor, New York, NY 10010, USA. Correspondence to: Pablo Lemos <EMAIL>. |
| Pseudocode | Yes | Algorithm 1 Calculation of ECP(ˆp, α, H) using highest posterior density regions, from a set of simulations {θi, xi}, i [1, Nsims] Algorithm 2 Calculation of ECP(ˆp, α, Dθr) using the TARP method, using a set of simulations {θi, xi}, i {1, . . . , Nsims}, parameter distance metric d : U U R 0 and reference point sampling distribution p( |x). |
| Open Source Code | Yes | Our code is available at https://github.com/Ciela-Institute/tarp. |
| Open Datasets | Yes | The simulator in this scenario samples the source galaxy s light θ from a multivariate-normal distribution that we fit to a dataset of galaxy images (Stone & Courteau, 2019; Stone et al., 2021). |
| Dataset Splits | No | The paper describes generating simulations and using them for validation (e.g., 'We first generate simulations', 'For each of these cases, we want to compare...'). However, it does not specify explicit train/validation/test splits with percentages or sample counts for any of the datasets used, nor does it refer to predefined splits. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments (e.g., CPU/GPU models, memory specifications). |
| Software Dependencies | No | The paper does not list specific software dependencies with version numbers (e.g., 'Python 3.8, PyTorch 1.9'). |
| Experiment Setup | Yes | We find 300 steps are sufficient to ensure convergence. |