Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Truthfulness of Calibration Measures
Authors: Nika Haghtalab, Mingda Qiao, Kunhe Yang, Eric Zhao
NeurIPS 2024 | Venue PDF | LLM Run Details | Input Tokens: 49,656 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 3,682 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We study calibration measures in a sequential prediction setup. We introduce a new calibration measure termed the Subsampled Smooth Calibration Error (SSCE), which is complete and sound, and under which truthful prediction is optimal up to a constant multiplicative factor. We answer this question in three parts: Part I: We show that existing calibration measures do not simultaneously meet these criteria. ... Part II: We introduce a new calibration measure, called SSCE, that is sound, complete, and approximately truthful. ... Part III: There is a forecasting algorithm that achieves O(T) SSCE even in the adversarial setting. |
| Researcher Affiliation | Academia | Nika Haghtalab, Mingda Qiao, Kunhe Yang, and Eric Zhao University of California, Berkeley EMAIL |
| Pseudocode | Yes | Algorithm 1: Forecaster for Product Distributions |
| Open Source Code | No | The paper does not contain any statement about releasing source code or links to a code repository. |
| Open Datasets | No | This is a theoretical paper and does not involve the use of datasets for training or evaluation. |
| Dataset Splits | No | This is a theoretical paper and does not describe any experimental data splits (training, validation, test). |
| Hardware Specification | No | This is a theoretical paper and does not describe any hardware used for experiments. |
| Software Dependencies | No | This is a theoretical paper and does not list any specific software dependencies with version numbers. |
| Experiment Setup | No | This is a theoretical paper and does not describe any experimental setup details such as hyperparameters or training configurations. |