Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Individual Calibration with Randomized Forecasting
Authors: Shengjia Zhao, Tengyu Ma, Stefano Ermon
ICML 2020 | Venue PDF | LLM Run Details | Input Tokens: 18,249 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,612 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We assess the benefit of forecasters trained with the new objective on two applications: fairness and decision making under uncertainty against adversaries. Calibration on protected groups traditionally has been a definition for fairness (Kleinberg et al., 2016). On a UCI crime prediction task, we show that forecasters trained for individual calibration achieve lower calibration error on protected groups without knowing these groups in advance.We perform simulations to verify that individual calibrated forecasters are less exploitable and can achieve higher utility in practice.Experiment Details. We use the UCI crime and communities dataset (Dua & Graff, 2017) and we predict the crime rate based on features of the neighborhood (such as racial composition). For training details and network architecture see Appendix B.3.The results are shown in Figure 3 and 4. |
| Researcher Affiliation | Academia | 1Computer Science Department, Stanford University. Correspondence to: Shengjia Zhao <EMAIL>, Tengyu Ma <EMAIL>, Stefano Ermon <EMAIL>. |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | 2https://github.com/Shengjia Zhao/Individual-Calibration |
| Open Datasets | Yes | We use the UCI crime and communities dataset (Dua & Graff, 2017).URL http://archive.ics.uci.edu/ml. |
| Dataset Splits | No | The paper mentions 'validation samples' and 'random training/validation partitions' but does not provide specific split percentages or sample counts for the validation set in the provided text. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | Yes | We will model uncertainty with Gaussian distributions {N(µ, σ2), µ R, σ R+}. We parameterize h = hθ as a deep neural network with parameters θ. The neural networks takes in concatenation of x and r and outputs the µ, σ that decides the returned CDF.We use a hyper-parameter α to trade-off between the two objectives: Lα(θ) = (1 α)LPAIC(θ) + αLNLL(θ) (4).We compare different values of α [0.1, 1.0]. |