Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Improving model calibration with accuracy versus uncertainty optimization
Authors: Ranganath Krishnan, Omesh Tickoo
NeurIPS 2020 | Venue PDF | LLM Run Details | Input Tokens: 17,963 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 6,251 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments demonstrate our approach yields better model calibration than existing methods on large-scale image classification tasks under distributional shift. |
| Researcher Affiliation | Industry | Ranganath Krishnan Intel Labs EMAIL Omesh Tickoo Intel Labs EMAIL |
| Pseudocode | Yes | Algorithm 1 SVI-Av UC optimization |
| Open Source Code | Yes | We have made the code 3 available to facilitate probabilistic deep learning community to evaluate and improve model calibration for various other baselines. 3https://github.com/Intel Labs/AVUC |
| Open Datasets | Yes | We use Res Net-50 and Res Net-20 [46] DNN architectures on Image Net [47] and CIFAR10 [48] datasets respectively. |
| Dataset Splits | No | The paper mentions using a 'held-out validation set' and 'test data (in-distribution)' but does not provide specific percentages or sample counts for these splits. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models or processor types) used for running its experiments. |
| Software Dependencies | No | The paper cites PyTorch as a deep learning library, but does not explicitly list its specific version or other software dependencies with their version numbers needed for replication. |
| Experiment Setup | Yes | We provide details of our model implementations and hyperparameters for SVI, SVI-TS, SVI-Av UC, SVI-Av UTS and Radial BNN in Appendix B. |