Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Validating Climate Models with Spherical Convolutional Wasserstein Distance
Authors: Robert Garrett, Trevor Harris, Zhuo Wang, Bo Li
NeurIPS 2024 | Venue PDF | LLM Run Details | Input Tokens: 31,446 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,622 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We apply this method to evaluate the historical model outputs of the Coupled Model Intercomparison Project (CMIP) members by comparing them to observational and reanalysis data products. Additionally, we investigate the progression from CMIP phase 5 to phase 6 and find modest improvements in the phase 6 models regarding their ability to produce realistic climatologies. |
| Researcher Affiliation | Academia | 1University of Illinois Urbana-Champaign 2Texas A&M University |
| Pseudocode | Yes | Algorithm 1 Spherical Convolutional Wasserstein Distance Approximation |
| Open Source Code | Yes | the code to reproduce the SCWD calculations for CMIP5 and CMIP6 to the reference datasets is provided in the supplemental material. |
| Open Datasets | Yes | To serve as references for climate model evaluation, we collect the European Centre for Medium Range Weather Forecasts (ECMWF) Reanalysis 5th Generation (ERA5) [Hersbach et al., 2020] as well as the Reanalysis-2 data from the National Centers for Environmental Protection (NCEP) [Kanamitsu et al., 2002] reanalysis datasets. |
| Dataset Splits | No | The paper uses observational and reanalysis datasets as "references for climate model evaluation" but does not specify training, validation, or test dataset splits in a traditional machine learning context. |
| Hardware Specification | Yes | All computations were performed using R version 4.3.2 on an Ubuntu operating system with an Intel i5-9600k processor (6 cores), 32GB RAM, and 3TB hard disk space. |
| Software Dependencies | Yes | All computations were performed using R version 4.3.2 |
| Experiment Setup | Yes | All SCWD calculations in this section use the Wendland kernel with range parameter of 1,000km for slicing. |