Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Utilitarian Algorithm Configuration
Authors: Devon Graham, Kevin Leyton-Brown, Tim Roughgarden
NeurIPS 2023 | Venue PDF | LLM Run Details | Input Tokens: 15,443 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,629 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We now illustrate the runtime costs of utilitarian algorithm configuration and the impacts of the adaptive improvements offered by UP over Naive. We leverage three datasets from Weisz et al. (2020). |
| Researcher Affiliation | Collaboration | Devon R. Graham Department of Computer Science University of British Columbia Vancouver, BC EMAIL Kevin Leyton-Brown Department of Computer Science University of British Columbia Vancouver, BC EMAIL Tim Roughgarden Columbia University & a16z crypto New York, NY EMAIL |
| Pseudocode | Yes | Algorithm 1 Runtime Oracle Procedure; Algorithm 2 Captime Verification Procedure; Algorithm 3 Naive Procedure; Algorithm 4 Utilitarian Procrastination |
| Open Source Code | Yes | Code to reproduce all plots can be found at https://github.com/drgrhm/utilitarian-ac. |
| Open Datasets | Yes | We leverage three datasets from Weisz et al. (2020). The first is a set of runtimes for the minisat SAT solver on data generated by the CNFuzzdd instance generator. The others are sets of runtimes for the CPLEX integer program solver on the combinatorial auction winner determination instances (regions) and on woodpecker conservation problems (rcw); see Appendix D of Weisz et al. (2020) for details. |
| Dataset Splits | No | The paper mentions using specific datasets and refers to a 'first seed' for CPLEX datasets but does not provide explicit details about training, validation, or test data splits. |
| Hardware Specification | No | The paper mentions 'cloud computing resources' but does not specify any particular hardware details such as GPU models, CPU types, or memory sizes used for the experiments. |
| Software Dependencies | No | The paper mentions using 'minisat SAT solver' and 'CPLEX integer program solver' but does not provide specific version numbers for these or any other software dependencies, such as programming languages or libraries. |
| Experiment Setup | Yes | We used values of 0.1, 0.15 and 0.2 and set δ = 0.1. |