Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Statistical Regimes and Runtime Prediction
Authors: Barry Hurley, Barry O'Sullivan
IJCAI 2015 | Venue PDF | LLM Run Details | Input Tokens: 16,215 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,473 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Supported by a large-scale empirical study employing many years of industrial SAT Competition instances including repeated runs, we present statistical and empirical evidence that such a performance variation phenomenon necessitates a change in the evaluation of portfolio, runtime prediction, and automated configuration methods. |
| Researcher Affiliation | Academia | Barry Hurley and Barry O Sullivan Insight Centre for Data Analytics, University College Cork, Ireland |
| Pseudocode | No | The paper does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper states that the dataset is available at a URL, but it does not provide an explicit statement or link for the open-source code of the methodology described in the paper. |
| Open Datasets | Yes | The dataset is available at http://ucc.insight-centre.org/bhurley/ |
| Dataset Splits | Yes | We use a standard randomised 10-fold cross-validation, where the dataset is split in 10 folds. |
| Hardware Specification | Yes | Performance data was collected on a cluster of Intel Xeon E5430 2.66Ghz processors running Cent OS 6. |
| Software Dependencies | Yes | We use Mini Sat 2.0 [E en and S orensson, 2004] as the solver... running Cent OS 6. |
| Experiment Setup | Yes | For the predictions in this section we will use the adjusted logtransformed runtime, log10(1 + runtime). ... We use a random forest with the same parameters as [Hutter et al., 2014], that is using 10 regression trees, using half of the variables as split variables at each node (perc = 0.5), a maximum of 5 data-points in leaf nodes (nmin = 5). |