Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
TopRank: A practical algorithm for online stochastic ranking
Authors: Tor Lattimore, Branislav Kveton, Shuai Li, Csaba Szepesvari
NeurIPS 2018 | Venue PDF | LLM Run Details | Input Tokens: 14,754 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,456 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 6 Experiments We experiment with the Yandex dataset [15], a dataset of 167 million search queries. ... Top Rank is compared to Batch Rank [17] and Cascade KL-UCB [6]. ... The results are averaged over 10 runs. |
| Researcher Affiliation | Collaboration | Tor Lattimore Deep Mind Branislav Kveton Google Shuai Li The Chinese University of Hong Kong Csaba Szepesvári Deep Mind and University of Alberta |
| Pseudocode | Yes | Algorithm 1 Top Rank |
| Open Source Code | No | The paper mentions using 'Py Click' and the 'implementation of Batch Rank by Zoghi et al. [17]' but does not provide any links or explicit statements about releasing their own source code for Top Rank. |
| Open Datasets | Yes | We experiment with the Yandex dataset [15], a dataset of 167 million search queries. ... [15] Yandex. Yandex personalized web search challenge. https://www.kaggle.com/c/yandexpersonalized-web-search-challenge, 2013. |
| Dataset Splits | No | The paper mentions selecting '60 frequent search queries' and learning 'CMs and PBMs' but does not specify explicit training, validation, or test dataset splits (e.g., percentages or sample counts). |
| Hardware Specification | No | The paper does not provide specific details about the hardware used to run the experiments (e.g., GPU/CPU models, memory). |
| Software Dependencies | No | The paper mentions using 'Py Click' and an 'implementation of Batch Rank', but it does not provide specific version numbers for these or any other software dependencies. |
| Experiment Setup | Yes | Our goal is to rerank L = 10 most attractive items with the objective of maximizing the expected number of clicks at the first K = 5 positions. This is the same experimental setup as in Zoghi et al. [17]. ... The parameter δ in Top Rank is set as δ = 1/n, as suggested in Theorem 1. |