Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Tree-of-Reasoning Question Decomposition for Complex Question Answering with Large Language Models
Authors: Kun Zhang, Jiali Zeng, Fandong Meng, Yuanzhuo Wang, Shiqi Sun, Long Bai, Huawei Shen, Jie Zhou
AAAI 2024 | Venue PDF | LLM Run Details | Input Tokens: 17,026 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,774 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our proposed framework on four benchmark datasets. The experimental results demonstrate that our proposed methods consistently outperform baseline methods outperform strong baselines by a substantial margin across all datasets. |
| Researcher Affiliation | Collaboration | 1CAS Key Laboratory of AI Security, Institute of Computing Technology, Chinese Academy of Sciences 2School of Computer Science and Technology, University of Chinese Academy of Sciences 3Pattern Recognition Center, We Chat AI, Tencent Inc, China 4Big Data Academy, Zhongke |
| Pseudocode | Yes | Algorithm 1: Processing a Complex Question |
| Open Source Code | No | The paper does not explicitly state that the source code for their methodology is publicly available, nor does it provide a link to a code repository for their work. |
| Open Datasets | Yes | We select four complex multi-hop question-answering datasets, Hotpot QA(Yang et al. 2018), Hy Brid QA(Chen et al. 2020), Musique(Trivedi et al. 2022), and Wiki Multi Hop QA (WMHQA)(Ho et al. 2020). |
| Dataset Splits | No | The paper describes the datasets used (Hotpot QA, Hy Brid QA, Musique, WMHQA) but does not explicitly state the specific training, validation, and test splits (e.g., percentages or sample counts) used for its experiments. |
| Hardware Specification | No | The paper mentions using 'gpt-3.5-turbo' via OpenAI's API and 'ColBERTv2' as a retrieval model, but it does not provide any specific details about the hardware (e.g., GPU models, CPU types) used for running their experiments or training their own models. |
| Software Dependencies | No | The paper mentions using 'gpt-3.5-turbo' and 'ColBERTv2' as well as 'T5' as the basis model, but it does not specify version numbers for these or any other software dependencies, such as programming languages or deep learning frameworks. |
| Experiment Setup | No | The paper describes the models used (gpt-3.5-turbo, ColBERTv2, T5) and data collection for training custom components (RTC, QG), but it does not provide specific experimental setup details such as hyperparameters (e.g., learning rate, batch size, number of epochs, optimizer settings) for training their models. |